2x MI100 Llama-2-70B Q6_K token-gen
8.32 tok/s decode across 2 cards
Fields
id
benchmarks/comm-mi100-2x-llama2-70b-q6k-tgMetric
decode_tok_s
Value
8.32
Unit
tok/s decode across 2 cards
Model
Llama-2-70B
Methodology
llama.cpp discussion #2824 (Aug-2023), 2x MI100 PCIe-passthrough VM, Q6_K; prompt eval 48.9 t/s
Measured by
third_party
Thermal state
unknown
notes
Single card ~4 t/s 70B. Multi-GPU needed HSA_AMD_SVM + PCIe passthrough into Ubuntu VM - bare-metal multi-MI100 segfaulted. Canonical MI100 report.
Notes
Single card ~4 t/s 70B. Multi-GPU needed HSA_AMD_SVM + PCIe passthrough into Ubuntu VM - bare-metal multi-MI100 segfaulted. Canonical MI100 report.