2x AMD Instinct MI100, llama.cpp: Llama-2-70B Q6_K decode tok/s (community-measured)
8.32 decode tok/s (single-stream, 2x MI100 combined)
Fields
id
benchmarks/comm-mi100-2x-llama2-70b-q6k-decodeMetric
decode_tok_s
Value
8.32
Unit
decode tok/s (single-stream, 2x MI100 combined)
Model
Llama-2-70B Q6_K (GGUF)
Methodology
COMMUNITY-MEASURED. llama.cpp discussion #2824 (Aug-2023, still the canonical MI100 report). Llama-2-70B Q6_K decode on TWO MI100 32GB cards = 8.32 tok/s; prompt eval 48.9 tok/s. Single card extrapolates to ~4 tok/s. IMPORTANT SETUP CAVEAT: >1 MI100 segfaults on bare-metal ROCm - the working recipe in #2824 is PCIe passthrough into an Ubuntu VM plus the HSA_AMD_SVM kernel option. This is an AGGREGATE across 2 cards, NOT per-card.
Measured by
third_party
Reproducible
yes
notes
The most-cited measured MI100 LLM number. gfx908 is still on the ROCm 10 compat matrix but is a maintenance-tier citizen, and multi-GPU is the landmine - bare-metal multi-card segfaults without the VM+SVM workaround. Best $/(GB x TB/s) in the used-datacenter slice at ~$720-1,299 per 32GB card, but budget real time for ROCm fighting.
Notes
The most-cited measured MI100 LLM number. gfx908 is still on the ROCm 10 compat matrix but is a maintenance-tier citizen, and multi-GPU is the landmine - bare-metal multi-card segfaults without the VM+SVM workaround. Best $/(GB x TB/s) in the used-datacenter slice at ~$720-1,299 per 32GB card, but budget real time for ROCm fighting.