2x RX 7900 XTX llama.cpp ROCm 7.1.1: Llama 3.1 70B Q4_K_M FA-on tg128
13.37 tok/s (tg128, per-request, batch 1, split across 2 GPUs)
Fields
id
benchmarks/rocm711-2x-rx7900xtx-llama31-70b-q4km-fa-tg128Metric
decode_tok_s
Value
13.37
Unit
tok/s (tg128, per-request, batch 1, split across 2 GPUs)
Model
Llama 3.1 70B Q4_K_M (39.59 GiB, 70.55B)
Methodology
Same box and build as the 7B rows: llama.cpp 34ce48d97, ROCm 7.1.1, llama-bench ngl 99, fa 1, test tg128. The model is 39.59 GiB, so it is split across both 24 GB cards; there is no GPU-to-GPU link beyond the host PCIe Gen4 bus. Reported 13.37 +/- 0.01. Per-request batch-1 decode of the split model, not an aggregate of two independent requests. The no-FA cell of the same run was 13.06 +/- 0.01.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
pp512 on the same dual-GPU run was 341.41 +/- 0.23 with FA on and 330.06 +/- 0.38 with FA off. The repo's own 7B dual-GPU test was slower than one card (tg128 94.13 against 118 to 130), so the split only pays for models that do not fit in 24 GB.
Notes
pp512 on the same dual-GPU run was 341.41 +/- 0.23 with FA on and 330.06 +/- 0.38 with FA off. The repo's own 7B dual-GPU test was slower than one card (tg128 94.13 against 118 to 130), so the split only pays for models that do not fit in 24 GB.