8x V100 32GB (DGX-1) on 70B Q4-class, TP8 NVLink (reported range)
40.0 tok/s (reported range 30-50; midpoint recorded)
Fields
id
benchmarks/comm-v100-x8-70b-q4-tp8-decode-estMetric
decode_tok_s
Value
40
Unit
tok/s (reported range 30-50; midpoint recorded)
Model
70B Q4-class (GPTQ/AWQ via vLLM/TGI)
Methodology
THIRD-PARTY REPORTED, not a single cited run. vLLM/TensorRT on 8x V100 32GB spread over the NVLink plane delivers roughly 30-50 tok/s on 70B Q4-class - recorded at 40 with the range in unit. Volta has no FP8/INT4 tensor cores; GPTQ/AWQ 4-bit via vLLM or text-generation-inference is the working path (llama.cpp/exllama quant kernels are tuned for Turing+; FA2 does not target Volta).
Measured by
third_party
notes
A ~$3,000 used DGX-1 (8x32GB, 256GB HBM2) runs 70B faster than a $9,000 M3 Ultra 512GB - at ~3kW and 200+ dB-class rack noise. Highest tok/s-per-dollar in the big-memory slice if you can tolerate the garage placement and software archaeology.
Notes
A ~$3,000 used DGX-1 (8x32GB, 256GB HBM2) runs 70B faster than a $9,000 M3 Ultra 512GB - at ~3kW and 200+ dB-class rack noise. Highest tok/s-per-dollar in the big-memory slice if you can tolerate the garage placement and software archaeology.