8x NVIDIA V100 32GB SXM2 (DGX-1 NVLink plane): 70B Q4-class decode tok/s (bandwidth-scaled estimate)
40.0 decode tok/s (8-way tensor parallel, NVLink, estimated)
Fields
id
benchmarks/comm-dgx1-8x-v100-llama-70b-q4-decode-estMetric
decode_tok_s
Value
40
Unit
decode tok/s (8-way tensor parallel, NVLink, estimated)
Model
Llama-3.3-70B-class (dense) Q4
Methodology
ESTIMATED, not a measured run - flagged est. The big-memory research file puts 8x V100 32GB tensor-parallel on vLLM/TGI at ~30-50 tok/s on a 70B Q4-class model; value is the 40 tok/s midpoint of that range. Basis: ~900 GB/s per GPU x 8 across the NVLink plane gives ~7.2 TB/s aggregate, far above the ~40 GB-per-token weight read of a 70B Q4 - the bound on Volta is kernel efficiency and the absence of FP8/INT4 tensor cores (GPTQ/AWQ-4bit via vLLM or text-generation-inference), not raw bandwidth.
Measured by
third_party
notes
CREATED 2026-10-15 Phase-A residual gap-fill: the ~30-50 tok/s figure previously lived only inside the used-dgx1-8xv100-32gb-chassis supply record's notes with no benchmark record. This is the fastest tok/s-per-dollar path in the big-memory guide (a ~$3,000 8xV100 box outruns a ~$9,000 M3 Ultra 512GB on 70B) at the cost of Volta software archaeology (FlashAttention-2 does not target Volta - FA1 or older paths; no FP8/INT4 tensor), ~3-3.5 kW draw, and rack-class noise. Treat the range, not the midpoint: real runs cluster wherever the quant kernel and TP overhead land.
Notes
CREATED 2026-10-15 Phase-A residual gap-fill: the ~30-50 tok/s figure previously lived only inside the used-dgx1-8xv100-32gb-chassis supply record's notes with no benchmark record. This is the fastest tok/s-per-dollar path in the big-memory guide (a ~$3,000 8xV100 box outruns a ~$9,000 M3 Ultra 512GB on 70B) at the cost of Volta software archaeology (FlashAttention-2 does not target Volta - FA1 or older paths; no FP8/INT4 tensor), ~3-3.5 kW draw, and rack-class noise. Treat the range, not the midpoint: real runs cluster wherever the quant kernel and TP overhead land.