Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Mixtral-8x7B-v0.1, 128in/128out, batch 1
195.84 tok/s aggregate across batch 1 (4x A100-40GB-SXM)
Fields
id
benchmarks/argonne-a100-vllm-mixtral-8x7b-fp16-4x-b1-128io-tpsMetric
tps_aggregate
Value
195.84
Unit
tok/s aggregate across batch 1 (4x A100-40GB-SXM)
Model
mistralai/Mixtral-8x7B-v0.1
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). vLLM on NVIDIA A100 SXM 40GB, FP16/BF16 HF weights, 128 in / 128 out, batch 1, 4 GPU(s) (ThetaGPU 4xA100-40GB-SXM node; NVLink when n>1). End-to-end latency 1.307s. Throughput = batch*(in+out)/latency per paper Eq.2 — AGGREGATE over the batch, not per-request decode tok/s.
Measured by
third_party
Reproducible
yes
notes
Argonne ThetaGPU A100 rows are the 40GB SXM variant (not the 80GB SXM nor the PCIe card). Aggregate = batch*tokens/latency; at bs1 it equals single-request e2e throughput (prefill+decode mixed), NOT pure decode tok/s.
Notes
Argonne ThetaGPU A100 rows are the 40GB SXM variant (not the 80GB SXM nor the PCIe card). Aggregate = batch*tokens/latency; at bs1 it equals single-request e2e throughput (prefill+decode mixed), NOT pure decode tok/s.