TensorRT-LLM: Llama 3.1 405B FP8 on 8x H100 SXM 80GB (TP=8), ISL/OSL 1000/2000
2884.69 output tokens/s (aggregate, 8 GPUs)
Fields
id
benchmarks/trtllm-llama31-405b-fp8-h100-tp8-tpsMetric
tps_aggregate
Value
2,884.69
Unit
output tokens/s (aggregate, 8 GPUs)
Model
nvidia/Llama-3.1-405B-Instruct-FP8
Methodology
TensorRT-LLM PyTorch backend, trtllm-bench throughput. Hardware: 8x NVIDIA H100 SXM 80GB, tensor-parallel size = 8, PP=1. ISL = 1000, OSL = 2000, uniform synthetic dataset, 1500 requests. Offline infinite-rate feed; concurrency not fixed and the achieved batch size is not stated. Aggregate output tokens/s over the 8-GPU TP group (about 361 tok/s per GPU). KV cache free-memory fraction 0.95; CUDA graphs 1-8192.
Measured by
vendor
Reproducible
yes
notes
Measured by NVIDIA, published in the TensorRT-LLM performance overview and reproducible with the documented `prepare_dataset.py` + `trtllm-bench throughput --backend pytorch` workflow (llm_options.yml with CUDA-graph batch sizes 1..8192, kv_cache_free_gpu_mem_fraction 0.95). Metric is 'Total Output Throughput (tokens/sec)' as trtllm-bench reports it: AGGREGATE output tokens/s across all in-flight requests at the stated TP size, under an offline infinite-rate request feed, NOT per-request and NOT per-GPU. Model weights are NVIDIA-published ModelOpt-quantized checkpoints. The page explicitly states these are reference points, not peak performance, and that batch size / parallelism were left mostly at defaults.
Notes
Measured by NVIDIA, published in the TensorRT-LLM performance overview and reproducible with the documented `prepare_dataset.py` + `trtllm-bench throughput --backend pytorch` workflow (llm_options.yml with CUDA-graph batch sizes 1..8192, kv_cache_free_gpu_mem_fraction 0.95). Metric is 'Total Output Throughput (tokens/sec)' as trtllm-bench reports it: AGGREGATE output tokens/s across all in-flight requests at the stated TP size, under an offline infinite-rate request feed, NOT per-request and NOT per-GPU. Model weights are NVIDIA-published ModelOpt-quantized checkpoints. The page explicitly states these are reference points, not peak performance, and that batch size / parallelism were left mostly at defaults.