TensorRT-LLM: Llama 3.1 8B FP8 on 1x H200 SXM 141GB, ISL/OSL 1000/1000
17162.49 output tokens/s (aggregate, 1 GPU)
Fields
id
benchmarks/trtllm-llama31-8b-fp8-h200-1k1k-tpsMetric
tps_aggregate
Value
17,162.49
Unit
output tokens/s (aggregate, 1 GPU)
Model
nvidia/Llama-3.1-8B-Instruct-FP8
Methodology
Identical workflow to the H100 point above and directly comparable: TensorRT-LLM PyTorch backend, trtllm-bench throughput, 1x NVIDIA H200 SXM 141GB (DGX H200), TP=1, PP=1, ISL/OSL 1000/1000 uniform, 3000 requests, offline infinite-rate feed so concurrency is NOT fixed and the achieved in-flight count is not stated by the source. Aggregate output tokens/s for the single GPU. KV cache free-memory fraction 0.95, CUDA graphs 1-8192.
Measured by
vendor
Reproducible
yes
notes
Measured by NVIDIA, published in the TensorRT-LLM performance overview and reproducible with the documented `prepare_dataset.py` + `trtllm-bench throughput --backend pytorch` workflow (llm_options.yml with CUDA-graph batch sizes 1..8192, kv_cache_free_gpu_mem_fraction 0.95). Metric is 'Total Output Throughput (tokens/sec)' as trtllm-bench reports it: AGGREGATE output tokens/s across all in-flight requests at the stated TP size, under an offline infinite-rate request feed, NOT per-request and NOT per-GPU. Model weights are NVIDIA-published ModelOpt-quantized checkpoints. The page explicitly states these are reference points, not peak performance, and that batch size / parallelism were left mostly at defaults.
Notes
Measured by NVIDIA, published in the TensorRT-LLM performance overview and reproducible with the documented `prepare_dataset.py` + `trtllm-bench throughput --backend pytorch` workflow (llm_options.yml with CUDA-graph batch sizes 1..8192, kv_cache_free_gpu_mem_fraction 0.95). Metric is 'Total Output Throughput (tokens/sec)' as trtllm-bench reports it: AGGREGATE output tokens/s across all in-flight requests at the stated TP size, under an offline infinite-rate request feed, NOT per-request and NOT per-GPU. Model weights are NVIDIA-published ModelOpt-quantized checkpoints. The page explicitly states these are reference points, not peak performance, and that batch size / parallelism were left mostly at defaults.