TensorRT-LLM performance overview tables (Total Output Throughput, trtllm-bench)

Fields

id

sources/bench-nv-trtllm-perf-overview

URL

https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html

Publisher

NVIDIA

Kind

benchmark

Published

2026-01

Accessed

2026-10-03

notes

Versioned (default 'latest') TensorRT-LLM performance overview. Throughput tables report 'Total Output Throughput (tokens/sec)' from trtllm-bench under an offline, infinite-rate request feed. Hardware list is explicit: H100 SXM 80GB, H200 SXM 141GB, GH200 96GB, B200 180GB, GB200 192GB. Full reproduce commands (prepare_dataset.py, trtllm-bench, llm_options.yml with CUDA-graph batch sizes, kv_cache_free_gpu_mem_fraction) are published alongside the tables. Page explicitly warns the numbers are a reference point, not peak performance.

Notes

Versioned (default 'latest') TensorRT-LLM performance overview. Throughput tables report 'Total Output Throughput (tokens/sec)' from trtllm-bench under an offline, infinite-rate request feed. Hardware list is explicit: H100 SXM 80GB, H200 SXM 141GB, GH200 96GB, B200 180GB, GB200 192GB. Full reproduce commands (prepare_dataset.py, trtllm-bench, llm_options.yml with CUDA-graph batch sizes, kv_cache_free_gpu_mem_fraction) are published alongside the tables. Page explicitly warns the numbers are a reference point, not peak performance.

Cross-references

This record references no other records.