verifiedmedium · 0.70updated 2026-10-15

NVIDIA GH200, TensorRT-LLM: Llama-3.1-8B FP8 TP1 128in/2048out aggregate output tok/s

24045.6 tok/s total output, aggregate (max-load)

Fields

benchmarks/trtllm-gh200-llama31-8b-fp8-128-2048-tps

Metric

tps_aggregate

Value

24,045.6

Unit

tok/s total output, aggregate (max-load)

Model

Llama-3.1-8B-Instruct-FP8

Methodology

VENDOR MEASURED (NVIDIA TensorRT-LLM performance overview, v1.1.0rc5 page). trtllm-bench throughput, PyTorch backend, max-load offline scenario on NVIDIA GH200 96GB HBM3 (480GB LPDDR5X), TP1, ModelOpt FP8 weights (nvidia/Llama-3.1-8B-Instruct-FP8). Metric = Total Output Throughput tok/s, SYSTEM-AGGREGATE across queued requests, NOT per-request. ISL/OSL per id.

Measured by

vendor

Reproducible

yes

Cross-references

NVIDIA GH200, TensorRT-LLM: Llama-3.1-8B FP8 TP1 128in/2048out aggregate output tok/s

Fields

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (1)