Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-8B-Instruct FP8 1HPU bs1536 128in/128out aggregate tok/s
23922 output tok/s AGGREGATE across 1x Gaudi 3 (batch 1536; ~23922 tok/s per HPU)
Fields
id
benchmarks/gaudi3-1hpu-llama31-8b-fp8-128-128Metric
tps_aggregate
Value
23,922
Unit
output tok/s AGGREGATE across 1x Gaudi 3 (batch 1536; ~23922 tok/s per HPU)
Model
Llama-3.1-8B-Instruct
Methodology
VENDOR-MEASURED (Intel model-performance table, Gaudi software 1.24, Optimum-Habana + Hugging Face model, maxabs FP8). Llama-3.1-8B-Instruct FP8 on 1 HPU, batch_size=1536, input_len=128, output_len=128. This is MAX THROUGHPUT under a very large batch — NOT a single-user decode figure. ~23922 tok/s per HPU.
Measured by
vendor
Reproducible
yes
notes
Optimum-Habana run_generation.py path, use_hpu_graphs + flash_attention_recompute. Reproducible via the vendor's Docker recipe; numbers are peak-serving, not latency-oriented.
Notes
Optimum-Habana run_generation.py path, use_hpu_graphs + flash_attention_recompute. Reproducible via the vendor's Docker recipe; numbers are peak-serving, not latency-oriented.