Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 2HPU bs450 128in/2048out aggregate tok/s
6514 output tok/s AGGREGATE across 2x Gaudi 3 (batch 450; ~3257 tok/s per HPU)
Fields
id
benchmarks/gaudi3-2hpu-llama31-70b-fp8-128-2048Metric
tps_aggregate
Value
6,514
Unit
output tok/s AGGREGATE across 2x Gaudi 3 (batch 450; ~3257 tok/s per HPU)
Model
Llama-3.1-70B-Instruct
Methodology
VENDOR-MEASURED (Intel model-performance table, Gaudi software 1.24, Optimum-Habana + Hugging Face model, maxabs FP8). Llama-3.1-70B-Instruct FP8 on 2 HPU, batch_size=450, input_len=128, output_len=2048. This is MAX THROUGHPUT under a very large batch — NOT a single-user decode figure. ~3257 tok/s per HPU.
Measured by
vendor
Reproducible
yes
notes
TP=2. ~3257 tok/s per HPU at bs450, 128->2048.
Notes
TP=2. ~3257 tok/s per HPU at bs450, 128->2048.