Argonne Gaudi2 x8 DS-MII Llama-3-8B b64
18187.0 tok/s aggregate
Fields
id
benchmarks/argonne-gaudi2-deepspeed-llama3-8b-bf16-8x-b64-128io-tpsMetric
tps_aggregate
Value
18,187
Unit
tok/s aggregate
Model
meta-llama/Meta-Llama-3-8B
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench arXiv:2411.00136, All_results.csv). niche-deepspeed-mii, FP16/BF16, 128in/128out, AGGREGATE over batch. 8x Gaudi2 bs64.
Measured by
third_party
Reproducible
yes
notes
Multi-accelerator count noted in methodology.
Notes
Multi-accelerator count noted in methodology.