Argonne Gaudi2 x8 DS-MII Mistral-7B b64
14502.3 tok/s aggregate
Fields
id
benchmarks/argonne-gaudi2-deepspeed-mistral-7b-bf16-8x-b64-128io-tpsMetric
tps_aggregate
Value
14,502.3
Unit
tok/s aggregate
Model
mistralai/Mistral-7B-v0.1
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench arXiv:2411.00136, All_results.csv). niche-deepspeed-mii, FP16/BF16, 128in/128out, AGGREGATE over batch. 8x Gaudi2 bs64.
Measured by
third_party
Reproducible
yes
notes
Multi-accelerator count noted in methodology.
Notes
Multi-accelerator count noted in methodology.