tt-metal Mixtral-8x7B on QuietBox (8x Wormhole, TP=8): 24.9 tok/s/user, 796.8 tok/s @b32
24.9 tok/s per user (batch-32, aggregate 796.8 tok/s; ISL=128)
Fields
id
benchmarks/ttmetal-wormhole-n300-mixtral8x7b-tp8-b32-tpsuMetric
tok_s_per_user
Value
24.9
Unit
tok/s per user (batch-32, aggregate 796.8 tok/s; ISL=128)
Model
Mixtral-8x7B
Methodology
VENDOR-MEASURED (Tenstorrent tt-metal README featured-models table). Model demos under models/tt_transformers and models/demos/llama3_70b_galaxy. ISL=128 for all LLM rows. T/S/U = tok/s per user (1/inter-token-latency); T/S = T/S/U x batch. TT-Metalium release pinned per row. Values are single-run vendor numbers on their own demo harness, not vLLM end-to-end (vendor notes results may differ under the vLLM TT plugin). Config: QuietBox Wormhole 8-chip node, TP=8, batch 32, TT-Metalium v0.62.0-dev20251015. TTFT 122 ms.
Measured by
vendor
notes
Target T/S/U 33 vs achieved 24.9 per the same vendor table.
Notes
Target T/S/U 33 vs achieved 24.9 per the same vendor table.