tt-metal Llama-3.3-70B on Galaxy (32x Wormhole, TP=32): 72.5 tok/s/user, 2268.8 tok/s @b32
72.5 tok/s per user (batch-32, aggregate 2268.8 tok/s across 32-chip Galaxy; ISL=128)
Fields
id
benchmarks/ttmetal-wormhole-galaxy-llama33-70b-tp32-b32-tpsuMetric
tok_s_per_user
Value
72.5
Unit
tok/s per user (batch-32, aggregate 2268.8 tok/s across 32-chip Galaxy; ISL=128)
Model
Llama-3.3-70B
Methodology
VENDOR-MEASURED (Tenstorrent tt-metal README featured-models table). Model demos under models/tt_transformers and models/demos/llama3_70b_galaxy. ISL=128 for all LLM rows. T/S/U = tok/s per user (1/inter-token-latency); T/S = T/S/U x batch. TT-Metalium release pinned per row. Values are single-run vendor numbers on their own demo harness, not vLLM end-to-end (vendor notes results may differ under the vLLM TT plugin). Config: Galaxy Wormhole 32-chip mesh, TP=32, batch 32, TT-Metalium v0.65.0-rc7, vLLM-TT 59be953. TTFT 53 ms.
Measured by
vendor
notes
Galaxy = 32 Wormhole chips; per-chip aggregate = 2268.8/32 = 70.9 tok/s. Recorded on the n300 record because Galaxy chips are the same Wormhole silicon in a larger mesh - the TP=32 Galaxy topology is why per-user rate is far above the 8-chip QuietBox rows. Target T/S/U 80 vs achieved 72.5.
Notes
Galaxy = 32 Wormhole chips; per-chip aggregate = 2268.8/32 = 70.9 tok/s. Recorded on the n300 record because Galaxy chips are the same Wormhole silicon in a larger mesh - the TP=32 Galaxy topology is why per-user rate is far above the 8-chip QuietBox rows. Target T/S/U 80 vs achieved 72.5.