inference/research
verifiedmedium · 0.80updated 2026-10-17

TT-Metal tech report: FlashAttention on Wormhole (Colman Glagovich)

whitepaper

Fields

id
sources/tt-metal-flashattention-wormhole-report
Publisher
Tenstorrent
Kind
whitepaper
Published
2024-09-04
Accessed
2026-10-17
notes
Vendor engineering report, not a peer-reviewed paper. Added in one commit, 0f7aa9bbf2043d309369c3fce0b1e9cdcdf17f9d, authored by Colman Glagovich and committed 2024-09-04 (PR #12228, message 'Create tech_reports/ directory and add first report on FlashAttention'); the GitHub commits API for this path returns that single commit, so the text on main as read 2026-10-17 is the original text. It measures a fused FlashAttention-2-style op against the stack's own unfused softmax attention on Wormhole, with hidden dim fixed at 2048, head dims 64/128/256, sequence length 512 to 16k, and batch chosen so the total token count is 16k. The absolute TFLOP/s numbers live in Figure 6 and are not transcribed into the markdown, so only the prose ratios are citable. The 20x average and the 9x-to-44x range are the vendor's own measurement against its own baseline.

Notes

Vendor engineering report, not a peer-reviewed paper. Added in one commit, 0f7aa9bbf2043d309369c3fce0b1e9cdcdf17f9d, authored by Colman Glagovich and committed 2024-09-04 (PR #12228, message 'Create tech_reports/ directory and add first report on FlashAttention'); the GitHub commits API for this path returns that single commit, so the text on main as read 2026-10-17 is the original text. It measures a fused FlashAttention-2-style op against the stack's own unfused softmax attention on Wormhole, with hidden dim fixed at 2048, head dims 64/128/256, sequence length 512 to 16k, and batch chosen so the total token count is 16k. The absolute TFLOP/s numbers live in Figure 6 and are not transcribed into the markdown, so only the prose ratios are citable. The 20x average and the 9x-to-44x range are the vendor's own measurement against its own baseline.

Cross-references

This record references no other records.