TensorRT-LLM issue #5581: 'Unsupported SM version for FP8 block scaling GEMM' on 8x RTX 6000 PRO; contains an NVIDIA engineer's arch explanation and a community counter-measurement
forum
Fields
id
sources/comm-github-trtllm-5581-8x-rtx6000pro-deepseekPublisher
TensorRT-LLM GitHub issues
Kind
forum
Accessed
2026-10-04
notes
24 comments, unusually substantive. Reporter grimulkan: 8x RTX 6000 PRO (768GB VRAM total), FP8 weights refused, FP4 model loads but runs at 27 tok/s at batch 1. VALLIS-NERIA gives the arch answer: SM100 supports FP8 block-scaled GEMM, SM120 does not, and SM90/100/120 have completely different FP8 instruction sets so newer is not a superset. Independent counter-report by aikitoria: 8x RTX 4090 with a P2P-patched driver runs Mistral Large fine, and ExLlamaV2 TP over PCIe P2P is much faster than mp/pp.
Notes
24 comments, unusually substantive. Reporter grimulkan: 8x RTX 6000 PRO (768GB VRAM total), FP8 weights refused, FP4 model loads but runs at 27 tok/s at batch 1. VALLIS-NERIA gives the arch answer: SM100 supports FP8 block-scaled GEMM, SM120 does not, and SM90/100/120 have completely different FP8 instruction sets so newer is not a superset. Independent counter-report by aikitoria: 8x RTX 4090 with a P2P-patched driver runs Mistral Large fine, and ExLlamaV2 TP over PCIe P2P is much faster than mp/pp.
Cross-references
This record references no other records.