inference/research
verifiedmedium · 0.70updated 2026-10-04

vLLM issue #40124: TurboQuant KV cache (k8v4) + hybrid MoE does not work on Ampere (SM 80-86); reporter ships 13 runtime monkey-patches

forum

Fields

id
sources/comm-github-vllm-40124-turboquant-ampere
Publisher
vLLM GitHub issues
Kind
forum
Accessed
2026-10-04
notes
Reporter Sandermage, 2x NVIDIA RTX A5000 24GB (SM86), vLLM 0.19.1rc1 nightly, driver 570.211.01, CUDA 12.8, Qwen/Qwen3-35B-A3B-FP8 hybrid MoE (30 MoE + 10 dense layers, E=256, top_k=8, block_shape 128x128), kv_cache_dtype=turboquant_k8v4, max_model_len=163840. Main mechanism is arch-gating: TritonBlockFP8ScaledMM uses fp8e4nv which requires SM >= 89 and reports 'supported' below that, so it is selected then fails. Corroborated by independent in-thread analysis (TCR020) against upstream PR #40105.

Notes

Reporter Sandermage, 2x NVIDIA RTX A5000 24GB (SM86), vLLM 0.19.1rc1 nightly, driver 570.211.01, CUDA 12.8, Qwen/Qwen3-35B-A3B-FP8 hybrid MoE (30 MoE + 10 dense layers, E=256, top_k=8, block_shape 128x128), kv_cache_dtype=turboquant_k8v4, max_model_len=163840. Main mechanism is arch-gating: TritonBlockFP8ScaledMM uses fp8e4nv which requires SM >= 89 and reports 'supported' below that, so it is selected then fails. Corroborated by independent in-thread analysis (TCR020) against upstream PR #40105.

Cross-references

This record references no other records.