vLLM silently downgrades FP8 activation scaling from per-token to per-tensor when CUTLASS FP8 is unavailable (Ampere, or CUDA older than 12.4 on Ada): the checkpoint says dynamic per-token, the engine computes per-tensor, nothing warns

An FP8 checkpoint quantized with per-token (per-row) dynamic activation scales produces worse accuracy than the same checkpoint quantized with per-tensor scales would predict, with no error and no warning, and the served config still describes itself as dynamic per-token. The operator sees a small, unexplained accuracy gap that is largest on the shapes where activation outliers matter most (large batches, long prompts), and it is easy to misattribute to the quantization recipe, the calibration set, or the model.

Fp8LinearMethod.__init__() chooses the activation quant key by a silent capability test, not by the checkpoint. The code is: if self.act_q_static: self.activation_quant_key = kFp8StaticTensorSym; elif cutlass_fp8_supported(): self.activation_quant_key = kFp8DynamicTokenSym; else: self.activation_quant_key = kFp8DynamicTensorSym. The weight key in the non-block-quantized branch is hard-coded to kFp8StaticTensorSym. cutlass_fp8_supported() calls ops.cutlass_scaled_mm_supports_fp8(cc), whose C++ body returns CUDA_VERSION >= 12000 for cc >= 90, CUDA_VERSION >= 12040 for 89 <= cc < 90, and false otherwise - so it is false on every Ampere part (SM80/SM86, no branch at all below SM89) and false on Ada or Hopper built against CUDA older than 12.4 / 12.0 respectively. On those parts the engine does not fail and does not warn: it substitutes a per-tensor activation scale for the per-token one the checkpoint was calibrated with. The scaling granularity changes, the numerics change, and nothing in the request path or the served model object records the substitution. A second, independent Ampere behaviour compounds it: MarlinFP8ScaledMMLinearKernel, whose module comment is 'For GPUs that lack FP8 hardware support, we can leverage the Marlin kernel for fast weight-only FP8 quantization', is admitted at has_device_capability(75) and is in both the FP8 and the W8A16 FP8 candidate lists, so on an A100 an FP8 checkpoint is served by a WEIGHT-ONLY kernel - FP8 weights dequantized, FP16 activations - which is the Marlin-for-FP4 silent degradation this repo already records, applied to FP8 instead of FP4.

Before deploying an FP8 checkpoint whose calibration used per-token activation scales, evaluate cutlass_scaled_mm_supports_fp8() for the exact target (it is exposed to Python as torch.ops._C.cutlass_scaled_mm_supports_fp8(capability)) and confirm the CUDA version meets the per-arch requirement: CUDA 12.4 on sm89, CUDA 12.0 on sm90 and above. If it returns false, either move to Ada/Hopper/Blackwell with a new enough toolkit, or re-quantize with per-tensor activation scales so the checkpoint matches what the engine will actually do. At runtime, confirm from the startup log which FP8 kernel was selected and whether it is Marlin; a MarlinFP8ScaledMMLinearKernel line means weight-only. Do not trust the checkpoint's quantization_config as a statement about the running kernel - that is the same class of error as the fp8_rowwise-versus-per-tensor confusion this repo already flags for ROCm, and this one is invisible because there is no warning at all.

CONFIRMED 2026-10-04 against vLLM main (vllm/model_executor/layers/quantization/fp8.py and csrc/.../w8a8/cutlass/scaled_mm_entry.cu). VERIFIED NEGATIVE WITH EVIDENCE for per-token FP8 activation scaling on Ampere in vLLM main: not an absence of checking but a positive assertion in the source - the else-branch assigns kFp8DynamicTensorSym, i.e. the code affirmatively chooses the weaker granularity, and cutlass_scaled_mm_supports_fp8 has no branch below SM89 so it returns false for cc < 89 regardless of CUDA version. VERIFIED NEGATIVE WITH EVIDENCE for a W8A8 FP8 GEMM on Ampere: CudapPlatform.supports_fp8() is defined as has_device_capability(89), and CompressedTensorsW8A8Fp8.get_min_capability() returns 89 with the comment '# lovelace and up'. The Ampere FP8 story is therefore the sharpest one in the whole matrix: an FP8 checkpoint LOADS on an A100 and is served weight-only through Marlin, which is exactly the case the task description flags as a real error in a support table - a cell marked as available where the engine has quietly taken a slower, lower-precision path. Do not mark Ampere FP8 as a plain yes anywhere. Related: the FP8 block-scaled (1x128 / 128x128) path is a different gate entirely - cutlass_scaled_mm_supports_block_fp8 requires CUDA 12.0 on sm90 and CUDA 12.8 on cc >= 100 and returns false below SM90, so block-scaled FP8 is unavailable on Ampere AND Ada AND sm120 through CUTLASS, a fact no vendor matrix expresses because it is a CUDA-version claim rather than an architectural one.