Quantization runtime cost: dequant, scale unpacking, and batch crossover

Quantization changes the bytes-per-flop ratio of every weight GEMM by the compression factor: a 4-bit weight format cuts weight bytes 4x versus bf16, so decode GEMM AI goes from ~B to ~4B flop/byte and the effective ridge crossing batch drops by roughly the compression factor. The overhead is that unpacking scales and converting to a compute-representable form costs real work and real memory traffic. The question that matters for serving is when quantized kernels stop paying: Marlin reports that for 4-bit weights, batches up to 16-32 still get close to the maximum ~4x quantization speedup, with batches up to 64-128 seeing gradually decreasing but still significant acceleration, and up to 2.8x end-to-end integrated with vLLM (arXiv 2408.11743). That crossover is the whole engineering question - below it, quantization is nearly free speedup; above it, the dequant traffic starts to be paid on every step.

Design consequence: the reason quantization speedups are quoted as 'up to 4x' is that they are batch-conditional, and the condition is a batch number. An engine that quantizes for a latency target (batch 1) gets nearly the full win; an engine that quantizes to serve high concurrency gets a decaying fraction of it, because batching already raised AI and the weight bytes stopped being the binding constraint. Hardware designers should treat 'does it have a native int4/fp8 path' as the decisive question and 'what is the batch crossover of that path' as the second. Also note the asymmetry the Marlin result exposes: a format with no native support is emulated by dequantizing to fp16, which converts a bandwidth win into a bandwidth cost - the dequantized intermediate must be written and re-read, so a '4-bit' model can run slower than bf16 on hardware without a native path. KV-cache quantization is a separate decision recorded under kv_transfer and decode_attention: it does not touch the GEMM ridge at all, it halves the attention and transfer traffic. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.