inference/research
verifiedhigh · 0.90updated 2026-10-05

Decode-phase GEMM: memory-bandwidth bound at batch size 1

memory bound

Fields

id
flops/qhw-decode-gemm-memory-bound
Class
decode_gemm
Bound by
memory
Arithmetic intensity
At batch size 1, the GEMM is memory-bandwidth bound: each weight is loaded from HBM and used for only one multiply-accumulate. The arithmetic intensity is ~1 flop/byte, far below the GPU's ridge point.
Scales with
batch_sizehidden_widthintermediate_sizenum_layersnum_experts (MoE)
Hardware parameters
parametermemory_bandwidtheffectAt batch size 1, the GEMM is memory-bandwidth bound: each weight is loaded from HBM and used for only one multiply-accumulate.basisunverifiedparameterpeak_flopseffectThe arithmetic intensity is ~1 flop/byte, far below the GPU's ridge point.basisrecordedmagnitude1 flop/byteparameterl2_cache_sizeeffect2026-10-04: affected_by_hardware previously held ['memory_bandwidth', 'l2_cache_size', 'tensor_core_count'] - hardware quantities, not products.basisunverifiedparametertensor_core_counteffect2026-10-04: affected_by_hardware previously held ['memory_bandwidth', 'l2_cache_size', 'tensor_core_count'] - hardware quantities, not products.basisunverified
Workarounds
Weight-only quantization (INT4, FP4) reduces bytes per weight, directly improving memory-bound throughputBatching increases arithmetic intensity but reduces throughput for interactive workloadsSpeculative decoding increases effective batch size for the target modelMoE routing reduces active parameters per token
notes
HARDWARE PARAMETERS, NOT RECORD IDS. 2026-10-04: affected_by_hardware previously held ['memory_bandwidth', 'l2_cache_size', 'tensor_core_count'] - hardware quantities, not products. They could never resolve, so the repo was carrying permanent false debt against a budget that exists to catch NEW bad references. Moved into this note; the field is now EMPTY and that is the correct value, not a gap. Why: These were hardware PARAMETERS (memory_bandwidth, l2_cache_size, tensor_core_count), not products. The record is part-independent too - it is a kernel-level argument citing the Marlin repo, so naming a GPU would invent a dependency it does not claim. The record's own point is that memory bandwidth, L2 size and tensor-core count are the variables that SET the ridge point on any part. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Notes

HARDWARE PARAMETERS, NOT RECORD IDS. 2026-10-04: affected_by_hardware previously held ['memory_bandwidth', 'l2_cache_size', 'tensor_core_count'] - hardware quantities, not products. They could never resolve, so the repo was carrying permanent false debt against a budget that exists to catch NEW bad references. Moved into this note; the field is now EMPTY and that is the correct value, not a gap. Why: These were hardware PARAMETERS (memory_bandwidth, l2_cache_size, tensor_core_count), not products. The record is part-independent too - it is a kernel-level argument citing the Marlin repo, so naming a GPU would invent a dependency it does not claim. The record's own point is that memory bandwidth, L2 size and tensor-core count are the variables that SET the ridge point on any part.

HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.