inference/research
verifiedhigh · 0.90updated 2026-10-05

Machete weight-only 4-bit GEMM: CUTLASS 3.x tiles plus an offline prepacked int4 B operand, Hopper only

compute bound

Fields

id
flops/machete-weight-only-hopper-gemm
Class
quantization_overhead
Bound by
compute
Arithmetic intensity
Same class of claim as Marlin - no FLOP change, dequant removed from the runtime path - but achieved differently: Machete keeps CUTLASS's Hopper WGMMA tensor-core mainloop and changes the B OPERAND LAYOUT instead. The kernel source is explicit that LayoutB is ColumnMajor only 'not actually used since B has the prepacked layout', and the tile K extent is computed as 128*8/sizeof_bits<MmaType> so the same integer-tile arithmetic is expressed in the quantized bit width. Weights, group scales and group zeros are prepacked offline into the exact fragment organization WGMMA consumes, so the inner loop issues MMA on already-correctly-laid-out data. Alignments are all 128-bit. The arithmetic consequence: this is a tensor-core bf16/fp16 GEMM over int4 weights dequantized into the MMA fragments on the fly, NOT a hardware int4 tensor-core path - there is no int4 MMA instruction being used.
Scales with
batch_size (M dimension; the kernel is a GEMM so small M is inefficient)group size (gated by query_machete_supported_group_sizes)hidden_width
Hardware parameters
parametertensor_core_counteffectSame class of claim as Marlin - no FLOP change, dequant removed from the runtime path - but achieved differently: Machete keeps CUTLASS's Hopper WGMMA tensor-core mainloop and changes the B OPERAND LAYOUT instead.basisrecordedvalues_fornvidia-h100-sxm,nvidia-h100-pcie,nvidia-h200-sxm
Workarounds
Offline prepacking of B weights, scales and zeros into the WGMMA fragment layoutBuild on CUTLASS 3.x collective GEMM with ArchTag Sm90 and OpClassTensorOpCompute TileShapeK from the quantized bit width rather than from element count
notes
The engine-level support matrix is the most decision-relevant fact and is hard-coded in vLLM: get_min_capability() returns 90, and can_implement() returns False with the message 'Machete requires compute capability of 90 (Hopper)' as well as 'Machete only supported on CUDA'. So a Hopper box gets Machete, an Ampere or Ada box falls through to Marlin, a consumer Blackwell (SM120) is not SM90 so also falls through, and ROCm is excluded outright. That is one model, one quantization format, three different kernels depending on the engine and the GPU - the support matrix of the kernel library, not the model, is what determines decode throughput. Also worth noting: because it is CUTLASS-based it inherits the sm_90a arch-accelerated-feature requirement, so building it for a different arch does not work at all. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Notes

The engine-level support matrix is the most decision-relevant fact and is hard-coded in vLLM: get_min_capability() returns 90, and can_implement() returns False with the message 'Machete requires compute capability of 90 (Hopper)' as well as 'Machete only supported on CUDA'. So a Hopper box gets Machete, an Ampere or Ada box falls through to Marlin, a consumer Blackwell (SM120) is not SM90 so also falls through, and ROCm is excluded outright. That is one model, one quantization format, three different kernels depending on the engine and the GPU - the support matrix of the kernel library, not the model, is what determines decode throughput. Also worth noting: because it is CUTLASS-based it inherits the sm_90a arch-accelerated-feature requirement, so building it for a different arch does not work at all.

HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Cross-references