vLLM MacheteLinearKernel dispatch and hardware gate
repo
Fields
id
sources/kern-machete-vllm-pythonPublisher
vllm-project
Kind
repo
Accessed
2026-10-03
notes
Engine dispatch layer (Apache-2.0). get_min_capability() returns 90; can_implement() returns False for non-CUDA platforms ('Machete only supported on CUDA') and False unless device capability is 90 ('Machete requires compute capability of 90 (Hopper)'), then gates on weight type and group size. This is the clearest in-repo example that the same weight-only 4-bit model silently takes a different GEMM kernel per engine and per GPU arch - a Hopper box gets Machete, an Ampere box falls back to Marlin, and llama.cpp uses an entirely different INT8/DP4A MMQ path.
Notes
Engine dispatch layer (Apache-2.0). get_min_capability() returns 90; can_implement() returns False for non-CUDA platforms ('Machete only supported on CUDA') and False unless device capability is 90 ('Machete requires compute capability of 90 (Hopper)'), then gates on weight type and group size. This is the clearest in-repo example that the same weight-only 4-bit model silently takes a different GEMM kernel per engine and per GPU arch - a Hopper box gets Machete, an Ampere box falls back to Marlin, and llama.cpp uses an entirely different INT8/DP4A MMQ path.
Cross-references
This record references no other records.