inference/research
drafthigh · 0.95updated 2026-10-03

vLLM MacheteLinearKernel dispatch and hardware gate

repo
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
sources/kern-machete-vllm-python
Publisher
vllm-project
Kind
repo
Accessed
2026-10-03
notes
Engine dispatch layer (Apache-2.0). get_min_capability() returns 90; can_implement() returns False for non-CUDA platforms ('Machete only supported on CUDA') and False unless device capability is 90 ('Machete requires compute capability of 90 (Hopper)'), then gates on weight type and group size. This is the clearest in-repo example that the same weight-only 4-bit model silently takes a different GEMM kernel per engine and per GPU arch - a Hopper box gets Machete, an Ampere box falls back to Marlin, and llama.cpp uses an entirely different INT8/DP4A MMQ path.

Notes

Engine dispatch layer (Apache-2.0). get_min_capability() returns 90; can_implement() returns False for non-CUDA platforms ('Machete only supported on CUDA') and False unless device capability is 90 ('Machete requires compute capability of 90 (Hopper)'), then gates on weight type and group size. This is the clearest in-repo example that the same weight-only 4-bit model silently takes a different GEMM kernel per engine and per GPU arch - a Hopper box gets Machete, an Ampere box falls back to Marlin, and llama.cpp uses an entirely different INT8/DP4A MMQ path.

Cross-references

This record references no other records.