inference/research
verifiedhigh · 0.85updated 2026-10-03

llmcompressor (Red Hat AI / vLLM project) + compressed-tensors checkpoint format

the scheme set is named, not numeric: FP8_BLOCK, FP8_DYNAMIC, INT8, W2A16..W8A16 (and G32/G64 group variants), MXFP4, MXFP8, NVFP4, plus arbitrary-bit and mixed-precision. The documented worked example is FP8 weights with RTN at block_size 128 and dynamic FP8 activations

Fields

id
quantization/llmcompressor-quant
Scheme
llmcompressor-compressed-tensors
Bits
the scheme set is named, not numeric: FP8_BLOCK, FP8_DYNAMIC, INT8, W2A16..W8A16 (and G32/G64 group variants), MXFP4, MXFP8, NVFP4, plus arbitrary-bit and mixed-precision. The documented worked example is FP8 weights with RTN at block_size 128 and dynamic FP8 activations
Weight group size
128
Activation scheme
per scheme: dynamic FP8 activations, quantized KV cache, or weight-only (16-bit activations). compressed-tensors also supports rotation-based quantization (Hadamard, random Hadamard, random matrix transforms)
Kernels
vLLM kernels consuming compressed-tensors checkpoints (Marlin / Machete / cutlass INT8 GEMM / CompressedTensorsWNA16)compressed-tensors config_groups: format (e.g. mxfp4-pack-quantized), targets, per-group weights/input_activations specstransform_config (Hadamard / random Hadamard / random matrix)
Quality delta
n/a - the README ships no controlled fp16 delta. The documented example uses Qwen/Qwen3-30B-A3B and only says 'Confirm generations of the quantized model look sane'. Null + note.
notes
VERIFIED EXISTENCE AND OWNERSHIP, which was in question: yes, llmcompressor exists. It is at vllm-project/llm-compressor (Apache-2.0, ~3.8k stars, 3300+ commits), cited as 'author = {Red Hat AI and vLLM Project}, year 2024', and its README links the HuggingFace org RedHatAI. So the 'newer Red Hat tool' is real and is jointly owned with vLLM, not a Red Hat-only fork. COMPRESSED-TENSORS IS STILL THE FORMAT NAME - verified, and the repo has MOVED from the neuralmagic org to vllm-project (GitHub redirects the old URL); the project name and the 'compressed-tensors' format identifier in config.json ('quant_method': 'compressed-tensors') are unchanged. Its self-description is 'A safetensors extension to efficiently store sparse quantized tensors on disk'. WHAT IT ACTUALLY SUPPORTS (README feature list): a unified checkpoint format; wide compatibility with GPTQ, SmoothQuant, AWQ and AutoRound; activation quantization; mixed precision; low/arbitrary-bit; KV cache quantization; NON-UNIFORM schemes where different layers are quantized differently; SPARSITY including unstructured and semi-structured 2:4; rotation-based quantization transforms; checkpoint conversion to and from AutoAWQ, ModelOpt NVFP4 and FP8 block; and transparent CPU/disk/distributed offloading for models larger than VRAM. It integrates with HF models, PyTorch, vLLM and SGLang. CONFIG SHAPE, quoted from a real emitted config.json: quantization_config.config_groups.group_0.format = 'mxfp4-pack-quantized', with per-group weights and input_activations objects carrying num_bits 4, group_size 32, strategy 'group', symmetric true, dynamic false for weights and dynamic true for input activations, scale_dtype torch.uint8, observer memoryless_minmax, plus top-level 'ignore': ['lm_head'], 'sparsity_config', 'transform_config' and 'quantization_status': 'compressed'. The per-layer ignore list is how the lm_head is kept at full precision. THE SPARSELLM QUESTION, answered definitively and it is a negative answer worth recording: the 2:4 structured-sparsity stack (DeepSparse / SparseML / SparseZoo / Sparsify) is DEAD. The SparseML README carries an explicit end-of-life notice: after Red Hat completed its acquisition of Neural Magic in January 2025, development of the community versions of DeepSparse (including Enterprise), SparseML, SparseZoo and Sparsify was ceased and deprecated on June 2, 2025 and they no longer receive updates or support, with focus shifted to offerings built around vLLM. So if you want 2:4 sparsity today it is via compressed-tensors' sparsity_config and vLLM, not via the SparseLLM-era toolchain. Do not write a 'SparseLLM' record citing SparseML as current. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'fp8-blockwise' (a proxy recorded while the enum lacked a legal value) and is now 'llmcompressor-compressed-tensors'.

Notes

VERIFIED EXISTENCE AND OWNERSHIP, which was in question: yes, llmcompressor exists. It is at vllm-project/llm-compressor (Apache-2.0, ~3.8k stars, 3300+ commits), cited as 'author = {Red Hat AI and vLLM Project}, year 2024', and its README links the HuggingFace org RedHatAI. So the 'newer Red Hat tool' is real and is jointly owned with vLLM, not a Red Hat-only fork. COMPRESSED-TENSORS IS STILL THE FORMAT NAME - verified, and the repo has MOVED from the neuralmagic org to vllm-project (GitHub redirects the old URL); the project name and the 'compressed-tensors' format identifier in config.json ('quant_method': 'compressed-tensors') are unchanged. Its self-description is 'A safetensors extension to efficiently store sparse quantized tensors on disk'. WHAT IT ACTUALLY SUPPORTS (README feature list): a unified checkpoint format; wide compatibility with GPTQ, SmoothQuant, AWQ and AutoRound; activation quantization; mixed precision; low/arbitrary-bit; KV cache quantization; NON-UNIFORM schemes where different layers are quantized differently; SPARSITY including unstructured and semi-structured 2:4; rotation-based quantization transforms; checkpoint conversion to and from AutoAWQ, ModelOpt NVFP4 and FP8 block; and transparent CPU/disk/distributed offloading for models larger than VRAM. It integrates with HF models, PyTorch, vLLM and SGLang. CONFIG SHAPE, quoted from a real emitted config.json: quantization_config.config_groups.group_0.format = 'mxfp4-pack-quantized', with per-group weights and input_activations objects carrying num_bits 4, group_size 32, strategy 'group', symmetric true, dynamic false for weights and dynamic true for input activations, scale_dtype torch.uint8, observer memoryless_minmax, plus top-level 'ignore': ['lm_head'], 'sparsity_config', 'transform_config' and 'quantization_status': 'compressed'. The per-layer ignore list is how the lm_head is kept at full precision. THE SPARSELLM QUESTION, answered definitively and it is a negative answer worth recording: the 2:4 structured-sparsity stack (DeepSparse / SparseML / SparseZoo / Sparsify) is DEAD. The SparseML README carries an explicit end-of-life notice: after Red Hat completed its acquisition of Neural Magic in January 2025, development of the community versions of DeepSparse (including Enterprise), SparseML, SparseZoo and Sparsify was ceased and deprecated on June 2, 2025 and they no longer receive updates or support, with focus shifted to offerings built around vLLM. So if you want 2:4 sparsity today it is via compressed-tensors' sparsity_config and vLLM, not via the SparseLLM-era toolchain. Do not write a 'SparseLLM' record citing SparseML as current. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'fp8-blockwise' (a proxy recorded while the enum lacked a legal value) and is now 'llmcompressor-compressed-tensors'.

Cross-references

Emulated support (2)

Native support (6)