mlx-lm issue #1573: RotatingKVCache.to_quantized() raises NotImplementedError, so --kv-bits crashes at first request
forum
Fields
id
sources/noncuda-mlxlm-1573-rotating-kv-quant-nyiPublisher
ml-explore/mlx-lm
Kind
forum
Published
2026-07-16
Accessed
2026-10-03
notes
mlx_lm.server starts cleanly and /health returns 200, then the first chat completion raises 'RotatingKVCache Quantization NYI' for Gemma 4's hybrid sliding-window layers (35 of 42 layers in 26B-A4B). The guard in maybe_quantize_kv_cache tests hasattr(c, 'to_quantized'), which is satisfied because the method exists and raises, so the failure is deferred to the first request instead of startup. Confirmed on mlx-lm 0.31.3 and current main. PR #1584 (RotatingQuantizedKVCache) was open and mergeable at the time of access; a reporter verified the hybrid Gemma 4 case on an M5 Max with mlx 0.32.0 using that branch.
Notes
mlx_lm.server starts cleanly and /health returns 200, then the first chat completion raises 'RotatingKVCache Quantization NYI' for Gemma 4's hybrid sliding-window layers (35 of 42 layers in 26B-A4B). The guard in maybe_quantize_kv_cache tests hasattr(c, 'to_quantized'), which is satisfied because the method exists and raises, so the failure is deferred to the first request instead of startup. Confirmed on mlx-lm 0.31.3 and current main. PR #1584 (RotatingQuantizedKVCache) was open and mergeable at the time of access; a reporter verified the hybrid Gemma 4 case on an M5 Max with mlx 0.32.0 using that branch.
Cross-references
This record references no other records.