Design goal
edge/mobileApple-onlylow-latencydeveloper-UX
Scheduling
prefix-cacheprompt-cachebatched-decode
Formats
safetensorsmlx-quantized-4bitmlx-quantized-8bitmlx-quantized-bf16
Notable features
mlx-lm is the LLM package on top of Apple's MLX array framework; Apple Silicon is the only targetRotating fixed-size KV cache (--max-kv-size) to cap memory at some quality costPrompt caching to disk and in memory, reusable as a prefix across turnsConfigurable prefill step size (--prefill-step-size, default 2048) to trade peak memory against prompt speedBatched generation (batch_generate_response, BatchGenerator) plus a threaded OpenAI-compatible server with separate decode and prompt concurrency knobsFirst-party quantization pipeline that converts and uploads models to the mlx-community HF orgDistributed inference and fine-tuning via mx.distributed, with a --pipeline switch for pipeline rather than tensor parallelismSpeculative decoding via --draft-model / --num-draft-tokens
Hardware caveats
Apple Silicon and macOS only - MLX has no CUDA, ROCm, Linux-CPU or Windows path. This is a hard boundary, not a preferencemlx_lm.server exposes batching, but --kv-bits (4 or 8 bit KV cache quantization) explicitly disables batching and serves one request at a time, so enabling KV quantization costs you concurrencyModels much larger than system RAM are slow unless macOS 15+ wired-memory behaviour is in play; the limit is raised with sysctl iogpu.wired_limit_mb, and without it you get a [WARNING] that the model will likely be slowAdapters (LoRA) and draft models cannot be combined with distributed mode - the loader raises on that combinationDistributed mode also loses adapter support entirely, and the prompt cache and quantized-KV paths are separate features from batchingLarge-model speed relies on MLX wiring model and cache memory outside the normal allocator, which needs macOS 15 or newer
Best for
Local Mac development and personal use, where Metal is fast and battery/thermals matterQuantized models from mlx-community that were converted specifically for MLX, since those are the tuned pathFine-tuning or LoRA work on Apple Silicon, which mlx-lm supports natively including on quantized models
Avoid for
Any shared or multi-tenant server - you want vLLM-Metal (see [[vllm-metal]]) if you need paged KV and real continuous batching on a MacNon-Apple hardware, without exceptionWorkloads needing KV-cache quantization and request concurrency at the same time
notes
'prefix-cache' here is mlx-lm's prompt cache (an LRU of saved prompt KV plus explicit prompt-cache files), not a radix tree and not vLLM's block-level automatic prefix caching. mlx-lm's own quantized formats (4-bit, 8-bit, and bf16) are distinct from GGUF and from AWQ/GPTQ; safetensors is the on-disk container. The batching flags --decode-concurrency (default 32) and --prompt-concurrency (default 8) come from reading mlx_lm/server.py directly. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-02 (0.1 months before this record), latest release v0.31.3 published 2026-04-22, 254 contributors (distinct committers including anonymous), open_issues_count 228 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 7217 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.