id
engines/niche-lmdeploy-turbomind
Design goal
throughput-firstlow-latencycompressionmulti-modelresearch
Scheduling
continuous-batchingpaged-kvprefix-cacheautomatic-prefix-caching
Formats
safetensorsawqw4a16mxfp4kv-quantized
Notable features
Two engines behind one toolkit: TurboMind (C++/CUDA) and a pure-Python PyTorchEngine that the README offers explicitly to lower the barrier to rapid experimentation. That pairing is unusual and usefulDeveloped by the MMRazor/MMDeploy team, so it inherits an OpenMMLab lineage and a compression-first identity that most serving frameworks do not havePersistent batch (their term for continuous batching), blocked KV cache, automatic prefix cachingCompressed weights: AWQ W4A16 with a claimed 2.4x over FP16, plus online int8/int4 KV-cache quantization added April 2024MXFP4 support on NVIDIA GPUs from V100 upward, and llm-compressor 4-bit symmetric/asymmetric support added February 2026A request-distribution service for multi-model, multi-machine, multi-card serving - a proxy layer rather than a single-process serverModel zoo is InternLM-centric but covers Llama, Qwen (including MoE variants), InternVL2, GLM, Gemma, DeepSeek and multimodal VLMsTurboMind got the EuroSys 2027 paper (arXiv:2508.15601), which is unusual for a serving framework and indicates real systems research8106 stars, v0.18.0 released 2026-09-28, last push 2026-09-28, 2271 open issues
Hardware caveats
ALL PERFORMANCE FIGURES IN THE README ARE VENDOR CLAIMS with no methodology. Read them as positioning: 'up to 1.8x higher request throughput than vLLM'; 4-bit '2.4x higher than FP16'; MXFP4 '1.5x the performance of vLLM on H800 for openai gpt-oss models'; the April 2024 GQA upgrade taking InternLM2-20B to 16+ RPS, 'about 1.8x faster than vLLM'. Every one names a different model, so none of them compose into a general statement about the engineThe vLLM comparisons are the ones to distrust least and still not trust: vLLM tunes itself against the same popular models, so a per-model win usually means a configuration winEffectively CUDA-first. The README's own backends list is CUDA/ROCm/CPU/Mac in the sense of an aspiration; the substantive work - MXFP4 from V100, AWQ, KV-cache quantization, CUDA graphs - is NVIDIA-specific. No NPU, no Metal-accelerated generation path, no VulkanVersion line is still v0.x at v0.18.0 despite being five years old, which is the honest maturity signal2271 open issues is the largest backlog in this slice. Part of that is the InternLM model zoo's own churnWindows support is documented only at tp=1 (from the 2023/08 news item) - single-GPU onlyModel coverage skews InternLM and Qwen. Outside that zoo the compatibility burden is realTwo engines means two behaviour sets: a bug or a perf difference may be TurboMind-specific or PyTorchEngine-specific
Best for
Deployment if you are in the InternLM/Qwen ecosystem, where this is the best-tuned option and the model support is first-classCompression-forward serving: AWQ, MXFP4 and KV-cache quantization are supported together here in a way most engines do not coordinateRunning a proxy in front of many models across many machines, via the request-distribution serviceReading a TurboMind paper (EuroSys 2027) on mixed-precision inference - one of very few serving frameworks with a peer-reviewed systems paperFast experimentation with the pure-Python PyTorchEngine when you do not want a C++ build
Avoid for
Deciding on vendor speed claims - there is no methodology and the model changes with every claimNon-NVIDIA hardware. This is a CUDA-first engine in practiceMulti-GPU on Windows (tp=1 only)Production without reading the support matrix; v0.18.0 with 2271 open issues is a moving target
notes
The strongest niche candidate in this slice that is neither dead nor a research prototype, and a genuinely distinct third way between vLLM and llama.cpp: a compression-first C++ engine from the OpenMMLab lineage with a EuroSys paper. TurboMind paper arXiv:2508.15601 (2025-08-21) verified via the arXiv API. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-28 (0.2 months), latest release v0.18.0 published 2026-09-28 - same day as the last commit, which for a project of this size is a strong liveness signal - 157 contributors, open_issues_count 610 (issues PLUS pull requests), NOT archived, 8106 stars. DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - see [[ev-production-evidence-is-scarce-and-nearly-always-single-site]]. FEATURE AXES, four filled from dated README entries, each qualified where the qualification matters. DISAGGREGATED P/D - YES but VIA INTEGRATION, not in-tree: '[2025/06] DeepSeek PD Disaggregation deployment is now supported through integration with DLSlime and Mooncake'. Read that precisely: the capability is reached through two other projects' integration, which is the same distinction this repo applies to vLLM's third-party disaggregation connectors, and it should not be recorded as an in-tree scheduler. MULTI-MODEL - '[2024/01] Support for multi-model, multi-machine, multi-card inference services' with a proxy_server doc and an 'Effortless Distribution Server' feature bullet. MULTIMODAL - '[2023/12] Turbomind supports multimodal input', plus InternVL2 full-series and XComposer2.5 (2024/07), Phi-3 and Phi-3.5-vision in the supported list, and separate VLM pipeline and VLM serving docs. PREFIX CACHING + KV CACHE QUANT compose: 'LMDeploy supports KV Cache Quant, AWQ and Automatic Prefix Caching to be used simultaneously' - a useful conjunction claim, since most engines list these separately and the composition is the thing that actually matters in a config. SPECULATIVE DECODING and CHUNKED PREFILL: no mention in the README's feature bullets or roadmap lines, so both are UNSTATED. tracing_support=none, verified_negatively=true (set 2026-10-04). What was searched: an in-repo search for opentelemetry in InternLM/lmdeploy returns 0 hits. The same search family also returned nothing for preempt in lmdeploy/metrics/loggers.py, which is how the missing preemption metric was established (obs-lmdeploy-metrics-loggers, obs-lmdeploy-metrics-doc, obs-lmdeploy-monitoring-dir). Turbomind is a backend inside that repo, so the absence covers it.