LMDeploy GitHub repository

LMDeploy README with TurboMind (C++/CUDA) and PyTorchEngine backends. Vendor claims to record as claims, not measurements: 'up to 1.8x higher request throughput than vLLM'; 4-bit inference '2.4x higher than FP16'; MXFP4 on NVIDIA from V100 up with '1.5x the performance of vLLM on H800 for openai gpt-oss models'; the 2024/04 GQA upgrade taking InternLM2-20B to 16+ RPS, 'about 1.8x faster than vLLM'. Also documents persistent (continuous) batching, blocked KV cache, AWQ, KV-cache quantization, automatic prefix caching, a request-distribution proxy for multi-model multi-machine serving, and llm-compressor 4-bit support (2026/02). API metadata 2026-10-03: created 2023-06-15, last push 2026-09-28, v0.18.0 (2026-09-28), Apache-2.0, 2271 open issues, 8106 stars.