vllm.cpp
cuda, metal, vulkan, cpu-avx512
Fields
id
engines/vllm-cppRepo
mudler/vllm.cpp
Languages
C++CUDAMetalVulkanSPIR-V
License
Apache-2.0
First release
2026
Design goal
minimal-depsno-Pythonthroughput-firstsingle-binary
Scheduling
continuous-batchingpaged-kvradix-cacheprefix-cache
Backends
cudametalvulkancpu-avx512
Formats
ggufsafetensorsexl3mxfp4nvfp4
Notable features
Ports vLLM's serving core - continuous batching, block-paged KV, the V1 scheduler, automatic prefix caching, speculative decoding - into from-scratch C++20 with no Python and no PyTorch at inference timeShips SGLang's scheduling ideas as documented toggles: RadixAttention, LPM cache-aware admission, jump-forward decodingDoes NOT use ggml: it reads GGUF and follows llama.cpp's C ABI style but has its own portable vt:: tensor runtimeVersioned flat C ABI (include/vllm.h, VLLM_ABI_VERSION) with append-only fields, so embedders can rely on itClaims token-for-token identical output to vLLM as its correctness gate, plus prebuilt CPU/CUDA/Vulkan/Metal/MLX release archives
Hardware caveats
Under heavy development by its own README: internals, CLI flags and server behaviour can change between commits, so tracking main means expecting breakageIndependent and unofficial - not affiliated with, endorsed by, or sponsored by the vLLM project, the PyTorch Foundation or the Linux FoundationKernel coverage is uneven by format: TQ1_0/TQ2_0 GGUF loading is unsupported even though Vulkan has ternary matmul and fused-MoE kernels for themAMD path is brand new: EXL3 gained a native ROCm path validated only on gfx1151 with no CPU fallbacks; discrete AMD validation and competitive performance are explicitly unmeasuredSeveral architectures are token-gated but not speed-gated (GLM-5.3's real 201.83 GiB artifact has never completed a load), and multimodal HTTP input has CPU tests with synthetic weights with real-checkpoint parity unverifiedIts performance claims are self-reported by the project against its own vLLM oracle on one GB10 machine; the concurrency sweep from c2 to c32 is inside the stated 0.5% noise band, and one headline comparison is marked SUPERSEDED in its own docs
Best for
Deploying a single binary with no Python runtime where vLLM-equivalent scheduling is still wantedReproducing vLLM scheduling semantics token-for-token outside the PyTorch ecosystemConstraints where a 66 MiB binary versus a multi-gigabyte Python install is the deciding factor
Avoid for
Production deployments that need a stable API surface - the project says so itselfAMD GPUs, where the path exists but is essentially unvalidatedAny claim of measured superiority over vLLM: the published comparison is one machine, one model, self-measured, and mostly inside its own noise band
notes
Benchmark numbers in the README (Qwen3.6-27B NVFP4 on NVIDIA GB10: 86.05 tok/s at c1 rising to 1095.01 tok/s at c32 vs vLLM's 82.32 to 1076.25; 45.45 vs 41.94 tok/s on Qwen3-8B MXFP4) are vendor self-reported and were NOT independently reproduced here. Recorded for traceability only; no benchmark record was created because this slice does not own benchmarks/. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v0.0.2 published 2026-08-11, 22 contributors (distinct committers including anonymous), open_issues_count 375 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 443 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
Benchmark numbers in the README (Qwen3.6-27B NVFP4 on NVIDIA GB10: 86.05 tok/s at c1 rising to 1095.01 tok/s at c32 vs vLLM's 82.32 to 1076.25; 45.45 vs 41.94 tok/s on Qwen3-8B MXFP4) are vendor self-reported and were NOT independently reproduced here. Recorded for traceability only; no benchmark record was created because this slice does not own benchmarks/. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v0.0.2 published 2026-08-11, 22 contributors (distinct committers including anonymous), open_issues_count 375 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 443 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.