llama.cpp

id

engines/llama-cpp

Repo

ggml-org/llama.cpp

Languages

License

MIT

First release

2023

Design goal

Scheduling

Backends

Formats

Notable features

Plain C/C++ with no dependencies, built on the standalone ggml tensor libraryggml is the widest-portability backend set in any LLM engine: CUDA, HIP, MUSA, Metal, SYCL, Vulkan, CANN, OpenCL, WebGPU, OpenVINO (in progress), Hexagon, ZenDNN, RPC, plus CPU NEON/AVX/AVX512/AMX/RVVQuantization at 1.5, 2, 3, 4, 5, 6 and 8 bits, plus EXL2/EXL3 and NVFP4/MXFP4CPU+GPU hybrid offload via --n-gpu-layers, including splitting MoE expert weights or dense FFN weights to CPU (--cpu-moe, --n-cpu-moe, --n-cpu-ffn)llama-server does continuous batching with slot-based parallel decoding and an OpenAI-compatible APIMultiple GPU backends can be compiled into one binary and selected at runtime with --device; backends can also be loaded as shared libraries via GGML_BACKEND_DLUnified-memory mode (GGML_CUDA_ENABLE_UNIFIED_MEMORY=1) swaps to system RAM instead of crashing when VRAM is exhaustedSpeculative decoding with ELEVEN distinct algorithms across three families - draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark, and five n-gram cache-keying variants (ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache) - the widest method menu of any engine in this repoConstrained generation via both --grammar (BNF-like, with samples in the grammars/ directory) and --json-schema / --json-schema-filePrefix-cache reuse without a named feature, through four separate mechanisms: a shared unified KV buffer, slot-prompt-similarity matching, idle-slot prompt-cache persistence, and up to 32 per-slot context checkpointsMultimodal with the most complete flag set in the repo: --mmproj, --mmproj-url, --mmproj-auto, --mmproj-offload, --mmproj-device, plus --image-min-tokens/--image-max-tokens for dynamic-resolution vision models; mmproj is downloaded automatically with --hf-repo unless --no-mmproj

Hardware caveats

Vulkan is the universal escape hatch but is not the fast path: it is the only route to a GPU with no vendor SDK, and on Windows it needs the LunarG Vulkan SDK plus w64devkit/MSYS2 plumbingHIP requires you to name GPU_TARGETS at build time (e.g. -DGPU_TARGETS=gfx1100) or the build compiles for every GPU present; unsupported gfx versions need HSA_OVERRIDE_GFX_VERSION, which AMD does not support on WindowsNVFP4 and MXFP4 layers marked W4A16 request 8-bit activations, so on Blackwell they silently run the W4A8 path rather than native W4A4. GGML_CUDA_MMQ_PREC=q4 forces native W4A4 at an accuracy costMMQ (the custom quantized matmul kernel) is off by default on GPUs without int8 tensor cores - that is V100, CDNA and RDNA3+ - so quantized models on those parts fall back to slower paths unless GGML_CUDA_FORCE_MMQ is setCUDA non-native builds (-DGGML_NATIVE=OFF) cover all CUDA GPUs but compile slowly and may still need JIT; otherwise nvcc silently defaults arch when it cannot detect your GPUOn Windows on ARM, MSVC compiles the ARM NEON intrinsics incompletely, so a CUDA build made entirely with MSVC has a slower CPU backend; the release workflow splits the build between clang (CPU) and MSVC (CUDA)OpenVINO backend is explicitly marked In Progress for Intel CPUs, GPUs and NPUs; SYCL is the maintained Intel GPU pathThe SYCL note also warns that the Intel oneMKL/oneAPI CPU build config does not support Intel GPU - use the SYCL path for thatUnified memory hurts performance on non-integrated GPUs while being what makes integrated GPUs usable at allNO DISAGGREGATED PREFILL/DECODE MODE and NO CHUNKED-PREFILL KNOB in the llama-server surface - these two axes are an affirmative NO for this engine as of 2026-10-04, read from tools/server/README.md, not merely unstated. This is the price of a single-process single-slot-oriented design and is worth stating plainly rather than leaving blankA DOCUMENTED CASE OF ITS OWN THROUGHPUT BEING MISMEASURED BY 10x: a persistent drafter context is reused across requests on a slot and the replay path 'never runs the draft model and never applies params.p_min', so a repeated-prompt benchmark protocol reads 9.5x-16.9x higher tok/s than varied-prompt traffic at identical settings (issue #26100, releases b8400/b8407, Qwen3-8B Q4_K_M + Qwen3-0.6B Q8_0). The maintainer-facing one-liner is the rule this whole repo rests on: 'Any benchmark that warms up and times with the same prompt (a very common pattern) measures the replay path, not speculative decoding.' The production face of the same bug is worse - acceptance 0.60 -> 0.12 and latency ~3x under long-shared-prefix chat traffic - and it is intermittent, since 'plain or streaming traffic mixes that do NOT share long prefixes never trigger it'

Best for

Avoid for

Tracing support

none

Verified negatives

yes

Docs

https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md

notes

Scheduling: llama-server does implement continuous batching with slot-based parallel decoding, and the server exposes prompt-cache reuse for shared prefixes, but there is no paged-KV block manager in the vLLM sense - KV is allocated per slot from a preallocated pool. 'exl2'/'exl3' are recorded from the project's published support for those quant families; the GGUF variants (Q4_K_M, IQ2_XS, IQ4_XS, UD-Q2_K_XL, TQ1_0/TQ2_0 ternary) are the primary path. safetensors support exists via converters rather than as a native load path. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0 months), latest release v0.5.0 published 2026-09-23, 2077 contributors (distinct committers INCLUDING ANONYMOUS - this project's count is inflated by anonymous contributors and should not be read as a team size), open_issues_count 2537 (issues PLUS pull requests), NOT archived, 130298 stars - the highest star count of any engine in this repo. Caveat worth carrying: issue #26100's reporter notes llama.cpp's releases are named by BUILD (b8400, b8407), not semver, so 'v0.5.0' from the API is a version-5.0 series release rather than a build number, and build-level pinning means reproducing a specific report requires its b-number. tracing_support=none, verified_negatively=true (set 2026-10-04). What was searched: an in-repo code search for opentelemetry in ggml-org/llama.cpp returns 0 hits; the only observability surface is the hand-built /metrics endpoint assembled by string concatenation in server_task_result_metrics::to_metrics(), which contains no span or exporter code (obs-llamacpp-metrics-source, obs-llamacpp-server-readme-metrics). The absence is repo-level, so it covers this record as well as obs-llama-cpp-observability.

Fields

Notes

Cross-references