Repo
EricLBuehler/mistral.rs
Design goal
minimal-depsmulti-modelsingle-binaryagentic-workloads
Scheduling
continuous-batchingpaged-kvprefix-cache
Backends
cudametalcpu-avx512cpu-avx2
Formats
safetensorsggufgptquqffisq
Notable features
Rust engine built on Hugging Face Candle, shipping as a single prebuilt binary with a Rust crate and a Python packageServes both OpenAI-compatible /v1 and Anthropic Messages endpoints from one process, with a built-in web UI at /uiFull multimodal in one engine: text, vision, video, audio, speech generation, image generation and embeddingsIts own UQFF quantization format plus ISQ (in-situ quantization), plus GGUF and GPTQ loadingMulti-GPU via NCCL tensor parallelism with P2P layer mapping, plus multi-node and ring attentionBuilt-in agentic runtime: web search, local Python and shell execution, MCP integration, OpenAI-compatible Skills bundlesmistralrs tune recommends quantization and device mapping from the model config and detected hardware; mistralrs doctor diagnoses CUDA/Metal issues
Hardware caveats
CUDA is the only accelerated Linux path; the prebuilt binary matrix is Metal on Apple Silicon, per-GPU CUDA or CPU on Linux, and CPU-only on Windows. Windows therefore has no GPU acceleration out of the boxOptional cuTile acceleration requires NVIDIA's separately installed tileiras tool, so the fastest CUDA path is not self-containedAccelerator-specific Python wheels (CUDA, Metal, MKL, Accelerate) are separate artifacts from the main package, so an install can silently land without accelerationIts published BF16 prefill/decode comparisons against vLLM are self-reported and mixed: it wins on Gemma 4 E4B across GB10/B200/H100 but loses badly on the 26B-A4B MoE at prefill (e.g. 2766 vs 26295 tok/s on H100 SXM), so do not generalise from the small dense modelThe supported-models reference is generated from the loader registry, so coverage is exactly what that registry lists - 45+ architectures as claimed, with no fallback for the rest
Best for
A single Rust binary with OpenAI plus Anthropic API parity and a built-in web UIMixed text/vision/video/audio workloads in one processEmbedders that want an inference engine as a library with both Rust and Python surfaces
Avoid for
Multi-node MoE at scale, where the published numbers trail vLLM substantiallyWindows GPU inference, which the prebuilt matrix covers only on CPUAnyone needing a comparable, third-party-measured benchmark record for this engine
notes
Benchmark figures in the README are project self-reported for v0.8.2 (mean tok/s, decode 256 generated tokens, prompt lengths 128-16384) and were not independently reproduced. 'isq' is the project's own in-situ quantization; 'uqff' is its UQFF format. Block-diffusion text generation (DiffusionGemma) is integrated with paged attention and prefix caching per the README. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-01 (0.1 months before this record), latest release v0.9.4 published 2026-09-24, 97 contributors (distinct committers including anonymous), open_issues_count 400 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 7733 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.