Candle (Hugging Face Rust ML framework)
cuda, metal, mkl, accelerate, amd-hip/rocm, wasm, cpu
Fields
id
engines/niche-candleRepo
huggingface/candle
Languages
Rust
License
MIT OR Apache-2.0
First release
2023
Design goal
embedded-inferencelibrary-not-serverminimal-depscross-platformwasm
Scheduling
none
Backends
cudametalmklaccelerateamd-hip/rocmwasmcpu
Formats
safetensorsggml/gguf
Notable features
Candle IS A LIBRARY, NOT A SERVER. There is no HTTP endpoint, no OpenAI-compatible API, no scheduler. It is a Rust tensor/nn crate you embed in your own binary. That is the entire point and the reason it never appears in serving comparisons - it is below the layer where engines are comparedThe framework behind several Hugging Face browser demos shipped as Spaces: whisper speech recognition, LLaMA2, T5 generation, yolo pose estimation and object recognition, and Segment Anything, all running entirely in the browser via WASM. Those demos are the project's most concrete public artifact and its best argument for why the framework existsDual MIT OR Apache-2.0 licensing - the permissive path is deliberate, because the audience includes commercial embedded vendorsCrates.io distributed as candle-core rather than GitHub releases, so versioning lives on crates.io and the GitHub releases list is empty. Worth knowing if you are tracking release cadenceNo GitHub releases at all (API returns an empty releases list as of 2026-10-04); the crates.io version is the release signalApache-2.0, 21140 stars, 1784 forks, 913 open issues, created 2023-06-19, last push 2026-10-02, not archived
Hardware caveats
NO PUBLISHED THROUGHPUT. Candle publishes no tok/s, no TTFT, no benchmark table comparing against llama.cpp, vLLM or SGLang. The Spaces demos demonstrate that models run, not how fast. Treat any Candle speed number you encounter as third-party and unverifiedNO SCHEDULER, so all the techniques this repo is about are absent: no continuous batching, no paged KV, no prefix caching, no chunked prefill. Single-stream only. Candle is strictly a batch-1 embedded runtime and it does not claim to be anything elseNO OPTIMIZED SERIALIZATION PATH COMPARABLE TO GGUF. It reads safetensors natively and has a ggml/gguf reader, but candle is not the place to get the wide quant-format coverage (AWQ, GPTQ, EXL2) that llama.cpp provides. Quantised-model breadth lives in llama.cpp, not hereKernel coverage is the honest limit. Candle implements a working but partial set of matmul and fused ops compared to what CUDA, cuBLAS or Metal ship. Expect to be slower than a specialised runtime on the same model and hardware, and expect occasional unsupported op to send you to writing a custom kernel913 open issues on 21140 stars is a high absolute count - a large surface with real gaps, and a signal that issue response is not fastCPU path depends on MKL or Accelerate being present; both are separate native dependencies and the plain-build experience is not as zero-friction as the README's minimalism claim suggests
Best for
Embedding a model in a Rust product that must not ship a Python runtime or a C++ dependency - the single clearest reason to use itWASM and browser inference, which is where its public demos actually live and where almost nothing else in this repo competesA small Whisper or LLaMA-class model inside an existing Rust binary, where throughput per second matters less than a single dependency and a static buildHugging Face ecosystem projects that want to prototype a model quickly in Rust without leaving the HF hub
Avoid for
Any multi-user or server deployment - there is no server to deployThroughput work. If you care about tokens per second under concurrency, use an engine from data/engines/ that has a schedulerMaximum speed on a single model: candle is a general framework, and a runtime specialised for one architecture will beat itAnyone who needs a quantisation zoo; that is llama.cpp's job
notes
The Rust counterpart to the Rust story in this slice, and the one with the most visible public artifacts (the HF browser Spaces). Recorded as a library rather than an engine because calling it an engine would overstate it: the honest framing is that candle occupies the layer underneath engines and is chosen for integration cost, not throughput. NO PUBLISHED THROUGHPUT FIGURES - explicitly recorded rather than filled with an invented number. Related: [[triton-inference-server]] and [[text-generation-inference]] are Rust servers if you actually need a server; candle is the case where you do not. All metadata from the GitHub API on 2026-10-04. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-02 (0.1 months before this record), NO GitHub release or tag found, 293 contributors (distinct committers including anonymous), open_issues_count 914 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 21141 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
The Rust counterpart to the Rust story in this slice, and the one with the most visible public artifacts (the HF browser Spaces). Recorded as a library rather than an engine because calling it an engine would overstate it: the honest framing is that candle occupies the layer underneath engines and is chosen for integration cost, not throughput. NO PUBLISHED THROUGHPUT FIGURES - explicitly recorded rather than filled with an invented number. Related: [[triton-inference-server]] and [[text-generation-inference]] are Rust servers if you actually need a server; candle is the case where you do not. All metadata from the GitHub API on 2026-10-04. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-02 (0.1 months before this record), NO GitHub release or tag found, 293 contributors (distinct committers including anonymous), open_issues_count 914 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 21141 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.