inference/research
verifiedhigh · 0.88updated 2026-10-03

MLC-LLM

cuda, rocm, vulkan, metal, opencl, webgpu, wasm

Fields

id
engines/mlc-llm
Repo
mlc-ai/mlc-llm
Languages
C++PythonTypeScriptRust
License
Apache-2.0
First release
2023
Design goal
compiler-basedcross-platformedge/mobileminimal-deps
Scheduling
continuous-batchingprefix-cache
Backends
cudarocmvulkanmetalopenclwebgpuwasm
Formats
safetensorsmlc-compiledq4f16q4f32
Notable features
An ML compiler (TVM/Relax + TensorIR) ahead of the runtime, so a model is compiled to a per-target artifact rather than JIT-kernel-specialised at serve timeOne MLCEngine behind Python, REST, JavaScript/WebGPU, iOS and Android, all with an OpenAI-compatible APIPlatform matrix is genuinely broad: CUDA, ROCm, Vulkan, Metal, OpenCL, WebGPU/WASM, OpenVINO-freePublished pre-compiled model artifacts on Hugging Face under the HF://mlc-ai/...-MLC naming, so no local compile step is required to startWebLLM is the same compiler targeting the browser
Hardware caveats
Models must be compiled per target; loading a prebuilt artifact on an unsupported target means going through the compiler yourselfOn Windows and Linux the Vulkan path requires a working Vulkan loader - the docs tell you to install libvulkan-loader via conda to avoid 'vulkan not found' failuresHardware coverage is uneven in practice: the README's matrix marks AMD/NVIDIA Vulkan and CUDA, Intel Vulkan, Apple and AMD Metal on macOS, WebGPU/WASM in browsers, Metal on iOS, and OpenCL on Adreno/Mali for Android - but the rows mix targets, so 'supported' means a build exists, not that performance is competitiveThe MLX-style compile step means model updates require recompiling an artifact; you cannot just point at a new safetensors checkpoint and serveThroughput and features trail the CUDA-specialised engines on datacenter NVIDIA parts, which is the usual reason it is not chosen there
Best for
Deploying one model to many target classes - browser, phone, iOS, Mac, Windows/Linux GPU - from a single compile pipelineIn-browser and on-device inference with an OpenAI-compatible API and no serverIntel, Adreno, Mali or other GPUs without a vendor-tuned inference stack
Avoid for
Maximum datacenter throughput on NVIDIA, where a purpose-built CUDA engine winsRapid model iteration, where the compile artifact is a step between a new checkpoint and a running serverAnyone who does not want a compiler in the loop
notes
'mlc-compiled' is MLC's own artifact format; q4f16/q4f32 are the published quantization variants (group-quantized weights with f16 or f32 scales). WebLLM (mlc-ai/web-llm) is the browser deployment of the same compiler and is recorded here as part of MLC-LLM rather than as a separate engine record. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0 months), NO RELEASE OBJECT EXISTS in the GitHub API (the tags endpoint returns a list but no releases/latest, so release publication dates are unverified for this project), 164 contributors, open_issues_count 349 (issues PLUS pull requests), NOT archived, 23205 stars - a high star count with no release history, which is a distinctive maintenance shape. DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - see [[ev-production-evidence-is-scarce-and-nearly-always-single-site]]. Note that this is a weaker negative than for a small library: the README publishes an offline batch-inference throughput TUNING GUIDE with a worked example (a log line showing 'Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317'), which is evidence that operators tune this engine seriously but is explicitly a configuration guide, not a report of anyone's deployment. FEATURE AXES - THIS IS A RECORDED NEGATIVE AND IT IS A NEGATIVE ABOUT THE SOURCE, NOT THE ENGINE: a case-insensitive scan of the README for speculat|eagle|mtp|grammar|json.schema|structured|chunked|prefix cach|radix|disagg|multi.model|multimodal|vision|audio returned ZERO matches. Speculative decoding, chunked prefill, disaggregated P/D and structured output are therefore UNSTATED-AND-UNVERIFIED from this source, and prefix caching - which the record asserts - is unconfirmed rather than contradicted. The README is a short project overview; the authoritative per-model surface is the model library, which was NOT enumerated, so absence of evidence here is not evidence of absence and none of these axes should be recorded as NO on the strength of this. ONE DISTINCTION TO PRESERVE: the record's existing claim of one engine behind Python/REST/JavaScript/iOS/Android is about serving SURFACES, not about serving several models at once, so it is not an answer on the multi-model axis. BUILD COST: no published figure found for MLC LLM's model compilation, which is a per-model artifact build by design and would be the cost to record; this record does not estimate it.

Notes

'mlc-compiled' is MLC's own artifact format; q4f16/q4f32 are the published quantization variants (group-quantized weights with f16 or f32 scales). WebLLM (mlc-ai/web-llm) is the browser deployment of the same compiler and is recorded here as part of MLC-LLM rather than as a separate engine record. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0 months), NO RELEASE OBJECT EXISTS in the GitHub API (the tags endpoint returns a list but no releases/latest, so release publication dates are unverified for this project), 164 contributors, open_issues_count 349 (issues PLUS pull requests), NOT archived, 23205 stars - a high star count with no release history, which is a distinctive maintenance shape. DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - see [[ev-production-evidence-is-scarce-and-nearly-always-single-site]]. Note that this is a weaker negative than for a small library: the README publishes an offline batch-inference throughput TUNING GUIDE with a worked example (a log line showing 'Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317'), which is evidence that operators tune this engine seriously but is explicitly a configuration guide, not a report of anyone's deployment. FEATURE AXES - THIS IS A RECORDED NEGATIVE AND IT IS A NEGATIVE ABOUT THE SOURCE, NOT THE ENGINE: a case-insensitive scan of the README for speculat|eagle|mtp|grammar|json.schema|structured|chunked|prefix cach|radix|disagg|multi.model|multimodal|vision|audio returned ZERO matches. Speculative decoding, chunked prefill, disaggregated P/D and structured output are therefore UNSTATED-AND-UNVERIFIED from this source, and prefix caching - which the record asserts - is unconfirmed rather than contradicted. The README is a short project overview; the authoritative per-model surface is the model library, which was NOT enumerated, so absence of evidence here is not evidence of absence and none of these axes should be recorded as NO on the strength of this. ONE DISTINCTION TO PRESERVE: the record's existing claim of one engine behind Python/REST/JavaScript/iOS/Android is about serving SURFACES, not about serving several models at once, so it is not an answer on the multi-model axis. BUILD COST: no published figure found for MLC LLM's model compilation, which is a per-model artifact build by design and would be the cost to record; this record does not estimate it.

Cross-references

This record references no other records — it is a leaf. It is cited by 2.