id
engines/local-onnxruntime-mobile
Repo
microsoft/onnxruntime
Languages
C++CJavaKotlinObjective-CSwiftPython
Design goal
edge/mobileminimal-depslow-latencyembedded
Backends
cpunnapicoremlxnnpackqnn
Notable features
ONNX Runtime Mobile is the mobile/embedded deployment path for ONNX Runtime, optimized for Android and iOSSupports ONNX format models with quantization support (int8, int4, uint8)Execution Providers: NNAPI (Android), CoreML (iOS), XNNPack (CPU), QNN (Qualcomm)Small binary footprint compared to full ONNX Runtime - designed for app bundlingSupports LLM models via ONNX Runtime GenAI API (2024+) with prefill/decode loopsNo dependency on PyTorch or TensorFlow at runtime - self-contained C++ runtime
Hardware caveats
ONNX format requires conversion from PyTorch/TensorFlow - no direct GGUF or safetensors supportLLM support is newer and less mature than llama.cpp or MLX - fewer optimized kernels for transformer architecturesNNAPI and CoreML delegation depends on OS version and device capabilities - not all ops are delegatedNo continuous batching or paged KV cache - single-stream inference onlyQuantization quality depends on the calibration data used during ONNX quantization
Best for
Deploying ONNX models to Android and iOS apps with a single runtimeCross-platform mobile inference where ONNX is the interchange formatQualcomm devices with QNN backend for NPU accelerationiOS devices with CoreML delegation for Neural Engine
Avoid for
Desktop or server inference - use full ONNX Runtime or vLLM insteadModels only available in GGUF format - conversion to ONNX is non-trivialHigh-concurrency serving - no batching or scheduling supportMaximum performance on Apple Silicon - MLX is faster on Metal
notes
ONNX Runtime Mobile is the standard path for deploying ONNX models to mobile devices. It is distinct from full ONNX Runtime in that it strips unused ops and backends to reduce binary size. The GenAI API (2024+) adds LLM-specific optimizations but the ecosystem is less mature than llama.cpp for local LLM inference. The key advantage is ONNX format interoperability - models trained in PyTorch or TensorFlow can be exported to ONNX and deployed without framework-specific runtimes. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v1.30.0 published 2026-09-10, 1004 contributors (distinct committers including anonymous), open_issues_count 1770 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 22007 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.