exo
metal, cpu
Fields
id
engines/niche-exoRepo
exo-explore/exo
Languages
PythonSvelteSwiftTypeScriptRustNix
License
Apache-2.0
First release
2024
Design goal
edge/mobileease-of-useconsumer-hardwareclusteringprivacy
Scheduling
batched-decode
Backends
metalcpu
Formats
safetensorsmlx-quantized
Notable features
The distinctive idea is not an inference technique, it is an interconnect: aggregating a pile of consumer devices into one AI cluster. Automatic device discovery with no manual configuration, then topology-aware automatic model parallelisation that places each layer based on a realtime view of device resourcesRDMA over Thunderbolt 5 shipped day-0, with a claimed 99% latency reduction between devices. This is the technically interesting part: Thunderbolt 5 exposes a coherent-ish fabric that consumer Macs can actually use as a memory networkVendor speedups to treat as claims: up to 1.8x on 2 devices and 3.2x on 4 devices for tensor parallelism. Note these are well below linear, which is what a fabric slower than NVLink would giveDemonstrated at real scale on consumer hardware: the README embeds third-party benchmarks of Qwen3-235B (8-bit), DeepSeek v3.1 671B (8-bit) and Kimi K2 Thinking (native 4-bit) on 4x M3 Ultra Mac Studio with tensor-parallel RDMA, credited to Jeff Geerling's '15 TB VRAM on Mac Studio - RDMA over Thunderbolt 5'Built-in dashboard for cluster management and chat, each device exposing an API and dashboard at localhostApple Silicon / macOS oriented: MLX-based, needs Xcode for the Metal ToolChain, macmon for monitoring. The README even notes a specific friction: 'Homebrew macmon 0.6.1 still crashes on Apple M5'47731 stars in about two years - the highest star count of any engine in this repo, which is a fact about the consumer-local-AI moment rather than about the engineeringv1.0.71 released 2026-04-23, last push 2026-10-03, 949 open issues
Hardware caveats
MAC-ONLY IN PRACTICE. Apple Silicon and macOS are the demonstrated path - MLX, the Xcode Metal ToolChain, macmon. There is no CUDA, no ROCm, no Linux-GPU story in the README, so the cluster mechanism is currently a Mac-cluster mechanism. The README says 'Run from Source (macOS)'Thunderbolt 5 is a hard requirement for the headline feature. Without RDMA over TB5 the aggregation degrades to something much closer to ordinary network or USB-speed transport, and the 99% latency-reduction figure does not applyThe linear-scaling expectation should be abandoned. 1.8x on 2 devices and 3.2x on 4 is the vendor's own number, and it is what a fabric far slower than NVLink looks like. Model sharding across devices also means each device pays full interconnect latency per layerUnified memory is the actual resource being aggregated, and it is not equivalent to VRAM. The 15 TB figure in the third-party link is a sum of system RAM across machines, and it is cache/coherent-memory constrained in ways HBM is notRequires per-device setup: Xcode, the Metal ToolChain, a Nix or Homebrew dependency chain, and macmon for monitoring, with known version-specific breakage (macmon on M5)Consumer-device thermals and power are not addressed in the README. Sustained cluster inference on Macs is a thermal problem before it is a bandwidth problem, and there are no numbersModel formats are MLX-side rather than the GGUF ecosystem people usually have. There is no Triton pathVendored dependencies: the README points at a Cachix binary cache and Nix, which is a deliberate choice but adds an install-model assumption949 open issues on a consumer-facing project with a two-year life. Expect sharp edges
Best for
Running a model too large for one Mac, across several Mac Studios you already own, at zero marginal costPrivate local inference on hardware you already have, where 'local' is the point rather than the fallbackStudying Thunderbolt 5 RDMA as a consumer-memory interconnect - the most interesting networking idea in this sliceThird-party-independent validation of cluster inference: the README cites external benchmark posts rather than only its own numbers
Avoid for
Production or SLA work of any kind; this is a research-and-demo-grade systemAny non-Apple hardware, and any Linux deployment pathExpecting near-linear scaling across devices - the vendor's own figures say otherwiseSystems where thermal throttling under sustained load would be a problem; not addressed in the README
notes
Recorded because it is the clearest current instance of a distinct niche: consumer-device clustering, which no engine in the main list covers. Its value is the interconnect idea (RDMA over Thunderbolt 5 as a model-parallel fabric) and its unusual citation practice of embedding third-party benchmarks. The 47731 stars are documented as a signal about interest in local AI, not about engineering quality. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-08-25 (1.3 months before this record), latest release v1.0.71 published 2026-04-23, 107 contributors (distinct committers including anonymous), open_issues_count 443 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 47745 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
Recorded because it is the clearest current instance of a distinct niche: consumer-device clustering, which no engine in the main list covers. Its value is the interconnect idea (RDMA over Thunderbolt 5 as a model-parallel fabric) and its unusual citation practice of embedding third-party benchmarks. The 47731 stars are documented as a signal about interest in local AI, not about engineering quality. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-08-25 (1.3 months before this record), latest release v1.0.71 published 2026-04-23, 107 contributors (distinct committers including anonymous), open_issues_count 443 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 47745 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.