Inference engines (119)

What each engine supports where. Read hardware_caveats and avoid_for before the README: in this repo the exclusions are better evidenced than the inclusions, and they are what actually bites.

contested 1draft 10verified 98deprecated 10→ sources (2562)

Filter

license

status

sort

RouteLLM (LMSYS LLM router)

api-only (proxies to external providers)A workload with a long tail of easy prompts where a router can send most traffic to a small model - this is the canonical use case and the reason the project has 5568 stars; Cutting inference spend without a quality collapse you can measure against your own eval set; Teams wantin…engines/niche-routellm

contestedmedium · 0.72

AirLLM

cuda, cpu, metalAnswering 'can this model run on this GPU' when the answer is technically yes but practically miserable - a 70B on a 4GB card is a real capability that no other engine in this repo offers; Demonstrating the layer-streaming idea, which is the correct mental model for why MoE exper…engines/niche-airllm

draftlow · 0.55

Cerebras CSL SDK and fabric simulator (Cerebras' own software stack)

cerebras-wseUnderstanding what wafer-scale programming demands: a language where layout and data movement are explicit, because the machine is an addressable mesh rather than a set of schedulable SMs; Studying the simulator-first development pattern - compile-then-run on a fabric simulator w…engines/alt-cerebras-csl-sdk

draftmedium · 0.72

GroqFlow and the GroqWare Suite (Groq's compile-and-run tool flow)

groq-lpuUnderstanding what a fully static compilation model does to a serving stack when the constraint is taken seriously - compile and runtime as separate products, static allocation, no dynamic batching; Single-stream low-latency decode where determinism and predictable variance matte…engines/alt-groq-groqflow

draftmedium · 0.72

mistral.rs

cuda, metal, cpu-avx512, cpu-avx2A single Rust binary with OpenAI plus Anthropic API parity and a built-in web UI; Mixed text/vision/video/audio workloads in one process; Embedders that want an inference engine as a library with both Rust and Python surfacesengines/mistral-rs

draftmedium · 0.68

Qualcomm AI Engine Direct / Apps SDK (AIC100 backend)

no backends listedengines/qualcomm-ai-engine-direct-apps-sdk-aic100-backend

draftlow · 0.50

SambaFlow (SambaNova RDU runtime)

npuvendor-benchmark rows where SambaNova publishes aggregate tok/s through its own stackengines/sambaflow-sambanova-rdu-runtime

draftlow · 0.50

Sequoia (tree-based speculative decoding) and UMbreLLa successor

cudaUnderstanding tree-structured speculative decoding and hardware-aware draft-tree optimisation - the per-hardware growmap concept is the idea worth taking, and it is unusual enough that the codebase remains a useful reference; Reproducing the paper's acceptance-rate measurements, …engines/niche-sequoia

draftmedium · 0.72

SpecExec (Massively Parallel Speculative Execution)

cudaRunning 70B+ LLMs interactively on consumer GPUs (16GB-24GB VRAM) with RAM offloading; Understanding massive draft-tree construction (budgets of 256-2048 tokens) when target model forward pass latency is measured in seconds; Lossless speculative sampling on offloaded setupstracing: noneverified negativesengines/niche-specexec

draftmedium · 0.80

VibeSys bespoke serving engine (agent-built)

rocm, hipsingle-deployment peak-goodput serving where agents write the engineengines/vibesys-bespoke-engine

draftlow · 0.50

vllm.cpp

cuda, metal, vulkan, cpu-avx512Deploying a single binary with no Python runtime where vLLM-equivalent scheduling is still wanted; Reproducing vLLM scheduling semantics token-for-token outside the PyTorch ecosystem; Constraints where a 66 MiB binary versus a multi-gigabyte Python install is the deciding factorengines/vllm-cpp

draftlow · 0.60

Burn (Tracel Rust deep learning framework)

cuda, rocm/hip, metal, vulkan, wasm, webgpu, cpuA team that trains a model in Rust and wants to ship that exact model, particularly on-device personalization or federated learning where an export round-trip is unacceptable; Portability across CPU/CUDA/ROCm/Metal/Vulkan/WASM/WebGPU from one codebase, including a web target; Any…engines/niche-burn

verifiedmedium · 0.76

Candle (Hugging Face Rust ML framework)

cuda, metal, mkl, accelerate, amd-hip/rocm, wasm, cpuEmbedding a model in a Rust product that must not ship a Python runtime or a C++ dependency - the single clearest reason to use it; WASM and browser inference, which is where its public demos actually live and where almost nothing else in this repo competes; A small Whisper or LL…engines/niche-candle

verifiedmedium · 0.78

CLIP-as-service

cuda, tensorrt, onnxruntime, pytorchZero-shot image/text retrieval and cross-modal ranking where you want a service rather than a library.; Multimodal RAG over image corpora (clip-as-service is what Jina's own image-search stack used).; Studying the bi-encoder (embed) vs cross-encoder (rank) split for CLIP, since b…engines/clip-as-service

verifiedlow · 0.62

ColBERT / PLAID late-interaction retrieval

cuda, cpuRetrieval quality above what single-vector embeddings deliver, when recall on hard queries matters more than index size or query latency; Building or reading a late-interaction index, and understanding why MaxSim scoring costs what it costs; Studying PLAID as the template for mak…engines/niche-colbert

verifiedhigh · 0.88

ColBERT reference implementation

cuda, cpuUnderstanding late interaction properly - this is the code that defines the MaxSim operator and the index structure.; Building a custom late-interaction retrieval system, or reimplementing candidate generation (PLAID-style) for your own corpus.; Benchmarking late interaction agai…engines/colbert-reference

verifiedmedium · 0.72

Core ML and coremltools (Apple's on-device model format and converter)

metal, cpu-avx512, cpu-avx2, npuDeploying small or medium models into an Apple app where CPU, GPU and Neural Engine can be mixed and power matters more than throughput; Classical and traditional ML on-device - the scikit-learn, XGBoost and LibSVM support is the honest signal of Core ML's centre of gravity; Unde…engines/apple-coreml

verifiedmedium · 0.78

DistServe (research prototype)

cudaStudying the disaggregation cost model: the paper is the cleanest statement of why colocating the phases couples two different bottlenecks, and of why placement must be bandwidth-aware; Reading rather than deploying - the OSDI 2024 goodput framing (max request rate under joint TT…engines/distserve

verifiedhigh · 0.85

DwarfStar

metal, rocmengines/dwarfstar

verifiedlow · 0.60

EAGLE (EAGLE-1/2/3 speculative decoding)

cudaReading the reference implementation of the speculative-decoding technique that the field actually standardised on; Understanding why feature-level drafting beats token-level drafting, and what training-time test changes about the loss surface; Training EAGLE-3 heads with SpecFor…engines/niche-eagle

verifiedmedium · 0.80

EleutherAI lm-evaluation-harness

nonecomparable downstream accuracy numbers across quantization formatsengines/lm-eval-harness

verifiedhigh · 0.85

Envoy AI Gateway (now Agent Router)

none (gateway layer - L7 proxy plus an external-processing extension; does not load weights)Kubernetes-native serving wanting model-aware routing, per-token quota and observability in one hop; Clusters already running Gateway API, Envoy Gateway, kgateway or GKE Gateway - the ext-proc integration is designed for exactly these; Serving llm-d / KServe LLMInferenceService t…tracing: per-requestengines/envoy-ai-gateway

verifiedhigh · 0.91

ExecuTorch

cpu, coreml, nnapi, qnn, vulkan, xnnpackPyTorch users who want to deploy models on-device without leaving the PyTorch ecosystem; Android apps using NNAPI or QNN for hardware acceleration; iOS apps using CoreML for Neural Engine acceleration; Models already in PyTorch format that need mobile deploymentengines/local-executorch

verifiedhigh · 0.85

ExLlamaV2

cudaGetting a very large model into a single consumer NVIDIA GPU card — EXL2's variable-bpwinth trick is the whole point.; Single-user local interactive use where decode tok/s on one card is what matters.; 2-3 bpw territory where generic quants fall apart and you want per-model calib…engines/exllamav2

verifiedhigh · 0.90

ExLlamaV3

cuda, cpu-avx512, cpu-avx2Current-generation NVIDIA consumer cards: best-quality low-bit quants plus explicit MoE expert/tensor parallelism for machines that are not a datacenter.; Fitting a large MoE model into a single card with CPU offload for the leftover experts.; Single-user interactive decode speed…engines/exllamav3

verifiedhigh · 0.92

exo

metal, cpuRunning a model too large for one Mac, across several Mac Studios you already own, at zero marginal cost; Private local inference on hardware you already have, where 'local' is the point rather than the fallback; Studying Thunderbolt 5 RDMA as a consumer-memory interconnect - the…engines/niche-exo

verifiedmedium · 0.82

faster-whisper

cuda, cpu-avx512, cpu-avx2Batch transcription of many audio files on NVIDIA GPUs; Python-based workflows that need Whisper accuracy with better speed; Word-level timestamps and VAD filteringengines/faster-whisper

verifiedhigh · 0.90

FasterTransformer

cudaoffline / static-batch throughput on NVIDIA GPUsengines/fastertransformer

verifiedmedium · 0.80

FlagEmbedding (BAAI BGE retrieval toolkit)

cuda, cpuProducing or fine-tuning the embedding and reranker models your retrieval stack will actually serve - it is a model factory first; BGE-M3 when you want one multilingual dense+sparse+multi-vector model instead of a three-model pipeline; Reproducing C-MTEB / MTEB retrieval quality …engines/niche-flagembedding

verifiedmedium · 0.80

FlashInfer

cudaSqueezing attention and sampling throughput inside SGLang or vLLM on NVIDIA GPUs; Researching custom attention plans and sampling strategies against a real serving harness; As the default attention path when you have already committed to SGLangengines/flashinfer

verifiedhigh · 0.85

Guidance

transformers, llama.cpp, openai, cuda, cpuWriting programs that interleave constraints with generation, which no other library in this slice does - regex fields appearing mid-output, conditions on later fields constraining earlier ones; Iterating on a constrained prompt cheaply using the Mock backend, with no GPU; Struct…engines/niche-guidance

verifiedhigh · 0.85

Habana Gaudi PyTorch Bridge and the Gaudi torch fork (torch-hpu)

intel-gaudi-hpuRunning existing PyTorch inference code on Gaudi with minimal model-code edits; Understanding what a hardware-first framework fork actually costs in version coupling and system-level install steps; Anyone tuning Gaudi throughput, since the host-CPU and BIOS prerequisites in the l…engines/xpu-gaudi-pytorch-bridge

verifiedhigh · 0.85

HailoRT (Hailo AI Software Suite runtime)

Hailo-8, Hailo-8L, Hailo-10H, Hailo-15Hsub-5W edge vision inference; Raspberry Pi 5 AI kit accelerationengines/hailo-rt

verifiedlow · 0.60

HIP and HIPIFY: the ROCm-native source and runtime layer

rocm, hipPorting CUDA kernels to AMD and understanding, from AMD's own generated table, exactly how far the translation goes per API family; Reading the clearest statement in this repo of what 'ROCm-native' does and does not mean - AMD hardware target, NVIDIA-shaped interface; Diagnosing …engines/amd-hip

verifiedhigh · 0.88

HuggingFace Optimum

onnxruntime, openvino, tensorrt, cuda, rocm, amd-npu, inferentia2, trainium, intel-gaudi, furiosaPorting an embedding or reranker model to a non-default accelerator without writing a custom runtime.; ONNX-based encoder deployment in general: the canonical export path for BERT-class retrieval models.; CPU-only or constrained targets (OpenVINO on Intel, ONNX Runtime on CPU) wh…engines/hf-optimum

verifiedmedium · 0.78

HuggingFace Text Embeddings Inference (TEI)

cuda, rocm, metal, cpu-avx512, mkl, onnxruntime, arm64-neonProduction embedding and reranking serving in general - this is the reference implementation most teams reach for first, and it is what Hugging Face itself deploys.; Serverless and scale-to-zero: small images, fast boot, no graph compilation, and a startup cost measured in second…tracing: per-phaseengines/hf-text-embeddings-inference

verifiedhigh · 0.92

ik_llama.cpp

cpu-avx512, cpu-avx2, cuda, arm64-neonMaximum decode speed on a modern x86 CPU, especially AVX-512 parts (Zen4, Sapphire Rapids+) built with the documented flags.; Sub-4-bit quality-preserving quants on consumer hardware — the reason to pick this fork over mainline.; Fitting a big MoE model on limited VRAM by leaving…engines/ik-llama-cpp

verifiedhigh · 0.85

Infinity

cuda, rocm, cpu, tensorrtOne server that has to do embeddings, reranking AND multimodal/ColBERT retrieval, without running three engines; CPU-only or non-GPU embedding throughput via the CTranslate2 backend, which is Infinity's clearest advantage over TEI; MIT-licensed embedding infrastructure; Fast expe…engines/niche-infinity

verifiedmedium · 0.78

Infinity embedding and reranking server

cuda, rocm, cpu, apple-mps, aws-inferentia2, onnxruntime, tensorrtOne server for embeddings AND reranking, when you would otherwise run two deployments and reconcile their batching policies.; Multi-model and multi-modal retrieval: CLIP/CLAP/ColPali alongside text encoders, with Infinity doing the orchestration.; Apple Silicon and AWS Inferentia…engines/infinity-embeddings

verifiedmedium · 0.82

Jan

cpu-avx512, cpu-avx2, cuda, rocm, hip, vulkan, metalA desktop ChatGPT replacement for a non-technical user who wants a GUI and a working download-and-go experience.; Apple Silicon users who want an app that uses MLX rather than only llama.cpp.; Offline, private use where cloud connectors are optional rather than the default.; Feed…engines/jan

verifiedhigh · 0.85

KoboldCpp

cuda, rocm, hip, vulkan, metal, cpu-avx512, cpu-avx2, arm64-neonRoleplay, creative writing, and story workflows — the bundled KoboldAI Lite UI is built for it and nothing else is this complete.; A single-user home box that should do text, images, speech, and music without five separate installs.; AMD owners who have no working ROCm path and w…engines/koboldcpp

verifiedhigh · 0.87

KServe

none (orchestration layer - drives vLLM, SGLang, TensorRT-LLM, Triton)Kubernetes-native LLM serving with standardized APIs; Multi-model fleets where requests should route to the model with the best cache hit; Teams already invested in Kubernetes who want a vendor-neutral serving layer; Autoscaling on LLM-specific metrics (KV cache utilization, queu…engines/niche-kserve

verifiedhigh · 0.88

KTransformers

cuda, cpu-avx512, cpu-avx2, npu-ascendRunning a very large MoE model on a consumer GPU with large system RAM, when datacenter hardware is not available; Researching heterogeneous placement and expert scheduling - that is what the project is for; MoE fine-tuning on a single workstation-class machineengines/ktransformers

verifiedmedium · 0.75

Lemonade

cpu-avx512, cpu-avx2, cuda, rocm, hip, vulkan, metal, npuAMD Ryzen AI / Strix Halo machines, where it is the most complete story available for NPU plus iGPU plus dGPU in one server.; One server for text, images, video, speech-to-text, TTS, audio and even 3D and classification — the modality coverage is unmatched by anything else here.;…engines/lemonade

verifiedhigh · 0.85

LightLLM

cuda, rocmReading how a serving framework is built when the whole thing is Python and the kernels are Triton - the fastest way to understand paged KV, continuous batching and prefix caching as readable code; Studying SLA-aware scheduling: the Past-Future Scheduler is a genuinely different …engines/niche-lightllm

verifiedhigh · 0.85

LiteLLM

none (proxy layer - no local compute backend of its own)One API key and one billing surface across many cloud providers.; Fallback and load balancing across heterogeneous deployments — including a local llama.cpp or vLLM box as one target among cloud providers.; Cost attribution and observability for LLM traffic, with an OpenTelemetry…tracing: per-phaseengines/litellm

verifiedhigh · 0.90

LiteRT (TFLite successor)

cpu, nnapi, gpu-delegate, coreml, hexagon, xnnpackTensorFlow users who want to deploy models on-device without leaving the TF ecosystem; Android apps using NNAPI or GPU delegate for hardware acceleration; iOS apps using CoreML for Neural Engine acceleration; Vision and small models on mobile - TFLite's traditional strengthengines/local-litert

verifiedmedium · 0.80

llama.cpp

cuda, hip, metal, vulkan, opencl, cpu-avx512, cpu-avx2, webgpu, openvinoRunning a quantized GGUF on whatever GPU you happen to own, including GPUs with no vendor compute SDK; Single-user or small-group local serving where install simplicity and low memory use matter more than multi-tenant throughput; Hybrid CPU+GPU inference for models larger than yo…tracing: noneverified negativesengines/llama-cpp

verifiedhigh · 0.93

llama.cpp embedding and rerank server paths

cuda, metal, vulkan, cpu, hip, syclServing an embedding model in GGUF alongside LLMs in one binary, one API surface and one deployment.; Edge, desktop and Apple Silicon retrieval where GGUF weights and a single static binary are the constraint.; CPU-only embedding and reranking at modest scale.; Reranking with a G…tracing: noneverified negativesengines/llama-cpp-embedding-server

verifiedmedium · 0.72

llama.cpp server (observability surface: 14 hand-built Prometheus metrics, off by default, no OTel)

cpu-avx2, cpu-avx512, cuda, metal, vulkan, hip, rpcA local or edge deployment where you want to know it is doing work at all: tokens processed, decode calls made, slots busy; Single-process monitoring without a Prometheus dependency - the output is plain text with HELP and TYPE lines that anything can scrape; Confirming that spec…tracing: noneverified negativesengines/obs-llama-cpp-observability

verifiedhigh · 0.92

llamafile

cpu-avx512, cpu-avx2, cuda, rocm, hip, metal, vulkan, arm64-neonHanding someone a model with zero install friction: email one file, they double-click it.; Air-gapped, edge, kiosk, or locked-down machines where you cannot install a Python or CUDA stack.; Ship a complete local AI app (LLM + speech-to-text + image gen) as portable executables pe…engines/llamafile

verifiedhigh · 0.90

llm-d

cuda, rocmKubernetes production serving where PD disaggregation and tiered KV need to be operated rather than hand-rolled; Shops that want disaggregation without betting on a single vendor's stack, given the CNCF governance and multi-vendor founding; Multi-turn or long-context workloads wh…engines/llm-d

verifiedhigh · 0.86

llm.c

cuda, cpu-avx2Understanding exactly what a transformer forward and backward pass costs, line by line, with nothing hidden behind a framework; Learning CUDA GEMM and attention kernel structure by reading a complete, small, correct implementation; Establishing the memory-bandwidth argument for d…engines/niche-llm-c

verifiedhigh · 0.85

lm-format-enforcer

cuda, rocm, metal, cpuReading how prefix-allowed-tokens constraint enforcement works, in about 150 lines of Python you can fully understand; Beam search or batched generation with per-sequence different constraints, where the README's own table says the alternatives are weaker; Cases where you want th…engines/niche-lm-format-enforcer

verifiedhigh · 0.85

LMCache

cuda, rocm, npuLong-context agentic, multi-turn and RAG workloads where TTFT is dominated by repeated prefill of a shared prefix; Fleets serving the same corpus across multiple engines or replicas, where a shared KV store beats per-engine prefix caches; Deployments that want the engine decouple…engines/lmcache

verifiedhigh · 0.85

LMDeploy (observability surface: near-copy of vLLM's metric set, no preemption metric, no OTel)

cuda, rocm, camb, maca, ascendA Turbomind or PyTorch deployment where you want vLLM-shaped metrics without running vLLM, including the shipped Grafana dashboard; Prefix-cache analysis: the per-request cached-token histogram and cache-hit-ratio histogram are finer-grained than vLLM's counters alone; Speculativ…tracing: noneverified negativesengines/obs-lmdeploy-observability

verifiedhigh · 0.90

LMDeploy (TurboMind)

cuda, cpuDeployment if you are in the InternLM/Qwen ecosystem, where this is the best-tuned option and the model support is first-class; Compression-forward serving: AWQ, MXFP4 and KV-cache quantization are supported together here in a way most engines do not coordinate; Running a proxy i…tracing: noneverified negativesengines/niche-lmdeploy-turbomind

verifiedmedium · 0.82

MediaPipe (Google on-device ML framework)

cpu, gpu, tflite-delegateVision, audio and text-classification on Android or iOS where you want packaged turnkey tasks rather than assembling a runtime yourself; Live camera and microphone pipelines - MediaPipe's actual reason to exist, and where its streaming graph model beats a request-oriented engine;…engines/niche-mediapipe

verifiedmedium · 0.80

Microscaling PTQ research harness

noneapples-to-apples comparison of INT vs MX formats and of PTQ algorithm combinationsengines/mxptq-research-harness

verifiedmedium · 0.80

MLC-LLM

cuda, rocm, vulkan, metal, opencl, webgpu, wasmDeploying one model to many target classes - browser, phone, iOS, Mac, Windows/Linux GPU - from a single compile pipeline; In-browser and on-device inference with an OpenAI-compatible API and no server; Intel, Adreno, Mali or other GPUs without a vendor-tuned inference stackengines/mlc-llm

verifiedhigh · 0.88

MLX (Apple's array framework for Apple silicon)

metal, unified-memoryThe framework layer for anything on Apple silicon, and the substrate for [[mlx-lm]] and [[vllm-metal]]; Learning what unified memory buys and costs: no copies anywhere in the stack, at the price of a RAM-sized model ceiling and HBM-class bandwidth being unavailable; Comparing aga…engines/apple-mlx

verifiedhigh · 0.88

MLX-LM

metalLocal Mac development and personal use, where Metal is fast and battery/thermals matter; Quantized models from mlx-community that were converted specifically for MLX, since those are the tuned path; Fine-tuning or LoRA work on Apple Silicon, which mlx-lm supports natively includi…engines/mlx-lm

verifiedhigh · 0.88

MNN (Alibaba mobile inference engine)

cpu, vulkan, opencl, cuda, metal, hexagon-dspShipping an LLM or multimodal model inside an Android or iOS app with no server dependency, which nothing else in this repo does; Mobile CPU, Vulkan or OpenCL inference where the .mnn format and a small footprint are the point; Anyon working with Qualcomm Hexagon DSPs - the 3.6.1…engines/niche-mnn

verifiedmedium · 0.82

Model2Vec (static embeddings) and model2vec-rs

cpu, rust-nativeOn-device, edge or embedded semantic search where a 400MB BERT encoder is simply not shippable - this is the case that no other record in this niche covers; High-throughput CPU embedding over a large corpus, where the transformer forward pass is the bottleneck; Rust applications …engines/niche-model2vec

verifiedmedium · 0.80

Mooncake

cuda, rocm, npu, tpuLong-context production serving where the KV working set exceeds HBM and the idle CPU/DRAM/SSD in the cluster can be used as a real KV tier; Overloaded deployments that need admission control rather than unbounded queueing; Heterogeneous fabrics - the transport matrix (RDMA, EFA,…engines/mooncake

verifiedhigh · 0.85

ncnn (Tencent)

cpu, vulkan, openclAndroid apps requiring Vulkan GPU acceleration with minimal dependencies; Mobile vision models (detection, segmentation, face recognition); Embedded Linux devices with Vulkan support; Applications where binary footprint is criticalengines/local-ncnn

verifiedhigh · 0.85

NVIDIA Dynamo

cuda, rocmMulti-node production serving where prefill and decode need to be scaled and costed independently; Kubernetes-based deployments that want an OpenAI-compatible frontend, SLA-driven autoscaling and KV-aware routing as one stack; Fleets that already run vLLM, SGLang or TensorRT-LLM …tracing: per-requestengines/nvidia-dynamo

verifiedhigh · 0.91

NVIDIA Inference Xfer Library (NIXL)

cuda, rocmAny disaggregated or KV-tiered inference deployment that needs to move KV blocks between processes, nodes or memory tiers with a pluggable transport; Deployments where the transport must be swappable (RDMA fabric vs libfabric vs GPUDirect Storage) without changing the engine; Mul…engines/nixl

verifiedhigh · 0.85

Ollama

cuda, rocm, hip, metal, vulkan, cpu-avx512, cpu-avx2, arm64-neonGetting from 'I want to run this model' to a working API in one command — nothing else in this list is this easy.; Single-user laptop and desktop chat across mixed hardware, including machines with no supported CUDA or ROCm path.; Switching between several small models quickly — …tracing: noneverified negativesengines/ollama

verifiedhigh · 0.88

Ollama (observability surface: none - no /metrics endpoint, no tracing, per-response timing only)

cuda, rocm, metal, vulkan, cpu-avx512, cpu-avx2Local and laptop use, where 'ollama ps' answering 'what is loaded and how much VRAM' is the entire operational question; A deployment where an external sidecar does the instrumentation - the third-party integrations in the README are the intended pathtracing: noneverified negativesengines/obs-ollama-observability

verifiedhigh · 0.90

ONNX Runtime GenAI

cpu-avx512, cpu-avx2, cuda, directml, openvino, qnn, webgpu, amd-gpu, tensorrt-rtx, arm64-neonONNX-first teams and Windows/Foundry Local users who want a supported, vendor-backed path rather than a hobby project.; NPUs and other accelerators that only ONNX Runtime targets: QNN for Qualcomm, DirectML for broad Windows GPU support, WebGPU for browsers.; Arm64 and Android de…engines/onnxruntime-genai

verifiedhigh · 0.85

ONNX Runtime Mobile

cpu, nnapi, coreml, xnnpack, qnnDeploying ONNX models to Android and iOS apps with a single runtime; Cross-platform mobile inference where ONNX is the interchange format; Qualcomm devices with QNN backend for NPU acceleration; iOS devices with CoreML delegation for Neural Engineengines/local-onnxruntime-mobile

verifiedhigh · 0.85

OpenAI Whisper (reference implementation)

cuda, cpu-avx512, cpu-avx2Research and accuracy comparisons; Understanding the reference algorithm; Fine-tuning and experimentationengines/openai-whisper

verifiedhigh · 0.90

OpenVINO GenAI

cpu-avx512, cpu-avx2, intel-gpu, npuIntel Xeon, Core Ultra and Arc hardware where you want a vendor-supported engine rather than a community fork.; Intel NPU deployment — this is one of the few engines with a first-class NPU story alongside ONNX Runtime's QNN path.; Getting the largest model you can onto a PC witho…engines/openvino-genai

verifiedhigh · 0.90

Outlines

cuda, rocm, metal, cpuRapid prototyping of schema-constrained generation, especially when you want to iterate on the constraint in Python without touching a C++ build; Local model work where you control the host process and can install a logits hook; Learning the design space of structured generation …engines/niche-outlines

verifiedhigh · 0.85

Portkey AI Gateway

none (gateway layer - forwards to any OpenAI-compatible or native provider endpoint)Multi-provider fleets where the routing decision is commercial rather than technical - cost split, canary rollout, or spreading across per-account provider rate limits; Teams wanting one endpoint and one config surface across many vendors, including their own self-hosted OpenAI-c…engines/portkey-ai-gateway

verifiedmedium · 0.80

Qdrant FastEmbed

onnxruntime, cudaIngest pipelines and one-off batch embedding where there is no server to run - CI, notebooks, ETL, Lambda.; CPU-only or constrained environments where pulling in PyTorch is unacceptable; the dependency footprint is the main reason to choose it.; Local RAG stacks around Qdrant: qd…engines/qdrant-fastembed

verifiedmedium · 0.78

Ray Serve

none (orchestration layer - drives any Python callable)Multi-model serving with complex composition (e.g., LLM + reranker + guardrail); Teams already using Ray for training who want to use the same framework for serving; Python-heavy inference pipelines that need flexible composition; Fractional GPU sharing across many small modelsengines/niche-ray-serve

verifiedhigh · 0.85

Semi-PD (research prototype)

cuda, rocmSingle-node disaggregation where the interconnect is too slow to make a KV handoff pay - the shared-memory design makes the transfer cost zero by construction; Studying the alternative to the copy: if you can avoid moving KV rather than move it faster, the cost model changes shap…engines/semi-pd

verifiedmedium · 0.80

Sentence Transformers

cuda, rocm, cpu, onnx, openvinoThe correctness baseline: if you want to know what a model's real embedding quality is, this is the implementation the leaderboards measure.; Training and fine-tuning embedding, reranker, sparse or multi-vector models - nothing else in this slice trains.; Offline batch embedding …engines/sentence-transformers

verifiedhigh · 0.85

SGLang

cuda, rocm, metal, xpu, tpu, cpu-avx512Agentic and RL workloads where many requests share long, branching context - the radix cache reuses partial prefixes, not just exact ones; Serving latency-sensitive multi-turn conversation, because prefix reuse removes the re-prefill of chat history; Teams that want to experiment…tracing: per-phaseengines/sglang

verifiedhigh · 0.93

SGLang (observability surface: Prometheus metrics + OpenTelemetry with per-stage slices)

cuda, rocm, metal, xpu, tpu, cpu-avx512Reading a disaggregated prefill/decode deployment: `sglang:kv_transfer_speed_gb_s`, `sglang:kv_transfer_latency_ms`, `sglang:kv_transfer_bootstrap_ms`, `sglang:kv_transfer_alloc_ms`, `sglang:kv_transfer_total_mb`, and per-stage gauges for prealloc/bootstrap/wait/transfer/host-rec…tracing: per-phaseengines/obs-sglang-observability

verifiedhigh · 0.93

SGLang Router (sgl-router)

none (routing layer - drives SGLang engine instances)Multi-model SGLang serving where requests should route to the model with the best cache hit; Data parallel SGLang deployments needing load balancing across workers; Agentic workloads with long shared prefixes where prefix-aware routing matterstracing: per-requestengines/sglang-router

verifiedhigh · 0.92

SmoothQuant reference implementation (PyTorch W8A8)

nonereproducing the paper's accuracy numbersengines/smoothquant-reference

verifiedmedium · 0.80

Sonar (formerly Aphrodite Engine)

cuda, rocm, cpu, metal, tpu, xpuA self-hosted single-model or small-mixture server for a hobbyist or small team that wants vLLM-class features with a one-line installer and more model/quantization coverage than upstream ships; Multi-model serving without standing up Ray; Reading what a well-run vLLM fork looks …engines/niche-aphrodite-sonar

verifiedmedium · 0.82

SpinQuant research harness (fake-quant + Triton INT4 kernels)

cudaunderstanding what W4A4 costs when activations AND KV cache are quantizedengines/spinquant-research-harness

verifiedmedium · 0.75

TensorRT-LLM

cudaMaximum throughput on a fixed NVIDIA fleet, especially MoE models where expert parallelism and load balancing are the bottleneck; Deployments that can afford a build-time/version-lock step in exchange for hand-tuned kernels, CUDA graphs and the overlap scheduler; Latency-bound si…tracing: per-requestengines/tensorrt-llm

verifiedhigh · 0.87

TensorRT-LLM (observability surface: Prometheus metrics + OpenTelemetry)

cudaAn NVIDIA-only fleet where you want vLLM-compatible metric semantics with TensorRT-LLM's extra inflight-batching and multi-tier KV counters; Measuring multi-tier KV offload (GPU/host/disk), where `trtllm_kv_cache_host_utilization`, `trtllm_kv_cache_disk_prefetch_blocks_total` and…tracing: per-requestengines/obs-tensorrt-llm-observability

verifiedhigh · 0.90

Text Embeddings Inference (TEI)

cuda, rocm, cpu, metalServing embeddings and rerankers in production, especially where the vector DB is the bottleneck and embedding throughput is the real constraint; Serverless or autoscaling deployments, because no graph compilation and small images are first-class design goals rather than accident…tracing: per-phaseengines/niche-text-embeddings-inference

verifiedhigh · 0.92

Text Generation Inference (observability surface: no TTFT, no KV cache usage, no tracing spans of its own)

cuda, rocm, neuronComparing prefill and decode batch cost directly, because the batch histograms carry a method label no other engine provides at that granularity; Existing TGI deployments where you accept queue time + inference time + time-per-token as the latency proxy settracing: basicverified negativesengines/obs-tgi-observability

verifiedhigh · 0.90

TinyChat (AWQ on-device runtime)

cudasingle-stream on-device generation on consumer GPUs and ARM CPUsengines/tinychat-awq

verifiedmedium · 0.80

Triton Inference Server

cuda, tensorrt, onnx, openvino, python, cpu, inferentiaServing a portfolio of many different models from many different frameworks behind one HTTP/gRPC contract - Triton's actual design target; Non-LLM workloads at scale: vision, speech, recommendation, classical ML alongside a couple of LLMs; Multi-stage inference pipelines where pr…tracing: per-requestengines/niche-triton-inference-server

verifiedhigh · 0.85

Triton Inference Server

cuda, tensorrt, onnxruntime, openvino, pytorch, rapids-fil, inferentia, cpuOrganisations already standardised on Triton who need embedding and rerank served alongside everything else, with one operational surface and one observability stack.; Full RAG serving pipelines: embedding, storage lookup and cross-encoder rerank composed as an ensemble behind a …tracing: per-requestengines/triton-inference-server

verifiedhigh · 0.88

Triton Inference Server (observability surface: request-shaped core metrics, LLM metrics only if a backend registers them)

cuda, rocm, cpu, inferentiaServing many different model types behind one endpoint where a uniform request-level metric set matters more than LLM-specific insight; Tracing an ensemble or BLS request end to end, since each composing model contributes its own model and compute spans; Choosing your own latency…tracing: per-requestengines/obs-triton-observability

verifiedhigh · 0.92

TT-Metal / TT-NN (Tenstorrent)

tenstorrent-wormhole, tenstorrent-blackhole, tenstorrent-grayskullDeploying on Tenstorrent Wormhole or Blackhole, where this plus the vLLM plugin is the only real path; Learning a non-CUDA kernel programming model - TT-Metalium is the clearest existing example of a full alternative to writing CUDA; Studying honest vendor benchmarking: the table…engines/niche-tt-metal

verifiedmedium · 0.82

vLLM

cuda, rocm, xpu, metal, cpu-avx512Multi-user servers with high request concurrency, where continuous batching plus paged KV is what actually buys throughput; Workloads with heavy shared prefixes (long documents queried repeatedly, multi-turn chat) because automatic prefix caching removes repeated prefill; Broad m…tracing: per-requestengines/vllm

verifiedhigh · 0.93

vLLM (observability surface: Prometheus metrics + OpenTelemetry)

cuda, rocm, xpu, metal, cpu-avx512Anyone who needs to build an SLO on TTFT or TPOT and defend it, because the histogram buckets, the definitions and the deprecation story are all documented; Capacity planning and autoscaling: running/waiting/deferred-by-reason plus KV usage and preemption count is the full satura…tracing: per-requestengines/obs-vllm-observability

verifiedhigh · 0.95

vLLM Ascend (Huawei Ascend NPU hardware plugin)

npu, ascend-cannRunning vLLM on Huawei Ascend NPUs, full stop - there is no realistic alternative at this feature level; Studying how a hardware vendor's ecosystem gets supported without forking the serving framework, and what that pattern costs; Anyone deploying into Huawei-dominated regions wh…tracing: noneverified negativesengines/niche-vllm-ascend

verifiedhigh · 0.90

vLLM Hardware Plugin for Intel Gaudi (vllm-gaudi)

intel-gaudi-hpuServing on Gaudi 2 or Gaudi 3 where vLLM's scheduler and paged KV semantics are wanted but CUDA kernels are unavailable; Intel shops that already standardised on vLLM's OpenAI-compatible API and want to keep that surface while changing hardware; Reading a worked example of how fa…engines/vllm-gaudi

verifiedhigh · 0.85

vLLM pooling models (embedding, rerank, late interaction)

cuda, rocm, cpu, xpu, tpu, metalOne server for both generation and retrieval: if you already run vLLM for chat, adding an embedding and a reranker endpoint on the same deployment removes an entire service.; Late-interaction (ColBERT-style) scoring as a first-class endpoint - the only mainstream engine here with…tracing: per-requestengines/vllm-pooling-models

verifiedhigh · 0.85

vLLM Production Stack

cuda, rocmDeploying vLLM to Kubernetes in a way that survives contact with a real traffic pattern - this is the maintained reference and it beats hand-rolled manifests; Multi-model serving where requests should route to the model that already has the prompt cached; Getting LMCache-based KV…engines/niche-vllm-production-stack

verifiedhigh · 0.88

vLLM TPU inference plugin (tpu-inference)

tpuServing on TPU v5e, v6e or v7x with real vLLM semantics - paged KV, continuous batching, prefix caching - which is what JetStream never quite had; Teams with PyTorch model code that need to move to TPU without a JAX rewrite, which is the plugin's headline claim; Anyone deciding w…engines/alt-tpu-inference-vllm-plugin

verifiedhigh · 0.85

vLLM-Metal

metalRunning vLLM's serving semantics (paged KV, continuous batching, prefix caching) on a Mac; Local development against a vLLM-compatible API on Apple Silicon; Serving a model that fits in unified memory with acceptable single-user latencyengines/vllm-metal

verifiedmedium · 0.82

WebAssembly (WASM) inference

wasm-simd, wasm-threadsCross-platform inference in browsers and standalone runtimes; Server-side inference with WASM sandboxing (Wasmtime, WasmEdge); Applications requiring near-native performance without native compilation; Small to medium models (1-13B) with WASM SIMDengines/local-wasm

verifiedmedium · 0.80

WebGPU (in-browser inference)

webgpu, wasm-simdZero-install LLM inference in the browser; Web applications that need local inference without server dependency; Cross-platform deployment (Windows, macOS, Linux, ChromeOS) via browser; Small models (1-7B) that fit in browser memory constraintsengines/local-webgpu

verifiedmedium · 0.80

WebLLM

webgpu, wasm, cpuClient-side inference where the data must not leave the device - the actual reason this exists and the one thing nothing else does; Zero-ops prototypes and demos: no API key, no server, no cost, works offline after first load; Learning the WebGPU compute path for LLM inference, w…engines/niche-webllm

verifiedhigh · 0.90

whisper.cpp

cuda, rocm, hip, metal, vulkan, cpu-avx512, cpu-avx2, opencl, xnnpack, npuCross-platform Whisper inference from a single binary; Edge and mobile deployment (iOS, Android, WASM); Scenarios where Python/PyTorch is not available or desired; Rapid prototyping and benchmarkingengines/whisper-cpp

verifiedhigh · 0.90

XGrammar

cuda, cpu, metal, opencl, vulkan, tpu, xnnpackAny production server that must return schema-valid JSON and currently retries on parse failure - this is the cheapest fix, because it removes retries rather than making them faster; Reading how a token-level grammar engine keeps overhead near zero, and why the C++ core with a th…engines/niche-xgrammar

verifiedhigh · 0.90

Xorbits Inference (Xinference)

cuda, rocm, cpu, vulkan, apple-mps, openclTeams running a mixed fleet (chat models plus embedding plus rerank) behind one server and one UI instead of three deployments.; Decoder-based rerankers (Qwen3-Reranker) and multimodal rerankers (Qwen3-VL-Reranker, jina-clip-v2, GME-Qwen2-VL), where BERT-cross-encoder engines lik…engines/xinference

verifiedmedium · 0.75

Android NNAPI (vendor-neutral on-device acceleration API)

nnapi (cpu/gpu/npu via vendor drivers)Understanding WHY the on-device Android ecosystem looks the way it does - NNAPI's retirement is the reason GPU and XNNPACK delegates are the current story; Reading the historical path from a vendor-neutral abstraction to Google-specific delegates, which is a genuine lesson about …engines/niche-android-nnapi

deprecatedhigh · 0.88

DeepSpeed-MII (DeepSpeed-Inference)

cudaReading the clearest pre-vLLM articulation of continuous batching + blocked KV + prefill/decode fusion as a single design - the DeepSpeed-FastGen blog is the primary source; Understanding Dynamic SplitFuse, which named the problem vLLM's chunked prefill and SGLang's overlap sched…engines/niche-deepspeed-mii

deprecatedhigh · 0.85

FastChat (LMSYS multi-model serving + evaluation)

cuda, cpuReproducing the leaderboard-era methodology - MT-Bench, judge-based evaluation, side-by-side battles - which is still the closest thing to a community standard for chat-model comparison; Reading the origin of the open-weights chat model ecosystem; the Vicuna release notes are the…engines/niche-fastchat

deprecatedhigh · 0.85

h2oGPT (H2O.ai Stateful Open-Source RAG + LLM inference)

cuda, cpu, metalStudying what a complete private-RAG product needs beyond an inference engine, and how much of that is vector-store plumbing rather than model serving; Reading the HYDE and semantic-chunking implementations as concrete code; Historical: it is the clearest public example of the OS…engines/niche-h2ogpt

deprecatedhigh · 0.88

JetStream (Google's XLA/TPU inference engine, archived)

tpuUnderstanding what a first-generation TPU serving stack looked like, and specifically that it was built on the compiler rather than the chip; Studying the TPU operational tooling that still matters: Prometheus metrics and JAX profiler integration were the mature part of this proj…engines/alt-jetstream

deprecatedhigh · 0.90

LoongServe (elastic sequence parallelism for long-context LLM serving)

cudaStudying elastic sequence parallelism as a technique - the code and the AE comparison matrix against DistServe, SplitFuse and vLLM make this the most useful artefact in this corner of the field; Reading how prefill-decode disaggregation was approached in 2024, alongside [[distser…engines/niche-loongserve

deprecatedmedium · 0.74

Medusa

cudaUnderstanding draft acceptance rates and why multi-head drafting was a dead end relative to feature-level drafting (EAGLE) - the clearest possible teaching artefact; Single-GPU, batch-1 latency experiments on a model you have heads for; Studying head-only fine-tuning (Medusa-1) v…engines/niche-medusa

deprecatedhigh · 0.85

SynapseAI / SynapseAI Core (Habana's Gaudi compiler and runtime)

intel-gaudi-hpu, first-gen-gaudi, gaudi2Understanding what Gaudi's graph-based execution model actually consists of, from the layer names down (driver, hl-thunk, TPC LLVM, Synapse Backend) rather than from marketing; Learning the TPC/MME engine split in the only form where both engines are visible in runnable code; Stu…engines/xpu-synapseai

deprecatedhigh · 0.90

Text Generation Inference

cuda, rocm, xpu, npu-gaudi, cpu-avx512Reading how a production HF serving stack was put together, as a historical reference; Existing deployments that have not yet migrated off ittracing: basicengines/text-generation-inference

deprecatedhigh · 0.87

TorchServe (PyTorch Serve)

cuda, rocm, cpu, mps (via torch)Understanding how general (non-LLM) model serving was done in the pre-LLM era, and why the Kubernetes-first design led to KServe; Learning the model-archiving pattern (.mar files, torch-model-archiver, chained handlers), which is durable and independent of LLM serving; Reading a …engines/niche-torchserve

deprecatedhigh · 0.86

Most referenced

The records other records point at. Useful as an entry point: a heavily referenced accelerator or engine is one many decisions in this repo assume.

Record	Referenced by	Cites
llama.cpp	3787	6
vLLM	582	14
TensorRT-LLM	193	11
SGLang	185	4
MLX-LM	117	3
Ollama	109	3
LMDeploy (TurboMind)	53	5
Text Generation Inference	46	2