Infinity
cuda, rocm, cpu, tensorrt
Fields
id
engines/niche-infinityRepo
michaelfeil/infinity
Languages
Python
License
MIT
First release
2023
Design goal
throughput-firstmulti-backendresearchease-of-use
Scheduling
continuous-batching
Backends
cudarocmcputensorrt
Formats
safetensorsonnxgguf
Notable features
Embedding, reranking and multimodal (text+image) retrieval serving in one engine, layered on three interchangeable backends: PyTorch, optimum (ONNX / TensorRT), and CTranslate2. The multi-backend design is the reason to pick it over TEI when you need breadthCTranslate2 in the stack is unusual for this class and fast on CPU - the practical way to get good embedding throughput with no acceleratorDocumented docker images for CUDA, CPU, ROCm (MI200 and MI300 series) and a separate ONNX-Cuda / CTranslate2 / TensorRT imageExperimental int8 on cpu/cuda and experimental fp8 on H100/MI300 - the fp8 line is explicitly scoped to those two data-centre partsHandles ColBERT and ColPALI-family multimodal embedding models, e.g. vidore/colpali-v1.2-mergedMIT-licensed, which is a genuine differentiator in the embedding-serving class2949 stars, last push 2026-03-24, 281 open issues
Hardware caveats
RELEASE CADENCE HAS SLOWED MARKEDLY. Last push 2026-03-24 and the last release is 0.0.77 from 2025-08-22 - a gap of over a year between the newest release and today, and the project is still on a 0.0.x version line. Commits continue but nothing is cut. Compare [[niche-text-embeddings-inference]] at v1.9.4 / 2026-09-15int8 is experimental on cpu/cuda and fp8 is experimental on H100/MI300 only - read the README's own qualifiers rather than assuming general quantisation supportROCm coverage is MI200 and MI300 series only, via a dedicated imageNo lora adapters: for the ColBERT/ColPali-family paths the README states 'No lora adapters supported, only merged models'. If you fine-tune with adapters you are outside this engineMulti-backend means multi-problem: a bug can be in PyTorch, in optimum's ONNX/TensorRT path, or in CTranslate2, and support has to cover all threeUNVERIFIED MARKETING-ADJACENT CLAIM: the story that Infinity was acquired by or merged into Milvus could not be confirmed. What could be checked: only 15 code references to 'infinity' inside milvus-io/milvus and 2 issue mentions in the Infinity tracker. That is consistent with a maintained-but-autonomous project that interoperates with Milvus, not with an acquisition. Confidence held at 0.78 partly because of thisA researcher's tool rather than an infrastructure project: 281 open issues and no release train means you are running something close to main
Best for
One server that has to do embeddings, reranking AND multimodal/ColBERT retrieval, without running three enginesCPU-only or non-GPU embedding throughput via the CTranslate2 backend, which is Infinity's clearest advantage over TEIMIT-licensed embedding infrastructureFast experimentation: it is Python-only, so the backend switch is a config change
Avoid for
Anything needing release-to-release reproducibility - a 0.0.x version with no release since 2025-08-22 is running close to mainAdapter-based fine-tuned embedding models; merged models onlyConsumer AMD GPUs; ROCm is MI-series onlyProduction where you want a support commitment rather than an active hobby project
notes
Recorded as the multi-backend counterpart to TEI in the embedding/reranking niche: TEI for the tightest hardware story and serverless, Infinity for backend breadth and MIT licensing. Cadence is the honest concern and is recorded as such. The Milvus claim is explicitly flagged unverified in both the source and the engine notes. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-03-24 (6.4 months before this record), latest release 0.0.77 published 2025-08-22, 35 contributors (distinct committers including anonymous), open_issues_count 135 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 2949 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
Recorded as the multi-backend counterpart to TEI in the embedding/reranking niche: TEI for the tightest hardware story and serverless, Infinity for backend breadth and MIT licensing. Cadence is the honest concern and is recorded as such. The Milvus claim is explicitly flagged unverified in both the source and the engine notes. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-03-24 (6.4 months before this record), latest release 0.0.77 published 2025-08-22, 35 contributors (distinct committers including anonymous), open_issues_count 135 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 2949 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.