inference/research
verifiedhigh · 0.90updated 2026-10-05

LiteLLM

none (proxy layer - no local compute backend of its own)

Fields

id
engines/litellm
Repo
BerriAI/litellm
Languages
PythonRustTypeScript
License
MIT, except content under the enterprise/ directory which is separately licensed
First release
2023
Design goal
multi-modelease-of-uselow-latencyobservabilityrouting
Scheduling
load-balancingrequest-routingfallback-chainrate-limit-aware-retries
Backends
none (proxy layer - no local compute backend of its own)
Formats
passthrough (speaks OpenAI, Anthropic, Bedrock, Vertex, Azure, and native provider formats; model formats are irrelevant because it is not loading weights)
Notable features
A gateway/proxy, not an inference engine: it exposes one OpenAI- (or native-)format API across 100+ upstream providers, including vLLM, NVIDIA NIM and self-hosted OpenAI-compatible servers.Rust core with a Python SDK, positioned by the project as a fast, lightweight gateway.Cost tracking, guardrails, load balancing and logging as first-class features.Also an MCP gateway.Native provider formats, not just OpenAI-shaped: Anthropic, Bedrock and Vertex requests pass through with their own shapes where needed.Helm chart, Terraform and vscode-extension in-repo for deployment.
Hardware caveats
This record has no local compute backend and that is the point: LiteLLM never loads weights and never touches a GPU, CPU or NPU. supported_backends is recorded as an explicit 'none' rather than left empty so a reader querying by backend does not mistake it for a missing value.Latency is added, not removed. Every request pays at least one extra network hop, so it is the wrong layer for a single-user local model — it exists for fleets.Licensing is split: everything outside the enterprise/ directory is MIT, everything under it is under enterprise/LICENSE. Embedding in a product needs the license split read, not just the repo badge.Because it routes across providers, an upstream that lacks a feature makes that feature appear intermittently missing — the gateway unifies the API surface, not the capability surface.
Best for
One API key and one billing surface across many cloud providers.Fallback and load balancing across heterogeneous deployments — including a local llama.cpp or vLLM box as one target among cloud providers.Cost attribution and observability for LLM traffic, with an OpenTelemetry flush endpoint.Teams standardizing on the OpenAI shape so a model or vendor can be swapped without touching application code.MCP gatewaying in front of many tools or models.
Avoid for
Single-user local inference — pure added latency and an extra process.Anything where you need the model's throughput characteristics to be what you measure end to end.Cutting dependency count; a Rust core plus Python SDK plus provider SDKs is not minimal.Any assumption that API compatibility implies capability parity across providers.
Tracing support
per-phase
Verified negatives
no
Docs
https://docs.litellm.ai/docs/
notes
Included per the delegated brief's allowance for a genuinely distinct serving/routing layer. It is deliberately scoped to the routing/proxy role and touches no inference kernel, so it does not overlap the GPU-serving engines other agents cover. supported_formats records passthrough behaviour rather than weight formats because no weight format reaches this layer — a weight-format query against this record will correctly return nothing useful. Scheduling fields are request-level (which upstream, in what order, with what fallback), not sequence-level; there is no continuous batching or paged KV here. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v1.104.0 published 2026-10-03, 1768 contributors (distinct committers including anonymous), open_issues_count 5179 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 60124 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. tracing_support=unknown (set 2026-10-04): the record mentions an OpenTelemetry flush endpoint only, which establishes neither that tracing is absent nor what a trace contains. LiteLLM is a proxy, so any spans it emits would describe upstream requests rather than inference phases. TRACING VERIFIED 2026-10-05. Read at main 2026-10-05. LiteLLM's integrations/otel/model/spans.py is a declared, closed span vocabulary rather than scattered call sites: SpanRole = PROXY_REQUEST (SERVER, the only root, name literal 'Received Proxy Server Request'), LLM_CALL, MCP_TOOL_CALL, MCP_LIST_TOOLS, GUARDRAIL, DB_CALL and SERVICE, each mapped in SPAN_REGISTRY to a LiteLLMSpanKind and an explicit parent. Phase spans exist inside the request: logger.phase_span(name) / start_phase_span creates an INTERNAL SERVICE span, and the auth phase in particular is emitted as a live phase span specifically so its DB lookups nest under it instead of appearing as flat post-hoc service spans. Because request phases (auth, guardrail, llm_call, db_call) are separately attributable within one trace, per-phase is the honest level. Worth knowing: _METRICS_ONLY_SERVICES = {self, router, proxy_pre_call, auth} documents which service timings deliberately do NOT become spans, and client-propagated trace context in params._meta is recorded as a span LINK rather than a parent - so a span rooted at the client will not appear under the gateway's trace.

Notes

Included per the delegated brief's allowance for a genuinely distinct serving/routing layer. It is deliberately scoped to the routing/proxy role and touches no inference kernel, so it does not overlap the GPU-serving engines other agents cover. supported_formats records passthrough behaviour rather than weight formats because no weight format reaches this layer — a weight-format query against this record will correctly return nothing useful. Scheduling fields are request-level (which upstream, in what order, with what fallback), not sequence-level; there is no continuous batching or paged KV here. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v1.104.0 published 2026-10-03, 1768 contributors (distinct committers including anonymous), open_issues_count 5179 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 60124 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. tracing_support=unknown (set 2026-10-04): the record mentions an OpenTelemetry flush endpoint only, which establishes neither that tracing is absent nor what a trace contains. LiteLLM is a proxy, so any spans it emits would describe upstream requests rather than inference phases. TRACING VERIFIED 2026-10-05. Read at main 2026-10-05. LiteLLM's integrations/otel/model/spans.py is a declared, closed span vocabulary rather than scattered call sites: SpanRole = PROXY_REQUEST (SERVER, the only root, name literal 'Received Proxy Server Request'), LLM_CALL, MCP_TOOL_CALL, MCP_LIST_TOOLS, GUARDRAIL, DB_CALL and SERVICE, each mapped in SPAN_REGISTRY to a LiteLLMSpanKind and an explicit parent. Phase spans exist inside the request: logger.phase_span(name) / start_phase_span creates an INTERNAL SERVICE span, and the auth phase in particular is emitted as a live phase span specifically so its DB lookups nest under it instead of appearing as flat post-hoc service spans. Because request phases (auth, guardrail, llm_call, db_call) are separately attributable within one trace, per-phase is the honest level. Worth knowing: _METRICS_ONLY_SERVICES = {self, router, proxy_pre_call, auth} documents which service timings deliberately do NOT become spans, and client-propagated trace context in params._meta is recorded as a span LINK rather than a parent - so a span rooted at the client will not appear under the gateway's trace.

Cross-references

This record references no other records — it is a leaf. It is cited by 2.