llm-d GitHub repository (llm-d/llm-d)

Apache-2.0, 4.7k stars, created 2025-04-29. 'a high-performance distributed inference serving stack optimized for production deployments on Kubernetes', and 'a CNCF sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA' - i.e. it is deliberately vendor-neutral where Dynamo is NVIDIA-originated. Same layering as Dynamo: 'Model servers like vLLM and SGLang handle efficiently running large language models... llm-d provides state-of-the-art orchestration and optimizations above model servers'. Four themes: intelligent routing (prefix-cache and load-aware balancing), advanced KV-cache management ('tiered offloading to CPU or disk and precise global indexing of the KV cache state'), serving large models (PD disaggregation + wide expert parallelism), operational excellence. Partner benchmark claims in the README: up to 70% higher tokens/sec with PD disaggregation vs standard vLLM on GPT-OSS on B200 (AWS); 10-30% throughput improvement with disaggregated serving on identical infrastructure on GPT-OSS-120B and Llama 3.3 70B on AMD MI300X (Oracle); 3x higher output throughput and 2x faster TTFT with prefix-cache-aware routing vs round-robin on Llama 3.1 70B on 4x MI300X (Tesla/Red Hat).