DistServe GitHub repository (LLMServe/DistServe)

Apache-2.0, 839 stars, repo created 2024-01-16, last push 2025-04-06 - i.e. it is a research artifact, not a maintained production system. README confirms it is built on a custom C++ backend (SwiftTransformer) supporting model/pipeline parallelism, FlashAttention, Continuous Batching and PagedAttention, and that it supports only GPT-2, OPT and LLaMA2 families. Uses Ray for distributed workers. 'It works just like a single instance which handles the KV-Cache communication and memory management automatically.' Recorded as a paper prototype, not a deployable engine.