FastChat (LMSYS multi-model serving + evaluation)
cuda, cpu
Fields
id
engines/niche-fastchatRepo
lm-sys/FastChat
Languages
PythonJupyter NotebookShellDockerfile
License
Apache-2.0
First release
2023
Design goal
researchevaluationmulti-modelease-of-useleaderboards
Scheduling
continuous-batching
Backends
cudacpu
Formats
safetensorsgguf
Notable features
The historical role is the important part, and it is large: FastChat is what made side-by-side human model comparison possible at scale. The README states Chatbot Arena has served over 10 million chat requests for 70+ LLMs and collected over 1.5M human votes for the Elo leaderboardThat produced the model ecosystem everyone uses: Vicuna (the March 2023 'impressing GPT-4 with 90% quality' release that started the open-weights chat-model wave), LongChat, and the Arena technical report (arXiv:2403.04132)Also the home of MT-Bench (the multi-turn judge-based eval, arXiv:2306.05685) and the LMSYS-Chat-1M and Chatbot Arena Conversations datasets - the leaderboard and the eval data are as consequential as the serving codeA distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs, built around loading many different models in one process so they can be compared under identical conditions39556 stars, Apache-2.0, last push 2026-05-01
Hardware caveats
RELEASES STOPPED IN FEBRUARY 2024. Latest release is v0.2.36 from 2024-02-11, while the last push was 2026-05-01. Commits continue (mainly to chat templates and model definitions) but the release train stopped over two and a half years ago. That split is the evidence for deprecated: maintained as a template registry, dead as an engineIts own function migrated off the serving side. Chatbot Arena, which the README names as FastChat's flagship deployment, now runs at lmarena.ai and the fighting is on the arena side; the serving system is not what the project is for any moreNo dedicated inference kernels. FastChat serves many models in one process for comparison, which is the opposite design pressure from a throughput engine - it optimises for 'fair comparison across models', not tokens per second per dollarContinuous batching is present but without paged KV, prefix caching, speculative decoding or chunked prefill in the sense that matters. This is the pre-vLLM generation and it shows in the feature list1983 open issues against a project whose releases stopped in 2024 - the backlog is effectively unmaintainedNot a production serving stack and was never positioned as one; the web UI and multi-model-in-one-process design is a research and evaluation toolBackend support is whatever it can load: safetensors HF models and llama.cpp/GGUF, which is why GGUF appears in supported_formats here and nowhere else in this slice
Best for
Reproducing the leaderboard-era methodology - MT-Bench, judge-based evaluation, side-by-side battles - which is still the closest thing to a community standard for chat-model comparisonReading the origin of the open-weights chat model ecosystem; the Vicuna release notes are the primary sourceA multi-model-in-one-process comparison harness when you genuinely need identical conditions across many modelsGetting the canonical conversation templates for older open chat models, which remain the main thing people use it for
Avoid for
Production serving - releases stopped 2024-02-11 and this was never a throughput enginePerformance work; the multi-model comparison design works against throughput optimisationNew model support beyond chat templates; actual inference work moved to vLLM/SGLangAnything where an unmaintained 1983-issue backlog is acceptable
notes
Included because it is the highest-leverage historical record in this slice even though it is not a competitive engine: FastChat plus Chatbot Arena determined which open models the field used, and MT-Bench plus the LMSYS-Chat-1M data are still the evaluation substrate. Deprecated status is based on the release/commit split (v0.2.36 in 2024-02 vs last push 2026-05), not on an announcement. arXiv:2306.05685 (MT-Bench) title verified via the arXiv API. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2025-06-02 (16.1 months before this record), latest release v0.2.36 published 2024-02-11, 246 contributors (distinct committers including anonymous), open_issues_count 1045 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 39555 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
Included because it is the highest-leverage historical record in this slice even though it is not a competitive engine: FastChat plus Chatbot Arena determined which open models the field used, and MT-Bench plus the LMSYS-Chat-1M data are still the evaluation substrate. Deprecated status is based on the release/commit split (v0.2.36 in 2024-02 vs last push 2026-05), not on an announcement. arXiv:2306.05685 (MT-Bench) title verified via the arXiv API. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2025-06-02 (16.1 months before this record), latest release v0.2.36 published 2024-02-11, 246 contributors (distinct committers including anonymous), open_issues_count 1045 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 39555 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.