id
engines/niche-ray-serve
Design goal
multi-modelmodel-compositionease-of-useflexible-scheduling
Scheduling
request-routingdynamic-batchingfractional-gpu
Backends
none (orchestration layer - drives any Python callable)
Formats
passthrough (does not load weights - wraps Python callables)
Notable features
Framework-agnostic model serving - wraps any Python callable as a deploymentModel composition via DeploymentHandles - build complex inference pipelines in PythonDynamic request batching for LLM workloadsMulti-node/multi-GPU serving with flexible schedulingFractional GPUs - share GPU resources across deploymentsResponse streaming for LLM workloadsBuilt on Ray core - scales to many machines44k stars, active development
Hardware caveats
Python-only - no Go or Rust coreFractional GPU sharing is not the same as MPS - it is Ray's own scheduling, not CUDA MPSModel composition adds latency - each hop in the composition graph is a network callNot Kubernetes-native - runs on Ray clusters, not K8s (though KubeRay provides a bridge)Dynamic batching is per-deployment, not global - no cross-deployment batchingNo built-in prefix-aware routing or KV cache management
Best for
Multi-model serving with complex composition (e.g., LLM + reranker + guardrail)Teams already using Ray for training who want to use the same framework for servingPython-heavy inference pipelines that need flexible compositionFractional GPU sharing across many small models
Avoid for
Kubernetes-native deployments - Ray Serve is not K8s-nativeLow-latency single-model serving where the composition overhead mattersTeams needing prefix-aware routing or KV cache managementNon-Python inference engines
notes
Ray Serve is a model serving library built on Ray. It is framework-agnostic and wraps any Python callable as a deployment. The key differentiator is model composition - you can build complex inference pipelines (LLM + reranker + guardrail) as a graph of deployments. Ray Serve supports dynamic request batching, multi-node/multi-GPU serving, and fractional GPU sharing. However, it is not Kubernetes-native (KubeRay provides a bridge), and it lacks LLM-specific features like prefix-aware routing and KV cache management. It is best suited for multi-model composition workloads rather than high-throughput single-model LLM serving. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-03 (0.0 months before this record), latest release ray-2.59.0 published 2026-10-02, 1751 contributors (distinct committers including anonymous), open_issues_count 3555 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 43970 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.