Clipper: A Low-Latency Online Prediction Serving System

ML frameworks handle training and not deployment, so every application reinvents a low-latency prediction serving layer over its own model, inconsistently and without batching or latency control.

Interpose a general-purpose serving layer between the app and the models: a front-end that routes, batches and caches predictions, a model container interface so any framework can be plugged in, and an adaptive batching layer that meets a latency target by varying batch size with load.

Per the arXiv abstract: Clipper achieves COMPARABLE throughput and latency to TensorFlow Serving while additionally enabling model composition and online learning. The paper evaluates four ML benchmark datasets. Note this is a parity claim against TensorFlow Serving, NOT a speedup — Clipper's value proposed in the paper is the model-composition and online-learning capability plus framework-agnostic deployment, so there is no large speedup multiple to cite.

Venue verified on the USENIX NSDI '17 program page (session 'Big Data Systems'). Adoption genuinely in-production as a design lineage: Clipper's batch-level front-end/cache abstraction is the direct ancestor of TensorFlow Serving, and NVIDIA cites it as the origin of Triton Inference Server's architecture. IMPORTANT NAMING TRAP: 'Clipper2' is the Triton Inference Server, a different project that reuses the name; do not treat this record as covering Triton — Triton has no peer-reviewed paper, so it is cited via the repository source pap-sys-clipper2-arxiv instead. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: nsdi has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.