Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

LLM-based applications are DAGs of many dependent LLM calls, but public serving APIs expose only a request-level interface, so the scheduler cannot see application structure and optimises each request in isolation.

Introduce Semantic Variable, a unified abstraction that annotates an input/output variable in a request's prompt and creates the data pipeline when connecting multiple LLM requests — a way to program LLM applications. Exposing Semantic Variables to the service lets it run conventional data-flow analysis to uncover correlation ACROSS requests, which opens an optimisation space the request-level API cannot reach.

Per the arXiv abstract: 'up to an order-of-magnitude improvement for popular and practical use cases of LLM applications'. This is an END-TO-END application-level metric for DAG-shaped agent workloads, not a serving-engine throughput number, and the abstract names no baseline system. Do not compare it against a serving engine's throughput figure — the denominator is a different thing.

Venue verified twice (arXiv comment 'To appear on USENIX OSDI 2024' plus the OSDI '24 program and presentation pages). Adoption research-only, and this one is worth flagging: agentic DAG serving has since become a large area, but no general engine adopted Parrot's application-level interface — engines still take request- or batch-level APIs. The functionality partially reappeared as tracing APIs rather than as Parrot's scheduler. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: osdi has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.