Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads (TetriInfer)

A production cluster mixes downstream tasks with very different prompt and output length distributions, and co-locating them means prefill-heavy and decode-heavy requests interfere destructively.

Three pillars. (1) Partition prompts into fixed-size chunks so the accelerator stays near its compute-saturated limit. (2) Disaggregate prefill and decode instances so each runs independently. (3) A two-level scheduling algorithm augmented with PREDICTED resource usage, to avoid decode scheduling hotspots.

Per the arXiv abstract: uses 38% less resources while lowering average TTFT by 97% and average job completion time (JCT) by 47%. Baselines are not named in the abstract; the metric is performance per dollar for mixed downstream workloads, which is why the resource reduction and the latency reduction can both be large at once.

Venue NOT verified — the arXiv comment field is empty and no proceedings page was confirmed. Recorded as arxiv-preprint deliberately. Naming trap: the system is TetriInfer, but the arXiv TITLE is 'Inference without Interference'; searching for 'TetriInfer' in arXiv titles returns this paper's successors and unrelated work, so search the body text, not the title. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.