Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

Existing multi-tenant isolation equalises client THROUGHPUT over the long run (queueing/batching fairness) but gives no LATENCY guarantee, so a well-behaved client can still see its token-level latency badly degraded by a noisy neighbour.

Introduce a delta-token fairness guarantee: for a well-behaved client, a token that takes d time units in isolation is produced within d + delta under multi-tenant execution. The scheduler enforces per-token deadlines while bounding delay from GPU compute sharing AND from the extra delay that shared KV caching introduces — the KV sharing term is the hard part, since it couples tenants without any scheduling action.

NOT RECORDED as a multiple — the abstract states the delta-token guarantee qualitatively without a verified headline figure carrying baseline and settings. The paper's contribution is the GUARANTEE, not a speedup: throughput fairness and latency isolation are being traded against each other and a 'speedup' framing would be misleading here.

Venue arxiv-preprint: no venue in the arXiv comment. Included because it names the failure mode every shared serving tier hits — long-run throughput fairness is NOT latency isolation — and because a delta-parameterised latency guarantee is the kind of claim that, if it holds, changes what you can sell as a multi-tenant service. The Zaharia/Krieger Berkeley lineage makes the provenance credible but the result is unreviewed as of this record. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.