FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems

LLM agents can generate GPU kernels, but a kernel an agent writes cannot be used unless it is specified identically, benchmarked with correctness awareness, and actually deployed - so agent kernel generation has no closed loop to production.

A closed-loop harness. FlashInfer Trace is a unified schema describing kernel definitions, workloads, implementations and evaluations, which is the piece that lets an agent, a benchmark and an engine agree. On top of it: a curated dataset built from real serving traces, a correctness-and-performance-aware benchmarking framework, a public leaderboard, and an apply() dynamic-substitution mechanism that injects the best-performing kernel into production engines (SGLang, vLLM) at runtime.

NOT RECORDED AS A MULTIPLE. The abstract makes no speedup claim about itself - it is an infrastructure contribution (framework, dataset, leaderboard, deployment mechanism). Its speedup-relevant content is indirect: it reports trade-off comparisons across GPU programming languages used by agents and evaluations of agent capability limits, but no baseline-plus-setting throughput figure. Recording a number here would mean inventing one.

arXiv ID 2601.00227 verified via abs-page meta tags on 2026-10-03; title matches. CORRECTION TO THE SOURCE RECORD: it says 'arXiv preprint; no journal_ref' with no venue evidence, but OpenReview returns venue='MLSys 2026', venueid='MLSys.org/2026/Conference' for the exact title. venue=mlsys, year=2026. DEDUP DECISION: KEPT as a separate record from flashinfer-paper. They are different works: different arXiv IDs (2601.00227 vs 2501.01005), different author teams (Xing/Zhai/Jiang/Dong vs Ye/Chen/Lai/Lin), and different objects - one is the attention engine, the other is the benchmark and deployment harness for agent-generated kernels. flashinfer-paper is 'the engine', flashinfer-bench is 'the harness that feeds kernels into the engine', and the pairing is the interesting fact. Adoption research-only: the harness is a benchmarking framework, not an inference path, so nothing in any serving engine executes it during a request. A benchmark record arguably belongs under type=benchmark, but this repo's paper type is defined as 'a contribution with a method, a claim, and an adoption status', and this has all three - kept as paper and the tension is noted. hardware_relevance empty because the framework is hardware-agnostic; the GPUs matter only through the traced workloads. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: mlsys has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.