Reproducibility: most published inference performance numbers lack the methodology needed to reproduce them

A paper or blog post reports 'X tokens/sec' but doesn't state batch size, input/output length, warmup procedure, clock settings, or kernel version. The number is impossible to reproduce or compare.

Academic papers and vendor blog posts routinely omit load-bearing experimental details. Batch size alone can change throughput by 10x. Warmup (or lack thereof) can change results by 2-5x. Clock settings (locked vs boost) can change results by 20-50%. Kernel version can change results by 10-30%.

Before citing any throughput number, require: (1) exact engine and version, (2) model and quantization, (3) batch size and concurrency, (4) input/output token lengths, (5) warmup procedure, (6) clock settings (locked/boost), (7) hardware configuration. If any of these are missing, treat the number as a rough upper bound, not a reproducible measurement.

The repo's own benchmark schema requires methodology (batch size, prompt len, output len, concurrency) for exactly this reason. The short-benchmark-not-representative gotcha (already in repo) shows that even with methodology, short probes can misrepresent sustained performance.