Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

DIRECT COUNTER-EVIDENCE TO 'LOSSLESS' SPECULATIVE DECODING CLAIMS. This paper directly disputes the common claim that speculative decoding is 'lossless' (distribution-preserving). Key findings: (1) 'Lossy verification schemes... silently rewrite the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality.' (2) The paper unifies existing methods into two categories: truncation-based verification and collaborative verification. (3) For truncation-based methods: 'performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion.' (4) For collaborative verification: 'well-designed relaxation principles, namely overshoot suppression and supervision quality, matter far more than the linear interpolation between draft and target.' This is a direct challenge to papers like EAGLE-2 which claim 'lossless acceleration algorithm.' The dispute is UNRESOLVED - the lossy verification paper shows quality degradation but does not claim all speculative decoding is lossless; it shows that lossy variants exist and can degrade quality.