inference/research
contestedmedium · 0.80majorupdated 2026-10-04

A logits processor that reports is_argmax_invariant() is never called under greedy decoding - the sampler returns before that stage

[major] security

Fields

id
gotchas/sec-logit-mask-argmax-invariant-skipped-under-greedy
Class
security
Severity
major
Concepts
logit-maskingsafety-decodinggreedy-decodingsamplingdecode-loopcontinuous-batching
Symptom
You ship a safety logits processor on a model served with temperature 0. It passes tests on sampled traffic. Callers who use greedy decoding get unfiltered output, and the only symptom is that the filter appears to work most of the time rather than failing loudly.
Root cause
vLLM's sampler orders its stages explicitly: logit processors that are NOT argmax-invariant run before sampling, but the argmax-invariant group (by default min_p) is applied INSIDE the random-sampling branch, after temperature. When the whole batch is greedy, sample() returns immediately after greedy_sample(logits) and the argmax-invariant group never executes. The interface makes each processor declare is_argmax_invariant() and the sampler trusts that declaration, so a processor that is wrong about its own argmax-invariance is silently skipped on greedy requests.
Workaround
In vLLM, never implement a safety transform as a processor reporting is_argmax_invariant() = True. If the transform can change which token is argmax it must report False. Treat all_greedy as a distinct configuration and test it explicitly rather than assuming the filter is uniform across sampling settings. Note the same batching hazard recorded elsewhere in this repo: greedy_sampled and random_sampled are merged with torch.where over one batch, so a partly-greedy batch mixes both paths.
notes
STATUS CONTESTED, deliberately. The ordering and the early return are read directly from the vLLM sampler source and the interface declaration, which is solid. What is NOT established from a source is the real-world incidence: no published report quantifies how often mixed greedy/random batches occur in production, so treat the mechanism as certain and the prevalence as unmeasured. SCHEME RECLASSIFIED (2026-10-04): was 'framework'.

Notes

STATUS CONTESTED, deliberately. The ordering and the early return are read directly from the vLLM sampler source and the interface declaration, which is solid. What is NOT established from a source is the real-world incidence: no published report quantifies how often mixed greedy/random batches occur in production, so treat the mechanism as certain and the prevalence as unmeasured. SCHEME RECLASSIFIED (2026-10-04): was 'framework'.

Cross-references