inference/research
verifiedhigh · 0.95updated 2026-10-04

vLLM vllm/v1/sample/sampler.py (Sampler.forward / Sampler.sample) source

repo

Fields

id
sources/sec-vllm-sampler-pipeline-source
Publisher
vLLM project
Kind
repo
Published
2026-10-04
Accessed
2026-10-04
notes
The exact order a safety transform must survive in: allowed-token-ids whitelist -> bad-words exclusion -> non-argmax-invariant logit processors (min_tokens, logit_bias) -> penalties -> sample (greedy or temperature -> argmax-invariant processors such as min_p -> top-k/top-p). Two things this makes concrete: (1) with all_greedy the engine returns BEFORE the argmax-invariant processor stage, so any safety transform declaring is_argmax_invariant() is silently skipped under greedy decoding; (2) if a request's transform is skipped, the sampler mixes argmax and non-argmax rows in one batch via torch.where, which is exactly the batch-invariant-batching hazard documented elsewhere in this repo.

Notes

The exact order a safety transform must survive in: allowed-token-ids whitelist -> bad-words exclusion -> non-argmax-invariant logit processors (min_tokens, logit_bias) -> penalties -> sample (greedy or temperature -> argmax-invariant processors such as min_p -> top-k/top-p). Two things this makes concrete: (1) with all_greedy the engine returns BEFORE the argmax-invariant processor stage, so any safety transform declaring is_argmax_invariant() is silently skipped under greedy decoding; (2) if a request's transform is skipped, the sampler mixes argmax and non-argmax rows in one batch via torch.where, which is exactly the batch-invariant-batching hazard documented elsewhere in this repo.

Cross-references

This record references no other records.