vLLM vllm/v1/sample/ops/penalties.py (main)
repo
Fields
id
sources/flop3-vllm-penalties-sourcePublisher
vllm-project/vllm
Kind
repo
Accessed
2026-10-03
notes
Shows the per-step host-side cost of presence/frequency/repetition penalties as shipped. Every step, apply_all_penalties takes the per-request output_token_ids Python lists, pads them into a CPU int64 tensor on PINNED memory with pad=vocab_size, and uploads it with .to(device, non_blocking=True) - so the payload is max_output_len x batch x 8 B, rebuilt and re-uploaded every step, and it grows with how long each sequence has been generating. It also fills -1 placeholders (async scheduling rows that will not be penalised) with vocab_size to keep the scatter valid. The file's own comment: 'The penalties implementation is currently quite inefficient and will be reworked anyhow.'
Notes
Shows the per-step host-side cost of presence/frequency/repetition penalties as shipped. Every step, apply_all_penalties takes the per-request output_token_ids Python lists, pads them into a CPU int64 tensor on PINNED memory with pad=vocab_size, and uploads it with .to(device, non_blocking=True) - so the payload is max_output_len x batch x 8 B, rebuilt and re-uploaded every step, and it grows with how long each sequence has been generating. It also fills -1 placeholders (async scheduling rows that will not be penalised) with vocab_size to keep the scatter valid. The file's own comment: 'The penalties implementation is currently quite inefficient and will be reworked anyhow.'
Cross-references
This record references no other records.