vLLM vllm/v1/sample/sampler.py (main) - the per-step sampler chain and its logit dtype conversions

Fields

id

sources/flop3-vllm-sampler-source

URL

https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/sample/sampler.py

Publisher

vllm-project/vllm

Kind

repo

Accessed

2026-10-03

notes

Every step: optional raw logprobs computation (a clone or a .to(float32) of the whole logits tensor), an unconditional logits.to(torch.float32) - so the [B, V] logits are materialised in fp32 every step even when the model produced bf16 - then apply_logits_processors, then sample, then a .long() cast and a .to(torch.int32) cast on the sampled ids. The dtype conversions alone mean the V-wide tensor is written and re-read at 4 bytes/element instead of 2 on the sampling path.

Notes

Every step: optional raw logprobs computation (a clone or a .to(float32) of the whole logits tensor), an unconditional logits.to(torch.float32) - so the [B, V] logits are materialised in fp32 every step even when the model produced bf16 - then apply_logits_processors, then sample, then a .long() cast and a .to(torch.int32) cast on the sampled ids. The dtype conversions alone mean the V-wide tensor is written and re-read at 4 bytes/element instead of 2 on the sampling path.

Cross-references

This record references no other records.