inference/research
verifiedhigh · 0.85updated 2026-10-03

vLLM issue #40896: v1 prefix caching - first request differs from subsequent identical requests at temperature=0

forum

Fields

id
sources/pd-gh-vllm-40896-prefix-cache-cold-vs-warm
Publisher
vllm-project/vllm
Kind
forum
Published
2026-04-26
Accessed
2026-10-03
notes
The canonical cold-path-vs-cache-hit-path divergence, independent of any external cache tier. With --enable-prefix-caching (v1 default for supported models) on Qwen3-8B, H100 80GB SXM, vLLM 0.19.0: the same /v1/completions request at temperature=0 returns output A on a freshly started server and output B on runs 2..N, with B stable across runs; restarting returns to A. Disabling prefix caching (--no-enable-prefix-caching) makes output deterministic across all runs. Note the config trap the reporter documents: in v1, omitting --enable-prefix-caching does NOT disable it, because the default is model_config.is_prefix_caching_supported. Reporter notes warm-up rounds appear to mask it. So the hit path and the cold path are not bit-identical here.

Notes

The canonical cold-path-vs-cache-hit-path divergence, independent of any external cache tier. With --enable-prefix-caching (v1 default for supported models) on Qwen3-8B, H100 80GB SXM, vLLM 0.19.0: the same /v1/completions request at temperature=0 returns output A on a freshly started server and output B on runs 2..N, with B stable across runs; restarting returns to A. Disabling prefix caching (--no-enable-prefix-caching) makes output deterministic across all runs. Note the config trap the reporter documents: in v1, omitting --enable-prefix-caching does NOT disable it, because the default is model_config.is_prefix_caching_supported. Reporter notes warm-up rounds appear to mask it. So the hit path and the cold path are not bit-identical here.

Cross-references

This record references no other records.