inference/research
verifiedhigh · 0.85updated 2026-10-03

vLLM PR #30351: cache added_vocab to avoid per-token overhead (DeepSeek-V3.2 tokenizer mode)

repo

Fields

id
sources/flop3-vllm-pr-30351-detokenizer-added-vocab
Publisher
vllm-project/vllm
Kind
repo
Accessed
2026-10-03
notes
Per-token detokenisation starving the event loop. detokenize_incrementally runs on every generated token and evaluates len(tokenizer); DeepseekV32Tokenizer.__len__ returned vocab_size + len(self.get_added_vocab()), and get_added_vocab() called the native tokenizer's get_added_vocab() every time, showing up as BTreeMap::insert in the native layer. Because the token id check is `0 <= new_token_id < len(tokenizer)`, EVERY generated token paid a native vocab-map rebuild/lookup on the GIL-holding API-server main thread. Reported symptom: the main thread is busy in detokenisation, CPU stays persistently high, and /health plus other HTTP endpoints hang - i.e. the per-token host cost escapes the decode loop entirely. Fix: prefetch the added-vocab dict and its size once at tokenizer init. Config in the repro: DeepSeek-V3.2 tokenizer-mode, max-model-len 163840, max-num-seqs 256, chunked prefill, DP 8, expert parallel.

Notes

Per-token detokenisation starving the event loop. detokenize_incrementally runs on every generated token and evaluates len(tokenizer); DeepseekV32Tokenizer.__len__ returned vocab_size + len(self.get_added_vocab()), and get_added_vocab() called the native tokenizer's get_added_vocab() every time, showing up as BTreeMap::insert in the native layer. Because the token id check is `0 <= new_token_id < len(tokenizer)`, EVERY generated token paid a native vocab-map rebuild/lookup on the GIL-holding API-server main thread. Reported symptom: the main thread is busy in detokenisation, CPU stays persistently high, and /health plus other HTTP endpoints hang - i.e. the per-token host cost escapes the decode loop entirely. Fix: prefetch the added-vocab dict and its size once at tokenizer init. Config in the repro: DeepSeek-V3.2 tokenizer-mode, max-model-len 163840, max-num-seqs 256, chunked prefill, DP 8, expert parallel.

Cross-references

This record references no other records.