Automatic Prefix Caching - vLLM documentation

States the two-path structure precisely: APC 'caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part'. Critically it also states the limit that most prefix-cache claims omit: 'APC only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)', so it 'does not bring performance gain when vLLM spends most of the time generating answers (e.g. when the length of the answer is long)' or when queries share no prefix. That asymmetry is the whole reason a prefix-cached workload has two distinct flop profiles.