vLLM's --gpu-memory-utilization is a fraction of TOTAL VRAM, so KV cache block allocation fails even with memory visibly free

Engine init fails with: ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine. Reported cases include llama-2 70B 4-bit AWQ on 4x A10G (worked on an older vLLM, fails from 0.2.5) and defog/sqlcoder2 at --gpu-memory-utilization 0.4 with only 8192 max tokens on a single GPU - a small model that should fit easily. From vLLM 0.2.5 onward the same setups stopped starting.

gpu_memory_utilization is a fraction of TOTAL device memory, not of memory left after weights. The engine profiles weights, subtracts activations and workspace, then requires the remaining budget to hold a whole number of KV cache blocks at --max-model-len and --max-num-batched-tokens. If the residual is not enough for even one block, init aborts with 'No available memory for the cache blocks' even though nvidia-smi shows plenty free - the memory is reserved for the KV pool and reported as used, so 'plenty free' and 'no blocks fit' coexist.

Raise --gpu_memory_utilization (the error names this), and/or lower --max-num-batched-tokens and --max-model-len so each request's KV working set is smaller. Check the startup log's reported KV cache size and block count to see how many blocks you actually got. In containers, --shm-size and --ipc=host interact with the same budget.

The reporter's docker invocations also set NCCL_IGNORE_DISABLED_P2P=1 and --shm-size=10.24gb, so container shared-memory sizing is entangled with the failure - treat that as a second thing to check, not the established cause. Long-open issue (2023-12-24 to 2025-08-14) collecting several distinct complaints, so the individual configurations do not share one mechanism beyond the utilization semantics above.