dudeperf3ct blog - Where vLLM cold-start time goes on GKE (phase-by-phase measurement)

Methodology stated verbatim: 'I define startup time as the duration from deployment creation to the first successful response from /v1/chat/completions. Unless stated otherwise, all measurements are taken on a GKE cluster in europe-west1 with a single L4 GPU node.' Model Mistral 7B Instruct v0.3, vLLM pinned to 0.18 via vllm/vllm-openai:v0.18.0, weights pulled from HuggingFace Hub, reproducible repo at github.com/dudeperf3ct/benchmark-vllm-startup. Network baseline quoted: 'an effective rate of ~397 MB/s for a cluster in the europe-west1 region. That number matters because remote artifact download is a major part of cold start, and it varies significantly by environment.' Image size and pull quoted: the vllm-openai:v0.18.0 image 'takes up 28 GB of space and in this setup it takes roughly 236 seconds to download the image onto the node', and 'The GKE node caches the vLLM image pulled so for the subsequent restart of the deployment, the time to pull vLLM image is practically zero' - the cold vs warm distinction is exactly 'has this node ever run this image before'. Enumerates the pre-server work: Initializing the engine, model downloading and loading, torch.compile, CUDA graph capture, server startup. Repo's existence makes this the most reproducible of the cold-start measurements in the corpus.