Where model weights live

Container startup is only the beginning of model startup. The checkpoint still has to become available to the process and reach graphics processing unit (GPU) memory.

Common deployment patterns

Bake weights into the image

This makes the image self-contained, but a large checkpoint can turn every image pull into a huge transfer and makes model updates inseparable from image updates.

It can be useful for smaller or tightly controlled artifacts, but it is rarely the only option.

Let the serving framework download them

A command such as:

vllm serve <model-id>

can resolve a remote model and use a local cache. The first start on a fresh machine may download the checkpoint; later starts can reuse the cache if that cache survives.

The cache helps only when it survives the Pod lifecycle. An ephemeral container filesystem disappearing with the Pod gives different startup behavior from a persistent or node-local cache.

Mount shared or persistent storage

A Pod can mount a PersistentVolume or network filesystem and start the inference engine against a path such as:

/models/my-model.

This separates model distribution from the application image, but storage throughput and startup contention now matter.

Use an init container

An init container can fetch or verify model files into a shared volume before the main inference container starts.

That keeps download/authentication/checksum logic separate from the serving process. Kubernetes guarantees that init containers run to completion before the app containers begin.

Preload node-local storage

For very large models, teams may put frequently used checkpoints on local non-volatile memory express (NVMe) storage attached to GPU nodes. A new replica scheduled onto a warm node can avoid remote model transfer.

That improves startup at the cost of cache management and placement complexity.

“Present on disk” is not “ready”

Even with a perfect local cache, startup can still require:

1. reading weight shards;
2. creating tensors;
3. initializing CUDA and distributed communication;
4. moving or mapping weights into accelerator memory;
5. compiling/capturing runtime artifacts where applicable;
6. sizing the key-value (KV) cache;
7. warming the serving path.

The storage layer therefore solves only part of cold start.

vLLM’s distributed-serving guidance explicitly recommends pre-downloading Hugging Face models on every node or storing them on a distributed filesystem accessible at the same path rather than relying on every process to fetch them independently.

Next: From container start to ready replica.

The shared model deployment reference shows an image without weights and the pinned-download, verification and publication contract. This page develops checkpoint placement and GPU loading.