Routing and serving many users

There is usually no single graphics processing unit (GPU) answering an entire large language model (LLM) product’s traffic. Concurrency comes from horizontal replicas and from sharing each replica efficiently among several active requests.

Horizontal replicas multiply independent serving capacity

If one replica reaches its useful concurrency limit, another replica gives the fleet another key-value (KV) cache pool, another scheduler, and another set of accelerator resources.

A router spreads new requests across those serving units.

How many replicas are required depends on measured service capacity: prompt lengths, output lengths, model size, quantization, GPU type, latency target, and scheduler configuration all matter.

Each replica still serves several requests

Inside a replica, continuous batching lets active sequences share model execution. The inference engine allocates KV-cache blocks and decides which work participates in upcoming execution steps.

That means a model replica is not equivalent to a traditional worker that handles exactly one request at a time.

Basic routing can be simple

Round-robin can be completely reasonable when replicas are homogeneous and requests are similar.

Least-loaded or queue-aware routing becomes useful when the work varies substantially. A request routed to a replica with a deep queue may wait even while another replica has room.

At larger scale, routing can account for:

• model/version availability;
• region and network latency;
• waiting/running request counts;
• estimated work;
• tenant or priority policy;
• prefix-cache locality.

Cache-aware routing is an optimization, not the starting point

Suppose several requests share a large prefix and replica A already has reusable prefix state. Sending the next related request to A can avoid prefill work.

But blindly preferring cache locality can overload A while B sits idle. A real policy has to balance locality against current load and latency.

This is a good example of why an LLM router can become more specialized than an ordinary round-robin load balancer without making ordinary load-balancing fundamentals obsolete.

Inference-aware routing uses serving state

As the fleet grows, routing can use more than endpoint availability.

Useful signals can include:

• waiting or running requests;
• KV-cache pressure;
• reusable prefix state;
• model, version, or adapter availability.

That gives a progression:

round robin → load-aware routing → inference-state-aware routing.

A small homogeneous fleet may still be better served by simple load balancing. More state can improve placement decisions, but it also makes the routing layer more complex and more dependent on fresh serving metrics.

What happens when every replica is busy?

Routing alone cannot create capacity.

Requests begin waiting, queue depth increases, time to first token (TTFT) rises, and eventually the system must choose among:

• admit the work and let it wait;
• reject or shed load;
• route to another model/region;
• add replicas;
• add GPU machines if the cluster lacks space.

Those choices move us from serving into production traffic control and autoscaling.

Before that, the next chapter answers a more basic deployment question: what actually gets deployed.