Large language model (LLM) autoscaling has two different control loops.
Swipe or scroll sideways to read the full diagram.
Queue/latency pressure → HPA/KEDA → desired replicas increase → Kubernetes creates Pod. If no GPU node has room, Pod stays Pending → node autoscaler provisions GPU node → Pod schedules → model loads/readiness → new serving capacity.
Replica demand and placement
The Horizontal Pod Autoscaler (HPA) and Kubernetes Event-driven Autoscaling (KEDA) adjust workload demand; a node autoscaler supplies suitable machines. The infrastructure scaling reference explains those control loops and scheduling constraints.
For an illustrative replica using two graphics processing units (GPUs), six desired replicas require twelve allocatable GPUs with the required placement. If the cluster can place only four replicas, the remaining two stay Pending. Their future capacity cannot satisfy the current queue, and newly provisioned replicas still need model startup.
Why GPU utilization alone can mislead
High GPU utilization can be a sign that an inference engine is doing its job well. Efficient batching may keep an accelerator busy across both healthy and overloaded traffic levels.
Amazon Web Services (AWS) Elastic Kubernetes Service (EKS) inference guidance recommends queue depth as a strong leading signal, with 95th-percentile (p95) latency, time to first token (TTFT), and key-value (KV) cache utilization as additional evidence.
Scaling should react to unserved demand or service-level objective (SLO) pressure, not only to a busy resource.
Useful signals
| Signal | What it tells you |
|---|---|
| Waiting requests / queue depth | Demand is exceeding current immediate service capacity |
| TTFT / request latency | Users are feeling queue or prefill pressure |
| KV-cache utilization | Active request state is approaching memory capacity |
| Tokens/sec | How much useful generation work the fleet is completing |
| GPU utilization | Hardware is busy, but not necessarily whether more capacity is needed |
Thresholds come from load testing
Do not pick “queue > 10” because it sounds reasonable.
Load test one replica with realistic prompt/output distributions, increase offered load, and observe where TTFT, time per output token (TPOT), errors, and queue depth stop meeting the target.
That gives you a capacity curve and an evidence-based scaling threshold.
Autoscaling still has a hard limitation: a new GPU replica is not useful until it exists and has loaded the model.
Continue to Cold starts and warm capacity.