Large language model (LLM) serving still uses familiar distributed-systems fundamentals: load balancing, replication, bounded queues, backpressure, health checks, rollouts, capacity planning, observability, and failure domains.
What changes is the resource profile. Model replicas are large, graphics processing unit (GPU) / accelerator-bound, stateful during generation, and expensive to start.
In this chapter
- End-to-end LLM serving architecturePlace the gateway, router, queue, inference engine, Kubernetes, and GPU node scaler by responsibility.
- Rate limits, concurrency, and backpressureSeparate customer policy from actual serving capacity and overload control.
- Autoscaling LLM inferenceSeparate replica scaling from GPU-node provisioning and choose useful demand signals.
- Cold starts and warm capacitySee why model loading changes autoscaling and capacity planning.
- Failures and recoveryDefine the real failure unit when a replica spans several GPUs or machines.
- Serving trade-offsConnect latency, throughput, quality, context length, and cost.
- AI platform tool choices and replacement triggersCompare search, inference, orchestration and storage alternatives by operating constraint.
- Prompt releases, context budgets and application cachingBind application versions, allocate context and invalidate cached answers safely.
- Streaming voice agents, turn-taking and recoveryTrace first audio, interruptions, cancellation and recoverable tool actions.