One running large language model (LLM) server can answer requests. A production serving system has to answer a different question: how do we keep latency and throughput acceptable when requests arrive concurrently and individual requests have very different token lengths?
In this chapter
- From one model to a serving systemAdd the layers only when the problem requires them.
- Inference enginesPlace vLLM, SGLang, Triton, and TensorRT-LLM by the work each system owns.
- Replicas and parallelismCompare splitting one model across GPUs with duplicating serving capacity.
- Routing and serving many usersConnect load distribution, cache locality, per-replica scheduling, and horizontal scale.
An inference engine schedules model execution. The serving platform gets traffic to enough healthy inference replicas and manages the capacity around them.
For several distinct models, multi-model placement compares shared servers, isolated deployments and compatible adapters, including memory arithmetic.