Begin below the API
A request to a large language model (LLM) endpoint eventually becomes tensor work on a graphics processing unit (GPU) or another accelerator. Before thinking about Kubernetes or autoscaling, understand what one replica is trying to do.
The serving problems in this chapter follow directly from four facts:
- the prompt is processed before generation begins;
- output is generated autoregressively rather than all at once;
- active requests keep attention state in a key-value (KV) cache;
- GPUs become efficient when useful work from several requests can be scheduled together.
In this chapter
- How LLM inference actually worksSeparate prompt processing from autoregressive token generation.
- Prefill, decode, and KV cacheSee why context length changes latency, memory pressure, and concurrency.
- Continuous batching and request schedulingUnderstand how many active sequences share expensive GPU execution.
- Making inference faster and cheaperPlace quantization, prefix caching, PagedAttention, and speculative decoding against the bottleneck they address.
- Transformer attention, embeddings and fine-tuningRecall scaled attention, contrastive objectives and parameter-efficient adaptation.
- Tokenization, logits, sampling and generation behaviorBPE, subword tokens, softmax temperature, top-p/top-k and decoding cost.
- NLP preprocessing, representations and entity resolutionTrace normalization, sparse features, sentence pooling and canonical record identity.
- Choosing prompting, RAG, SFT and preference tuningMatch knowledge, behavior and preference gaps to evidence-driven interventions.
Compute, memory, waiting, communication and model quality create different symptoms. Identify which one constrains the request before choosing an optimization.