A trained model becomes useful to an application when callers can send inputs and receive predictions. Both conventional machine learning (ML) models and large language models (LLMs) need versioned artifacts, deployments, healthy replicas, and monitoring.
The request lifecycle changes the serving choices. A conventional prediction often finishes in one bounded pass. An autoregressive LLM processes a prompt, generates output over many steps, and keeps per-request attention state while it does so.
In this chapter
- Traditional ML serving vs LLM servingSee which deployment fundamentals carry over and which constraints change for token generation, batching, graphics processing unit (GPU) memory, and scaling.
- From trained model to prediction serviceChoose execution mode, package fitted transforms, download pinned artifacts at runtime and release a ready service.
- Multi-model serving, grouping and memory capacityCount loaded copies, choose isolation and budget resident plus request memory.
Continue into LLM inference for the execution details or Infrastructure & DevOps for the shared deployment mechanics.