Large language model (LLM) observability connects user-visible latency to queueing, token generation, key-value (KV) cache pressure, and graphics processing unit (GPU) state. A high GPU-utilization number by itself cannot tell you which of those paths is causing the delay.
In this chapter
- What to measure in LLM servingSeparate user latency, serving pressure, GPU state, and economics.
- Debugging a slow or saturated LLM serviceUse TTFT, queue time, prompt length, TPOT, KV cache, and startup state to narrow the bottleneck.
The useful path is short: start with the user-visible symptom, then follow the few signals that can explain it. Time to first token (TTFT) and time per output token (TPOT) split waiting/prefill from ongoing generation before lower-level metrics enter the investigation.
Serving telemetry explains runtime behavior. Observability, explainability, traceability and auditability extends the evidence to source revisions, model decisions, authorization and confirmed tool outcomes.