What to measure in LLM serving

Large language model (LLM) serving metrics are most useful when grouped by the question they answer.

Is the user waiting too long to see anything?

Time to first token (TTFT) measures from sending the request until the first streamed output arrives.

It can include:

network / gateway
queue waiting
prompt prefill
scheduler delay
first decode step

A poor TTFT therefore does not identify the bottleneck by itself. It tells you where to start.

Once generation begins, is it progressing quickly?

Time per output token (TPOT) amortizes generation time after the first token across the remaining generated tokens.

Inter-token latency (ITL) measures gaps between streamed outputs.

vLLM notes that terminology differs across tools, especially with speculative decoding, so compare definitions and measurement points rather than only metric names.

Is demand exceeding immediate serving capacity?

Watch:

• waiting requests / queue depth;
• queue time;
• running requests;
• rejected/admission-limited requests.

A growing queue with healthy individual decode performance usually points toward capacity or routing rather than a broken kernel.

Is request state filling accelerator memory?

Watch key-value (KV) cache utilization or available cache blocks/tokens.

This tells you whether the active sequence mix is approaching the memory budget that supports concurrency.

Prompt and generation length distributions should be tracked alongside it; a fleet can become cache-bound even when request count has not changed.

Is the hardware doing useful work?

Graphics processing unit (GPU) utilization, video memory (VRAM) occupancy, kernel behavior, power, and communication metrics help at the infrastructure layer.

Do not use a busy GPU as proof that latency is healthy. A saturated queue and an efficiently busy GPU can exist at the same time.

What is the service costing?

Useful economic signals include:

input tokens
output tokens
tokens/sec/GPU
cost/request
cost/token
GPU-hours by model
cache hit rate
model/route selection

A performance improvement matters differently if it saves 5 ms versus if it doubles useful token throughput on the same fleet.

A compact layer map

Next: Debugging a slow or saturated LLM service.