Tuesday, November 17, 2026 San Francisco Bay Area Notes by RSS
Asim Zafir

Reliability AI Serving Infrastructure Distributed Systems

Designing SLOs for a GPU inference path

Why tail latency, queue depth, and NCCL hiccups demand SLOs that ordinary web stacks do not.

By Asim Zafir · November 4, 2025 · 5 min read · slo, gpu, inference, observability, sre

The thing that breaks first is rarely the GPU

It is tempting to write the SLO for an inference service the same way we wrote it for a CRUD API: p95 request latency under 250 ms, 99.9% availability, 30-day rolling window. But the inference path is not a CRUD API. Tokens are streamed; latency is a function not of one request but of many concurrent decode steps colliding on shared SM partitions; failures rarely look like 5xx. By the time the response code is wrong, the system has already been broken for a long time, often in ways that selectively harm the most-expensive users.

The first SLO I'll defend, and the one that pulled the longest tail out of a 50k-node fleet, is one most teams do not write at all: time-to-first-token (TTFT) at p99.9. Anything coarser smooths over the shape of the failure.

Three latency surfaces, three different SLIs

A token-streaming endpoint actually has three distinct latencies, and each needs its own SLI:

  1. Time-to-first-token (TTFT) — the queue-and-prefill cost. This is dominated by admission control, KV-cache eviction, and prefill GEMM packing efficiency. When TTFT degrades, the cause is almost always upstream of the GPU.
  2. Inter-token latency (ITL) — the decode steady-state. A clean ITL is a sawtooth-free near-constant. Any irregular spikes are usually a tensor-parallel rank straggler, a CUDA graph cache miss, or a noisy neighbor inside the same SM partition.
  3. End-to-end completion time — the user-felt thing. It is a derived quantity, not an SLI you should alert on directly; alerting on it muddles two completely different failure surfaces.

The mistake I see most often is collapsing all three into one histogram and watching the p99 of the sum. You will detect outages, but you will not learn anything from them, and you will miss the slow degradations that quietly turn 19% of your premium users into the angriest cohort of the quarter.

SLO targets that survive contact with reality

Numbers worth defending, on the kind of stack I usually inherit (decoder-only model, TP=8 within node, PP=1, FP8 weights, paged-attention KV cache):

  • TTFT p99 ≤ 600 ms, p99.9 ≤ 1.2 s. The p99.9 is the budget for "did anything go badly today" and is the one that catches admission-control regressions.
  • ITL median ≤ 25 ms, ITL p99 ≤ 60 ms. The gap between median and p99 — not p99 itself — is the canonical "queue health" indicator.
  • Decode-pause rate ≤ 0.5 pauses-per-thousand-tokens. A "pause" is any inter-token gap above 150 ms. This catches NCCL collective hiccups that p99 ITL hides because they are vanishingly rare per token but devastatingly visible per request.
  • Successful-stream rate ≥ 99.95% measured at completed token count ≥ requested, not HTTP 2xx. A request that hung mid-stream and returned 200 is not a success.

Why decode-pause rate matters more than p99

A 5-second NCCL hang affects every request currently in the decode batch. If your average batch size is 64, that single pause produces 64 user-visible 5-second freezes — but those are 64 events out of millions of inter-token intervals over the window. Your p99 ITL will look unbothered. Your users, who saw a frozen cursor for five seconds during the most expensive part of their session, will not.

The shape of an outage in token streaming is rarely shaped like an outage on the latency histogram. Pick SLIs that match the shape.

The error budget is not what you think it is

An inference path does not get to spend the same 0.05% error budget that a CRUD API spends, because failures here are correlated. A 20-minute degradation on a single replica during peak hours can dump 4% of monthly traffic into the failure bucket. The error budget is not a smooth burn — it is a series of cliffs.

What I recommend, and have used in production: split the budget into a steady-state slow burn envelope and a spike envelope. The slow-burn budget is small (think 0.02%/30d) but tightly defended; the spike budget tolerates incidents up to a maximum severity at a maximum frequency. Burn-rate alerts on the slow-burn budget fire on multi-hour fast burns; spike-budget alerts fire only when the incident-frequency invariant is at risk. This lets you ship reasonable changes without having every nvidia-fabricmanager blip page someone.

Telemetry: histogram-native or it is not telemetry

If your ITL is being summarized as a min/avg/max per minute, you are not observing it; you are guessing about it. Three things I will not run an inference path without:

  • Native histograms (Prometheus 1.x, or HDR-style native exponential buckets). Pre-aggregated p99 from a downstream collector loses information you cannot recover.
  • A separate queue-time histogram emitted by the admission-control layer, distinct from the worker-side prefill histogram. When TTFT degrades, this is the first thing you look at.
  • NCCL collective-time histograms exported per-rank, with the rank label attached. Per-rank matters: a single straggler is the most common cause of decode pauses, and aggregating across ranks erases the signal.

The dashboards that get used

Three panels solve 80% of incidents:

  1. TTFT and ITL p50 / p99 / p99.9 over the last 24h, broken out by model and tenant. If a tenant is degraded but the fleet is not, this is the panel that tells you.
  2. Decode-pause rate over the last hour, with annotations from the deploy and config-change streams. Most "mystery" pauses are explained the moment the deploy line appears.
  3. Per-rank NCCL collective p99, sorted descending. The straggler floats to the top of this panel within seconds of a fabric problem.

What does not belong in your SLO

A long-running war is the temptation to put GPU utilization, KV-cache hit rate, or batch-fill ratio into the SLO. They are useful operational metrics, but they are inputs, not outputs. The user does not care if your KV cache is at 92% hit rate; they care that the next token arrives. Pick SLIs that measure what the user feels, then use the input metrics as the explanatory variables.

A short checklist before signing the SLO

  • Can you reproduce the decode-pause rate from raw per-token timestamps? If not, your sampling is too coarse.
  • Does each SLI have an associated burn-rate alert with documented receivers? An SLO without an alert pathway is documentation, not a contract.
  • Does the error budget policy say what gets stopped when the budget is exhausted, and who signs off on the freeze lift? If the answer is "we'll figure it out," it will be figured out badly, under stress, at 3 a.m.
  • Is the SLO denominated in user-meaningful events (requests, streams) and not worker-side events (token decode steps)? The user count is the only one that matters for the contract.

Closing

The right SLO for a GPU inference path is the one that catches the failure modes you actually have, denominated in the units your users feel. TTFT, ITL, decode-pause rate, and successful-stream rate cover the surface area you cannot afford to lose. Everything else is operational telemetry. Get the contract right, and the on-call rotation stops being a rotation through unrelated mysteries; it becomes a small set of well-rehearsed plays.

More notes

Found this useful?

Send me a note. I read every one.

Write to the author