A classical web stack has two latency stories, and you usually only need to tell one of them with any care. There is the request that returned a response, and there is the request that did not. P95 covers most of the first and 5xx covers most of the second. A team can run a CRUD service well for years without ever having to make their observability denser than that.
An inference path is a different surface, and the difference shows up almost immediately. Tokens are streamed; latency is a function not of one request but of many concurrent decode steps colliding on shared SM partitions; failures rarely look like 5xx. By the time the response code is wrong, the system has been broken for a long time, often in ways that selectively harm the most expensive users. The dashboards you brought with you from the CRUD world will keep showing green throughout. You will discover the outage from a support ticket.
What follows is a survey of the five places I am currently spending my attention, ordered roughly by how often they have produced incidents I cared about in the last two years.
I. Tail latency on the token-streaming path
The single most useful framing I have on inference latency is that a token-streaming endpoint has three distinct latencies, and each needs its own service level indicator.
Time-to-first-token is the queue-and-prefill cost. It is dominated by admission control, KV-cache eviction, and the geometry of how prefill GEMMs pack onto the available compute. When this number degrades, the cause is almost always upstream of the GPU itself — a load balancer making a poor pinning decision, an admission policy starving a tenant, a cache fragmenting after a deploy.
Inter-token latency is the decode steady-state. A clean ITL looks like a near-constant with no sawteeth. Any irregular spike is usually one of three things: a tensor-parallel rank straggler, a CUDA-graph cache miss after a configuration change, or a noisy neighbor inside the same SM partition. The shape of the spike tells you which.
End-to-end completion time is the user-felt latency. It is a derived quantity — the sum of the first two, plus stream-close overhead — and it should not be alerted on directly. Alerting on it muddles two very different failure surfaces and produces the kind of pages whose runbook is “look at the graphs; the cause is somewhere.”
The mistake I see most often is collapsing all three into one histogram and watching the p99 of the sum. You will detect outages. You will not learn anything from them.
The number I will defend most strongly, and the one that pulled the longest tail out of a multi-thousand-node fleet I worked on, is the decode-pause rate — pauses per thousand tokens, where a “pause” is any inter-token interval above 150 ms. It catches a class of failure that p99 ITL cannot see, because a five-second NCCL hang shows up as a handful of catastrophic intervals against a denominator of millions of clean ones, and the histogram quietly averages it into invisibility.
A pause rate above the SLO target is almost always one of: a misbehaving NIC pausing its upstream and ringing PFC up the leaf-spine; a single rank failing collective health for ECC reasons and getting evicted halfway through a decode batch; an admission-control regression that lets a long prompt block the decode queue. 1 Each has a distinctive shape in the pause-time distribution, and a well-labeled per-rank histogram resolves which it is in well under a minute.
II. Lossless Ethernet, and the lie inside that word
Marketing says lossless. The packet captures say something more careful. A RoCEv2 fabric offers, in practice, an extremely specific contract: under correctly-tuned priority-flow-control, the transmit side will not drop frames due to receive-side buffer exhaustion. Everything outside that sentence — congestion-induced unfairness, head-of-line blocking, PFC pause propagation, ECN-induced rate cuts that fight your application's own backpressure — is a problem you still own.
The summary I tell every team about to ship a RoCE deployment: lossless Ethernet is a re-distribution of where the pathology lives, not an elimination of it. The packet drops you used to get on a congested TCP fabric are replaced by congestion-spreading PFC pauses, which produce a far worse tail-latency distribution if you do not tune them. The good news is that the tuning is well-understood. The bad news is that the defaults are wrong on every vendor's switch and on every NIC's firmware.
The failures, ordered by frequency
In two years of operating a multi-pod fabric at scale, the four problems that bit hardest were, in this order:
One, a PFC storm started by a single misbehaving host. A NIC stops draining its receive ring — driver bug, NUMA contention, IRQ moderation badly tuned against kernel bypass — and begins asserting pause frames upstream. The pause propagates through the leaf, then the spine, then the rest of the leaf-spine within a few seconds. Mean-time-to-recovery is short once you know what you are looking at; mean-time-to-detect is hours if you do not have per-priority Rx-pause counters on the wall.
Two, ECN marking too aggressive. DCQCN reacts to ECN by cutting the rate of the sending flow. Set the marking thresholds too low and the steady state is “everything is at thirty percent of line rate.” Set them too high and ECN does not fire until PFC has already tripped, and the storm happens anyway.
Three, slow-start collisions on incast. A freshly-rebooted rank rejoins an all-reduce in a state the rest of the ring is not expecting. PFC counters spike for a few hundred milliseconds during the ramp; the all-reduce takes three times its steady-state time. The fabric counters look fine. The application metric is a lie.
Four, cable degradation that does not page anyone. A 400G link with a corrected-symbol rate creeping up but still below the FEC error threshold. No alarm. But the tail latency on the affected path moves by tens of microseconds, which is enough to push a tensor-parallel all-gather into the next bucket of the latency histogram.
III. SLOs as contracts, not metrics
An SLO that lives in a slide deck is documentation. An SLO that lives in production has, at minimum: an SLI defined in user-meaningful units, a target value, a measurement window, a burn-rate alert with a documented receiver, an error-budget policy that says what gets stopped when the budget is exhausted, and a human who has signed for it.
Most of the teams I have walked into have the first three. The interesting question is what happens in the absence of the last three.
What happens, in approximately this order: the SLO is referenced in planning but not respected. The burn-rate alerts are noisy, so they get muted. The freeze policy is unwritten, so when it is needed it is invented under stress at three a.m., badly. The signing human is unclear, so the post-incident review has no organizing principle and produces three action items that nobody owns.
An SLO that does not have an alert pathway is not an SLO. It is a number on a slide. The pathway is the contract.
The reliability layer for an inference service has its own complication: error budgets here are not smooth burns. A twenty-minute degradation on a single replica during peak can dump four percent of monthly traffic into the failure bucket. The split that has worked for me is a small slow-burn envelope that is tightly defended, and a separate spike envelope that tolerates incidents up to a maximum severity at a maximum frequency. This lets ordinary deploys ship without every nvidia-fabricmanager blip paging someone.
IV. Observability for agentic systems
The hard part of operating an agentic system is not the GPUs. It is that the workload — the thing you are trying to keep healthy — is making decisions about what to do next, and those decisions are conditioned on context you did not author. The classical observability triad is necessary here, and not sufficient. You need to capture what the agent did and why it thought that was the right thing, with enough fidelity to reconstruct the decision and enough structure to alert on regressions in decision quality rather than decision latency.
The shape that works is three timelines and a provenance trail.
The model timeline records every prompt, every response, every tool call as a structured event. The tool timeline is a span tree, inputs and outputs by size and schema, retries by class. The context timeline records every piece of context the model could have conditioned on at the moment it decided — retrieved documents, session memory, system-prompt revisions, attached files. The three on the same axis are the equivalent of a request trace in classical microservices. They are the difference between a four-hour incident and a forty-minute one.
The provenance trail is the tag that links each piece of model output to the context fragment that drove it. Approximate for closed models, useful anyway. When a user complains that the agent quoted a stale document, you should be able to point at the indexer that emitted the stale chunk in under a minute. Without provenance, you cannot.
Eval as an SLI
The piece most teams skip is treating decision quality as a streaming signal, not an offline batch. A curated shadow set, graded on every deploy and on a fixed cadence, produces a grade distribution that is itself an SLI. A burn-rate alert on the shadow-set quality SLO fires the moment a regression ships, hours or days before the support tickets land. The discipline I have grown attached to: no incident closes without contributing at least one session to the shadow set. Six months of that produces a regression detector with surprising precision.
V. The economics of running models in production
Capacity planning for a token-streaming service is dominated by tail behavior, not steady state. Two services with identical median latency and very different p99.9 latencies have different cost structures, different SLO postures, and ultimately different products. A planning model that averages this away will produce a number that is wrong in interesting ways.
The unit economics that interest me most are the ones where the model's own behavior produces tail-shaping decisions — speculative decoding, prompt caching at the edge, batch coalescing, prefill–decode disaggregation. Each of these is a knob that trades steady-state efficiency for tail latency or vice versa. The right operating point depends on what the SLO denominates, which is downstream of what the product promises, which is downstream of what the user is paying for.
The interesting part is that that whole stack is now a single problem. Reliability engineers used to be able to leave the product side of these decisions to other teams. On an inference path, the tail-latency budget is the product. The two have collapsed into one conversation, and the engineer who can hold both ends is, in my belief, the engineer who will make the largest difference at an AI lab in the next five years.
Closing
These five lines of inquiry are deliberately overlapping. The PFC tuning in section II is what keeps the decode-pause rate in section I respectable. The decode-pause rate is what the SLO contract in section III hangs on. The SLO contract is the thing the observability surface in section IV exists to monitor. The economics in section V is the answer to “why is anyone paying for any of this.”
I am writing about each of them in the notes — currently three pieces, with more landing as I have time to set them properly. If any of them are useful to you, I would like to hear which ones; the contact form goes straight to my inbox.
- A small fraternity of teams have learned this the expensive way. The decode-pause rate is what they all converge on once they have rebuilt their dashboards two or three times. ↩