Tuesday, November 17, 2026 San Francisco Bay Area Notes by RSS
Asim Zafir

Reliability AI Serving Infrastructure Distributed Systems

Where the model meets the metal

Five overlapping lines of inquiry, organized around the slice of reliability that becomes interesting when the workload is a model and the substrate is a GPU fabric.

By Asim Zafir · Twenty-minute read · San Francisco

A classical web stack has two latency stories, and you usually only need to tell one of them with any care. There is the request that returned a response, and there is the request that did not. P95 covers most of the first and 5xx covers most of the second. A team can run a CRUD service well for years without ever having to make their observability denser than that.

An inference path is a different surface, and the difference shows up almost immediately. Tokens are streamed; latency is a function not of one request but of many concurrent decode steps colliding on shared SM partitions; failures rarely look like 5xx. By the time the response code is wrong, the system has been broken for a long time, often in ways that selectively harm the most expensive users. The dashboards you brought with you from the CRUD world will keep showing green throughout. You will discover the outage from a support ticket.

What follows is a survey of the five places I am currently spending my attention, ordered roughly by how often they have produced incidents I cared about in the last two years.

I. Tail latency on the token-streaming path

The single most useful framing I have on inference latency is that a token-streaming endpoint has three distinct latencies, and each needs its own service level indicator.

Time-to-first-token is the queue-and-prefill cost. It is dominated by admission control, KV-cache eviction, and the geometry of how prefill GEMMs pack onto the available compute. When this number degrades, the cause is almost always upstream of the GPU itself — a load balancer making a poor pinning decision, an admission policy starving a tenant, a cache fragmenting after a deploy.

Inter-token latency is the decode steady-state. A clean ITL looks like a near-constant with no sawteeth. Any irregular spike is usually one of three things: a tensor-parallel rank straggler, a CUDA-graph cache miss after a configuration change, or a noisy neighbor inside the same SM partition. The shape of the spike tells you which.

End-to-end completion time is the user-felt latency. It is a derived quantity — the sum of the first two, plus stream-close overhead — and it should not be alerted on directly. Alerting on it muddles two very different failure surfaces and produces the kind of pages whose runbook is “look at the graphs; the cause is somewhere.”

The mistake I see most often is collapsing all three into one histogram and watching the p99 of the sum. You will detect outages. You will not learn anything from them.

The number I will defend most strongly, and the one that pulled the longest tail out of a multi-thousand-node fleet I worked on, is the decode-pause rate — pauses per thousand tokens, where a “pause” is any inter-token interval above 150 ms. It catches a class of failure that p99 ITL cannot see, because a five-second NCCL hang shows up as a handful of catastrophic intervals against a denominator of millions of clean ones, and the histogram quietly averages it into invisibility.

A pause rate above the SLO target is almost always one of: a misbehaving NIC pausing its upstream and ringing PFC up the leaf-spine; a single rank failing collective health for ECC reasons and getting evicted halfway through a decode batch; an admission-control regression that lets a long prompt block the decode queue. 1 Each has a distinctive shape in the pause-time distribution, and a well-labeled per-rank histogram resolves which it is in well under a minute.

II. Lossless Ethernet, and the lie inside that word

Marketing says lossless. The packet captures say something more careful. A RoCEv2 fabric offers, in practice, an extremely specific contract: under correctly-tuned priority-flow-control, the transmit side will not drop frames due to receive-side buffer exhaustion. Everything outside that sentence — congestion-induced unfairness, head-of-line blocking, PFC pause propagation, ECN-induced rate cuts that fight your application's own backpressure — is a problem you still own.

The summary I tell every team about to ship a RoCE deployment: lossless Ethernet is a re-distribution of where the pathology lives, not an elimination of it. The packet drops you used to get on a congested TCP fabric are replaced by congestion-spreading PFC pauses, which produce a far worse tail-latency distribution if you do not tune them. The good news is that the tuning is well-understood. The bad news is that the defaults are wrong on every vendor's switch and on every NIC's firmware.

The failures, ordered by frequency

In two years of operating a multi-pod fabric at scale, the four problems that bit hardest were, in this order:

One, a PFC storm started by a single misbehaving host. A NIC stops draining its receive ring — driver bug, NUMA contention, IRQ moderation badly tuned against kernel bypass — and begins asserting pause frames upstream. The pause propagates through the leaf, then the spine, then the rest of the leaf-spine within a few seconds. Mean-time-to-recovery is short once you know what you are looking at; mean-time-to-detect is hours if you do not have per-priority Rx-pause counters on the wall.

Two, ECN marking too aggressive. DCQCN reacts to ECN by cutting the rate of the sending flow. Set the marking thresholds too low and the steady state is “everything is at thirty percent of line rate.” Set them too high and ECN does not fire until PFC has already tripped, and the storm happens anyway.

Three, slow-start collisions on incast. A freshly-rebooted rank rejoins an all-reduce in a state the rest of the ring is not expecting. PFC counters spike for a few hundred milliseconds during the ramp; the all-reduce takes three times its steady-state time. The fabric counters look fine. The application metric is a lie.

Four, cable degradation that does not page anyone. A 400G link with a corrected-symbol rate creeping up but still below the FEC error threshold. No alarm. But the tail latency on the affected path moves by tens of microseconds, which is enough to push a tensor-parallel all-gather into the next bucket of the latency histogram.

III. SLOs as contracts, not metrics

An SLO that lives in a slide deck is documentation. An SLO that lives in production has, at minimum: an SLI defined in user-meaningful units, a target value, a measurement window, a burn-rate alert with a documented receiver, an error-budget policy that says what gets stopped when the budget is exhausted, and a human who has signed for it.

Most of the teams I have walked into have the first three. The interesting question is what happens in the absence of the last three.

What happens, in approximately this order: the SLO is referenced in planning but not respected. The burn-rate alerts are noisy, so they get muted. The freeze policy is unwritten, so when it is needed it is invented under stress at three a.m., badly. The signing human is unclear, so the post-incident review has no organizing principle and produces three action items that nobody owns.

An SLO that does not have an alert pathway is not an SLO. It is a number on a slide. The pathway is the contract.

The reliability layer for an inference service has its own complication: error budgets here are not smooth burns. A twenty-minute degradation on a single replica during peak can dump four percent of monthly traffic into the failure bucket. The split that has worked for me is a small slow-burn envelope that is tightly defended, and a separate spike envelope that tolerates incidents up to a maximum severity at a maximum frequency. This lets ordinary deploys ship without every nvidia-fabricmanager blip paging someone.

IV. Observability for agentic systems

The hard part of operating an agentic system is not the GPUs. It is that the workload — the thing you are trying to keep healthy — is making decisions about what to do next, and those decisions are conditioned on context you did not author. The classical observability triad is necessary here, and not sufficient. You need to capture what the agent did and why it thought that was the right thing, with enough fidelity to reconstruct the decision and enough structure to alert on regressions in decision quality rather than decision latency.

The shape that works is three timelines and a provenance trail.

The model timeline records every prompt, every response, every tool call as a structured event. The tool timeline is a span tree, inputs and outputs by size and schema, retries by class. The context timeline records every piece of context the model could have conditioned on at the moment it decided — retrieved documents, session memory, system-prompt revisions, attached files. The three on the same axis are the equivalent of a request trace in classical microservices. They are the difference between a four-hour incident and a forty-minute one.

The provenance trail is the tag that links each piece of model output to the context fragment that drove it. Approximate for closed models, useful anyway. When a user complains that the agent quoted a stale document, you should be able to point at the indexer that emitted the stale chunk in under a minute. Without provenance, you cannot.

Eval as an SLI

The piece most teams skip is treating decision quality as a streaming signal, not an offline batch. A curated shadow set, graded on every deploy and on a fixed cadence, produces a grade distribution that is itself an SLI. A burn-rate alert on the shadow-set quality SLO fires the moment a regression ships, hours or days before the support tickets land. The discipline I have grown attached to: no incident closes without contributing at least one session to the shadow set. Six months of that produces a regression detector with surprising precision.

V. The economics of running models in production

Capacity planning for a token-streaming service is dominated by tail behavior, not steady state. Two services with identical median latency and very different p99.9 latencies have different cost structures, different SLO postures, and ultimately different products. A planning model that averages this away will produce a number that is wrong in interesting ways.

The unit economics that interest me most are the ones where the model's own behavior produces tail-shaping decisions — speculative decoding, prompt caching at the edge, batch coalescing, prefill–decode disaggregation. Each of these is a knob that trades steady-state efficiency for tail latency or vice versa. The right operating point depends on what the SLO denominates, which is downstream of what the product promises, which is downstream of what the user is paying for.

The interesting part is that that whole stack is now a single problem. Reliability engineers used to be able to leave the product side of these decisions to other teams. On an inference path, the tail-latency budget is the product. The two have collapsed into one conversation, and the engineer who can hold both ends is, in my belief, the engineer who will make the largest difference at an AI lab in the next five years.

Closing

These five lines of inquiry are deliberately overlapping. The PFC tuning in section II is what keeps the decode-pause rate in section I respectable. The decode-pause rate is what the SLO contract in section III hangs on. The SLO contract is the thing the observability surface in section IV exists to monitor. The economics in section V is the answer to “why is anyone paying for any of this.”

I am writing about each of them in the notes — currently three pieces, with more landing as I have time to set them properly. If any of them are useful to you, I would like to hear which ones; the contact form goes straight to my inbox.

  1. A small fraternity of teams have learned this the expensive way. The decode-pause rate is what they all converge on once they have rebuilt their dashboards two or three times. ↩

Eight open questions

Open questions I want to spend the next decade with: the surface the industry is moving toward, and the one I have been operating against for the last few years. The list is opinionated and concrete.

  1. Inference path observability at fleet scale

    Today the inference path is observable in pieces: host metrics here, GPU utilisation there, queueing in another tool, network somewhere else. I want to build the surface that composes them into one queryable story per request, so on-call can attribute long-tail latency to a specific cause class in seconds, not hours. Histogram-based telemetry, tail-aware SLOs, cross-layer correlation IDs. This is my primary interest.

    Per-request causal traces · Tail SLOs (p99.9 / p99.99) · Cross-layer correlation · Queue-aware autoscaling

    // slo: p99.9 token-latency, 30d rolling
    // SLI = histogram_quantile(0.999,
    //   sum(rate(serve_inference_token_latency_bucket[5m])) by (le))
    // budget burn-rate alerts: 2h fast, 6h slow
  2. RoCEv2 / PFC / ECN at scale

    Lossless fabric is brittle in interesting ways: PFC pause storms, ECN-marked but-not-yet-throttled microbursts, head-of-line blocking on shared buffers. How do we operate these networks like we operate Kubernetes, with SLIs, budgets, blameless reviews, and a postmortem culture?

    PFC storm telemetry · ECN tuning · HoL blocking diagnosis

  3. NCCL / GPUDirect topology debugging

    When a training run slows down by 8%, was it bad NVLink topology assignment, a bad PCIe slot, NCCL ring versus tree, or a flaky RDMA NIC two racks over? The answer is recoverable but expensive. The tools could be much better.

    NVLink layout · NCCL topology files · GPUDirect-TCPX / RDMA

  4. Reliability of training pipelines

    Training runs are long, expensive, and stateful. They have entirely different reliability semantics from request-serving systems. Checkpointing, elastic recovery, hardware-failure replay, deterministic re-runs: this is a different discipline that deserves its own SRE practice.

    Checkpoint cadence · Elastic recovery · Deterministic re-runs

  5. RLHF / RLAIF feedback loops as production telemetry

    Model output quality is a production signal, equal in dignity to p99 latency or error rate. I want feedback loops where reliability degradations show up in eval scores, and eval regressions trigger the same incident review machinery as a serving outage.

    Eval-as-SLI · Quality regressions · Postmortem coupling

  6. Multi-region, multi-cloud serving

    Production AI infrastructure has to span clouds and regions, and the failure modes get exotic: DNS skew, asymmetric BGP withdrawals, certificate fan-out clocks, region-level brownouts. I'm interested in the operational surface that makes these failures legible.

    GKE, EKS and AKS · BGP / EVPN-VXLAN · Region brownouts

  7. Incident command for AI systems

    Classic incident command assumes a system you can debug by inspection. AI systems often need evaluations as a debugging tool. What does an incident command discipline look like when the runbook ends with "run the eval suite and compare to last week"?

    IMOC patterns · Eval as RCA tool · Blameless reviews

  8. Operability of agentic systems

    Agents are long-lived, tool-using, multi-step processes. The operability surface (what they are doing right now, what they have tried, what they are about to do) needs first-class observability. I built one of these, Woobi; I want to build a better one.

    Timeline events · Tool-call traces · Provenance

Want to collaborate on one of these?

I'm always up for a back-and-forth on any of these surfaces. No selling.

Write to the author