Thursday, November 19, 2026 San Francisco Bay Area Notes by RSS
Asim Zafir

Reliability AI Serving Infrastructure Distributed Systems

Uptime on 10 PB+ HBase platforms
99.996%
Nodes under reliability ownership
50,000+
Years in SRE and infrastructure
19
Kubernetes versions operated
1.18 to 1.34

Nineteen years keeping large-scale production and AI infrastructure available, fast, and observable

A staff reliability engineer for AI serving infrastructure and large-scale distributed systems, now building infrastructure for agentic AI.

Most recently I owned production reliability for GPU training and inference serving paths, extending observability across the serving path and jumping into unfamiliar systems mid-incident to drive resolution. I reason holistically about how systems compose and where the seams are, and partner across teams as an embedded teammate rather than an outside voice.

My track record includes 99.996% uptime on 10 PB+ platforms, reliability ownership across 50,000+ node fleets, and SLO/SLI design, incident command, and blameless postmortem practice for distributed systems.

The record in brief
YearsRoleEmployer
2026 to presentFounding Infrastructure EngineerAI Stealth
2024 to 2026Senior Staff Infrastructure SRETikTok USDS, via NERDSTOP
2021 to 2024Site and Platform Reliability EngineerNERDSTOP and stealth startups
Before 2021Reliability, infrastructure and network engineeringPalo Alto Networks, Kaiser Permanente, Cisco Systems, eBay, Yahoo!
Continued in Experience, below

Working on inference reliability, GPU networking, or large-scale distributed systems?

I'd like to hear about it.

Write to the author

Notes

Observability for agentic systems

Tool calls, traces, provenance, eval-as-SLI — what changes when the workload is a model deciding what to do next.

February 8, 2026 · 6 min read
All notes

Experience

  1. 2026 to present

    Founding Infrastructure Engineer, Agentic AI Platform

    AI Stealth
    • Building enterprise-grade infrastructure for agentic AI: eval engines and a context-aware intelligence layer wrapping coding agents.
    • Designing RLHF / RLAIF feedback loops that couple agent output quality to underlying infrastructure and reliability signals.
  2. 2024 to 2026

    Senior Staff Infrastructure SRE, AI Serving and Distributed Systems

    TikTok USDS, via NERDSTOP
    • Owned production reliability for ML training and inference serving pipelines and the data-warehouse pipeline backing them.
    • Operated and optimized TCE compute infrastructure for training and inference datacenter workloads: Ray Serve and TensorRT-LLM serving paths on a large Kubernetes fleet.
    • Led incident response and workflow optimization to scale training and inference deployment across core Kubernetes fleets.
    • Extended serving-path observability with histogram-based latency and congestion telemetry over Clos fabrics and RoCEv2 / PFC / ECN.
  3. 2021 to 2024

    Site Reliability Engineer and Platform Reliability Engineer

    NERDSTOP and stealth startups
    • Built compute, networking, and platform infrastructure from zero to one for multiple startups, scaling them from early stage through later-stage growth.
    • Designed Prometheus / Grafana / Stackdriver observability across Kubernetes services and large data platforms.
    • Administered 10 PB+ HBase infrastructure supporting 99.996% uptime objectives through proactive monitoring, capacity planning, and structured RCA.
  4. Before 2021

    Reliability, infrastructure and network engineering

    Earlier engagements
    • Palo Alto Networks, Kaiser Permanente, Cisco Systems, eBay, Yahoo! and others. Full history available on request.

Areas of expertise

AI serving and GPU infrastructure

LLM inference and training serving paths (Ray Serve, TensorRT-LLM); GPU fleets with NVLink, NCCL and GPUDirect-TCPX / RDMA; inference autoscaling; training-pipeline reliability; observability across the serving path. This is my primary focus.

Reliability and SRE

SLO/SLI and error-budget design, incident command, root-cause analysis, blameless postmortems, and alert tuning at 50k+ node scale.

ML and data-centre networking

RoCEv2 / PFC / ECN lossless fabrics and RDMA, EVPN / VXLAN, Clos topologies, BGP/OSPF/MPLS, and multi-DC operations.

Observability

Prometheus, Grafana, OpenTelemetry and Stackdriver; histogram-based latency and congestion telemetry; queryable, auditable dashboard surfaces.

Cloud and Kubernetes

Multi-region, multi-cloud (GCP/GKE, AWS/EKS, Azure/AKS); Kubernetes 1.18 to 1.34 fleet operations; Terraform, Helm, Istio / Envoy mesh, CI/CD.

Distributed systems

Sync semantics, conflict resolution and idempotency; HBase, Kafka, Hadoop and Spark; capacity planning and production debugging.

Languages

Python (primary), Bash (fluent), SQL (comfortable), Java (production) and Go (working).

Selected work

GPU inference serving reliability and observability

Designed serving-path observability for GPU inference infrastructure, surfacing latency and congestion signals previously invisible to platform teams. Histogram-based telemetry across RoCEv2 fabrics, mapped through the inference path so on-call could attribute long-tail latency to specific fabric segments and queue depth events.

RoCEv2, PFC and ECN · Histogram telemetry · Long-tail latency

Woobi, an AI observability platform

Built a session-intelligence platform inspecting AI-agent activity through timeline events and tool-call traces, turning raw activity into a queryable, auditable observability surface for agent runs.

Timeline events · Tool-call traces · Audit surface

Distributed session-state synchronization

Designed cross-harness session synchronization with explicit conflict-resolution, idempotency, and provenance semantics; operationalized as resilient background services with built-in diagnostics.

Conflict resolution · Idempotency · Provenance

AI engineering contributions

Governed operating infrastructure for compound AI systems

Across my recent AI engineering work, the highest-leverage contribution is the operating layer around models and coding agents: deterministic human-approval gates, event-driven observability, calibrated statistical evaluation, hybrid retrieval, and explicit security boundaries—alongside clearly positioned simulated labs and formal verification models.

Governed multi-agent autonomy

End-to-end change control for agentic workflows: spec-gated intake, decision cards with explicit dissent, claim-time verdict checks, and cross-harness convergence—putting boundaries in state machines rather than advisory prompts.

Implemented · Human-approver autonomy

Observability and context continuity

Typed event-stream capture, deduplicated replay, tool-call timelines, cost metering, and cross-harness session continuity—treating agent traces and operating memory as distinct, auditable engineering surfaces.

Implemented · Event streams · Memory drift

Statistical evaluation and calibration

Paired bootstrap comparisons, confidence intervals, LLM-judge versus human agreement calibration, and budgeted batch evaluation so quality regressions carry uncertainty bounds instead of unverified scalar scores.

Implemented · Judge calibration · Uncertainty

Bounded harnesses and multi-path retrieval

Provider-independent agent loops with token/turn budgets, permission middleware, and read-only MCP tool contracts, alongside hybrid lexical, vector, and ontology retrieval fused with reciprocal-rank attribution.

Implemented · Typed tools · Hybrid RRF

Security boundaries and formal models

Post-hoc redacted trace scanning, disposable sandbox audits, and Ed25519 provenance—paired with fail-closed Z3 SMT remediation modeling and OPA/Rego admission blueprints with explicit scope guarantees.

Implemented & formal model · Explicit scope

Simulated labs and exploratory studies

Containerized GPU-fabric to NCCL stall correlation lab, GPU capacity-exchange prototypes, and a pilot layer-wise representation-analysis study—positioned transparently as simulations and early experiments.

Simulated lab & pilot · Controlled claims

Explore the full AI engineering landing page, ranked insights, and simulated designs

Read enough? Write to the author.

Research collaboration, hiring conversations, infrastructure consulting, or a technical back-and-forth. I read every message that comes through the form.

Write to the author