ReliabilityAI Serving InfrastructureDistributed Systems
Uptime on 10 PB+ HBase platforms
99.996%
Nodes under reliability ownership
50,000+
Years in SRE and infrastructure
19
Kubernetes versions operated
1.18 to 1.34
Nineteen years keeping large-scale production and AI infrastructure available, fast, and observable
A staff reliability engineer for AI serving infrastructure and large-scale distributed systems, now building infrastructure for agentic AI.
By Asim Zafir · San Francisco Bay Area · Open to relocation
Most recently I owned production reliability for GPU training and inference serving paths, extending observability across the serving path and jumping into unfamiliar systems mid-incident to drive resolution. I reason holistically about how systems compose and where the seams are, and partner across teams as an embedded teammate rather than an outside voice.
My track record includes 99.996% uptime on 10 PB+ platforms, reliability ownership across 50,000+ node fleets, and SLO/SLI design, incident command, and blameless postmortem practice for distributed systems.
The record in brief
Years
Role
Employer
2026 to present
Founding Infrastructure Engineer
AI Stealth
2024 to 2026
Senior Staff Infrastructure SRE
TikTok USDS, via NERDSTOP
2021 to 2024
Site and Platform Reliability Engineer
NERDSTOP and stealth startups
Before 2021
Reliability, infrastructure and network engineering
Palo Alto Networks, Kaiser Permanente, Cisco Systems, eBay, Yahoo!
Founding Infrastructure Engineer, Agentic AI Platform
AI Stealth
Building enterprise-grade infrastructure for agentic AI: eval engines and a context-aware intelligence layer wrapping coding agents.
Designing RLHF / RLAIF feedback loops that couple agent output quality to underlying infrastructure and reliability signals.
2024 to 2026
Senior Staff Infrastructure SRE, AI Serving and Distributed Systems
TikTok USDS, via NERDSTOP
Owned production reliability for ML training and inference serving pipelines and the data-warehouse pipeline backing them.
Operated and optimized TCE compute infrastructure for training and inference datacenter workloads: Ray Serve and TensorRT-LLM serving paths on a large Kubernetes fleet.
Led incident response and workflow optimization to scale training and inference deployment across core Kubernetes fleets.
Extended serving-path observability with histogram-based latency and congestion telemetry over Clos fabrics and RoCEv2 / PFC / ECN.
2021 to 2024
Site Reliability Engineer and Platform Reliability Engineer
NERDSTOP and stealth startups
Built compute, networking, and platform infrastructure from zero to one for multiple startups, scaling them from early stage through later-stage growth.
Designed Prometheus / Grafana / Stackdriver observability across Kubernetes services and large data platforms.
Administered 10 PB+ HBase infrastructure supporting 99.996% uptime objectives through proactive monitoring, capacity planning, and structured RCA.
Before 2021
Reliability, infrastructure and network engineering
Earlier engagements
Palo Alto Networks, Kaiser Permanente, Cisco Systems, eBay, Yahoo! and others. Full history available on request.
Areas of expertise
AI serving and GPU infrastructure
LLM inference and training serving paths (Ray Serve, TensorRT-LLM); GPU fleets with NVLink, NCCL and GPUDirect-TCPX / RDMA; inference autoscaling; training-pipeline reliability; observability across the serving path. This is my primary focus.
Reliability and SRE
SLO/SLI and error-budget design, incident command, root-cause analysis, blameless postmortems, and alert tuning at 50k+ node scale.
ML and data-centre networking
RoCEv2 / PFC / ECN lossless fabrics and RDMA, EVPN / VXLAN, Clos topologies, BGP/OSPF/MPLS, and multi-DC operations.
Observability
Prometheus, Grafana, OpenTelemetry and Stackdriver; histogram-based latency and congestion telemetry; queryable, auditable dashboard surfaces.
Sync semantics, conflict resolution and idempotency; HBase, Kafka, Hadoop and Spark; capacity planning and production debugging.
Languages
Python (primary), Bash (fluent), SQL (comfortable), Java (production) and Go (working).
Selected work
GPU inference serving reliability and observability
Designed serving-path observability for GPU inference infrastructure, surfacing latency and congestion signals previously invisible to platform teams. Histogram-based telemetry across RoCEv2 fabrics, mapped through the inference path so on-call could attribute long-tail latency to specific fabric segments and queue depth events.
RoCEv2, PFC and ECN · Histogram telemetry · Long-tail latency
Woobi, an AI observability platform
Built a session-intelligence platform inspecting AI-agent activity through timeline events and tool-call traces, turning raw activity into a queryable, auditable observability surface for agent runs.
Designed cross-harness session synchronization with explicit conflict-resolution, idempotency, and provenance semantics; operationalized as resilient background services with built-in diagnostics.
Conflict resolution · Idempotency · Provenance
Read enough? Write to the author.
Research collaboration, hiring conversations, infrastructure consulting, or a technical back-and-forth. I read every message that comes through the form.