Operating infrastructure for compound AI systems
A synthesis of my AI engineering work for researchers, scientists, and AI systems specialists: where the core contributions sit, what architectural principles held up under scrutiny, and how implemented systems are separated from simulated designs and exploratory studies.
By Asim Zafir · Compound AI systems, governed autonomy, evaluation, and serving reliability · San Francisco
The central contribution
From isolated model calls to governed, observable compound systems
In production AI, capability rarely fails at the single prompt; it fails at the seams between models, tools, retrieval, persistent state, evaluation, and human accountability. My strongest and most distinctive contribution is building the operating, governance, and observability substrate around multi-agent and compound AI systems, bringing nineteen years of distributed-systems SRE discipline into agent execution, statistical evaluation, and GPU serving paths.
Implemented systems
Working control planes, approval state machines, event-driven trace pipelines, cross-harness session sync, statistical evaluation libraries, typed tool servers, and hybrid retrieval engines built and exercised across multi-vendor agent workflows.
Control planes · Telemetry · Evaluation · Retrieval
Simulated designs and formal models
Containerized GPU-fabric correlation labs, SMT-verified remediation models, policy-as-code sandboxes, and compute-exchange prototypes positioned explicitly as architectural testbeds rather than production deployments.
Lab simulations · Formal constraints · Reference blueprints
Explicit epistemic boundaries
Exploratory interpretability artifacts, integration prototypes, and heuristic scanners are labeled as such. Source existence is never presented as a validated research breakthrough, clinical model, or benchmark superiority claim.
Scope discipline · Controlled claims · Reproducibility first
Top AI engineering contributions, ranked
Ordered by architectural depth and differentiation across demanding AI systems domains, framed at the systems-design level without discrete proprietary implementation details.
-
1. Governed autonomy and human-in-the-loop change control
Designed and operated an end-to-end governance layer for multi-vendor coding and operational agents: structured request-to-specification compilation with pre-execution validation gates, per-decision review surfaces that require explicit trade-offs and dissent before approval, handover packets that remain blocked until a recorded human verdict exists, claim-time verdict re-verification, and managed configuration convergence across heterogeneous agent runtimes.
Key insight: Prompt-level rules are advisory and drift under long contexts. Enforceable human-approver autonomy belongs in deterministic state machines, append-only verdict logs, and claim-time contract checks outside the model loop.
Implemented · Human-approver autonomy · Approval state machines · Spec-gated execution · Cross-harness convergence
-
2. Agent observability, trace pipelines, and cross-harness context continuity
Built an event-driven observation layer over multi-agent workflows: streaming session capture and deduplicated replay over durable messaging, typed tool-call and timeline materialization, token and cost metering, runtime metadata registries, and a feedback loop that turns human corrections into durable operating principles. Separated operational observability from context continuity through cross-harness session translation, replay, and versioned memory-drift tracking.
Key insight: Raw transcript logging is neither observability nor learned memory. Treating agent runs as typed, replayable event streams makes multi-vendor failure attribution, cost accounting, and memory-drift analysis tractable.
Implemented · Event-driven telemetry · Tool-call traces · Cross-harness replay · Memory drift
-
3. Statistical evaluation, judge calibration, and experiment discipline
Developed statistical evaluation tooling covering paired bootstrap comparisons, proportion confidence intervals, significance tests, LLM-as-judge versus human agreement calibration, and distribution-drift measurement, paired with budgeted batch evaluation runners and cross-tool trace-interoperability fixes so evaluator pipelines do not silently drop agent observations.
Key insight: Uncalibrated scalar judge scores invite false confidence. Treating evaluation like an empirical measurement discipline—with confidence bounds, human-agreement checks, and explicit batch budgets—turns evals into actionable engineering signals.
Implemented · Paired bootstrap · Judge-human calibration · Drift statistics · Trace interoperability
-
4. Bounded agent harnesses, typed tool contracts, and multi-path retrieval
Engineered provider-independent agent execution loops enforcing explicit turn and token budgets, layered permission middleware, typed tool registries, and read-only Model Context Protocol (MCP) tool servers fronting operational registries. Built standalone hybrid and offline knowledge-retrieval systems combining lexical ranking (BM25F), latent semantic or dense vector search, and ontology traversal fused via reciprocal-rank fusion with per-path attribution.
Key insight: Decoupling retrieval from generation—and enforcing least-privilege, typed contracts at the agent-computer interface—allows retrieval quality, tool safety, and model routing to be audited and improved independently.
Implemented · Provider-neutral loops · MCP servers · BM25F + vectors + ontology · Reciprocal-rank fusion
-
5. AI security boundaries, trace auditing, and controlled execution
Implemented post-hoc sensitive-data and PII exposure scanning across multi-harness agent transcripts—persisting only redacted masks and cryptographic hashes, never plaintext—alongside disposable sandbox workflows for auditing untrusted dependencies, Ed25519-signed runtime alignment and licensing artifacts, and event-backed host and network telemetry graphs.
Key insight: Agent security claims are credible only when their guarantee boundary is explicit: post-hoc trace scanners detect leakage without falsely claiming inline interception, disposable sandboxes contain supply-chain inspection, and cryptographic signatures prove artifact provenance.
Implemented · Redacted trace auditing · Sandboxed audits · Ed25519 provenance · eBPF telemetry graphs
-
6. AI serving reliability, GPU fabric operations, and applied compound systems
Grounded the agentic control-plane work in production SRE ownership for ML training and inference serving paths (Ray Serve, TensorRT-LLM, large Kubernetes fleets, and RoCEv2 / PFC / ECN lossless fabrics), complemented by applied compound-AI systems spanning streaming AI tutoring with task-tier model routing and isolated code sandboxes, domain advisory tool loops over environmental forecasting pipelines, and orchestrated ML feature and drift workflows.
Key insight: Model serving tail latency, collective communication stalls, and agent execution reliability form a single coupled surface; observability and SLO design must span from NIC congestion queues up to evaluator regressions.
Production & applied · Inference serving · Lossless RDMA fabrics · Model-tier routing · MLOps drift checks
Simulated designs, formal models, and reference architectures
Several demanding AI infrastructure and safety problems were explored through lab simulations, formal constraint verifiers, architectural directives, and pilot experiments. Each is positioned here with its hypothesis and exact scope boundary.
GPU-fabric and NCCL stall correlation lab
Containerized lossless-fabric simulation coupled to a streaming Kafka and Prometheus correlator matching same-rail network telemetry against collective-communication stall events within bounded windows. Positioned as a lab architecture for cross-layer triage: temporal correlation narrows fault domains without claiming causal proof or frontier-cluster production validation.
Simulated lab · Containerized Clos/RoCE · Streaming correlation
SMT-verified remediation and optimization spike
Automated operational remediation architecture combining human approval gates with a Z3 SMT policy verifier that fails closed whenever the solver is unavailable, paired with an optimization spike over programmatic prompt and graph workflows. Positioned as formal-model engineering: proofs hold strictly over encoded model assumptions (such as regenerable targets), not arbitrary shell safety or autonomous self-improvement.
Formal model & spike · Z3 constraint checking · Fail-closed guard
Policy-as-code and threat-detection blueprints
Declarative OPA/Rego policy-as-code experiments and Kubernetes admission-control sidecar patterns across clouds, alongside a pre-implementation architecture directive linking endpoint and network threat detection to AI provenance hooks. Positioned as policy sandboxes and architectural specifications rather than a deployed enterprise EDR/XDR runtime.
Sandbox & architecture · OPA/Rego · Admission control · Provenance hooks
Compute capacity exchange and cloud migration designs
Two-sided GPU capacity marketplace prototype modeling a compute ontology, capacity warrants, ranked offers, and dynamic pricing logic, alongside structured multi-cloud Kubernetes migration blueprints. Positioned as domain and infrastructure architecture prototypes rather than live trading liquidity or completed fleet migrations.
Prototype & blueprint · Compute ontology · Multi-cloud topology
Layer-wise representation analysis pilot
Exploratory interpretability artifacts on an open-weight transformer model: experiment manifests, layer-scan scores, random and zero control conditions, activation arrays, and blinded response-review materials. Positioned strictly as an early pilot with a small held-out set and human semantic validation pending—demonstrating experimental control discipline without claiming a reproduced safety breakthrough.
Exploratory pilot · Layer probes · Random/zero controls · Blinded review
Multimodal workflow and predictive-SRE prototypes
End-to-end integration prototypes combining medical-imaging preprocessing, service APIs, and agent interfaces (using mocked inference outputs), alongside a multi-prism heuristic SRE signal scanner and convergence prototype. Positioned transparently as systems-integration and heuristic architectures, not validated diagnostic models or calibrated outage predictors.
Integration prototype · Heuristic signals · Mocked inference boundary
Capability map by AI discipline
A compact index aligning the work to compound-AI-systems and agent-engineering literature so technical evaluators can locate depth and maturity at a glance.
Disciplines, architectural focus, and evidence posture
| Discipline |
Primary focus |
Posture |
| Governance |
Approval gates, decision cards with dissent, spec-gated handovers, harness sync |
Built (core) |
| Telemetry |
Event capture, replay, tool traces, cost metering, correction-to-principle loops |
Built |
| Evaluation |
Paired bootstrap intervals, judge-human calibration, drift stats, budgeted evals |
Built |
| Tools & RAG |
Bounded loops, permission layers, read-only MCP, lexical + vector + ontology RRF |
Built |
| AI security |
Redacted trace scans, sandbox audits, Ed25519 signing, Z3 & OPA / Rego models |
Built & formal |
| GPU & serving |
Inference and training SRE (Ray Serve, TensorRT-LLM, RoCEv2), fabric stall lab |
Prod & lab |
| Model probes |
Layer-wise activation scans, random and zero controls, blinded review |
Early pilot |
Read the long-form research essay on inference, lossless fabrics, and agent operability
Want to discuss one of these architectures in depth?
Happy to walk researchers, scientists, and engineering leaders through architecture trade-offs, evaluation methodology, or governed agent operations.
Write to the author