PROBLEM On-call SREs drown in alerts. A single Kubernetes incident throws signals across pods, app runtimes, dependencies (Kafka/Redis/DB), traces, recent deploys and history and a human has to correlate all of it under pressure, while most pages are noise. Single-LLM "AIOps" tools hallucinate confident root causes from one slice of data and erode trust. SENTINEL SENTINEL is an autonomous, multi-agent SRE brain that reasons about production incidents the way a senior on-call engineer does but in parallel and with discipline. A fleet of SEVEN specialist agents each independently investigates ONLY its own data slice and votes on the failure signatures it can justify: • Kubernetes (pods, OOMKills, CrashLoops, scheduling, probes) • App Runtime (logs, JVM/Go/Node runtime errors) • Dependency (Kafka lag, Redis/DB saturation, downstream APIs) • Observability (golden signals, SLO burn, traces) • Change Correlation ("what changed right before this started?") • Runbook (matches vetted procedures) • Historical Memory (recurrence of past incidents) THE MULTI-SIGNAL RULE (the core reasoning idea) An Evidence Correlation Engine confirms a signature ONLY when ≥2 INDEPENDENT telemetry agents agree on it. One agent shouting is noise; two independent agents agreeing is signal worth paging a human for. This single rule is what makes the system trustworthy and kills false positives every confirmed cause carries its corroboration count and the agents that voted. It then ranks root-cause hypotheses with a precise confidence qualifier (confirmed / most-likely / possible / needs-more-evidence), proposes remediations pulled ONLY from a vetted knowledge base (never free-form), risk-classifies them, and forces human approval for anything medium-risk or above. The agent proposes; the human disposes. SENTINEL never auto-closes or runs high-risk changes alone. MICROSOFT IQ - FOUNDRY IQ Foundry IQ is SENTINEL's knowledge & grounding layer. The Historical Memory and Runbook agents ground their reasoning through Foundry IQ's agentic retrieval over the organization's incident history, runbooks and service docs, so diagnoses are grounded in what THIS org has actually seen and fixed, not generic web knowledge. Every resolved incident is written back, so the system compounds institutional knowledge: the next similar incident is recognized instantly. HOW IT'S BUILT (real, running system not a mock) • A distributed microservices system: a gateway, an orchestrator that runs the diagnostic saga, a 7-agent fan-out over a Redis Streams event bus (one consumer group per role), and a stateful memory tier deployable as a right-sized 14-container stack or a full per-stage split. • Full OpenTelemetry instrumentation across every service → an LGTM stack (Tempo/Loki/Prometheus/ Grafana). Trace context propagates across the async bus, so one /analyze is a single connected distributed trace; per-agent latency, anomalies and corroboration are live Grafana metrics. • Event-driven ingestion: point Alertmanager or a Kubernetes event stream at it and the fleet auto-triages, debounced against flapping. • A live "mission-control" UI: a real incident lights up each component of the pipeline as it executes every stage activates, stays lit, shows its real findings, and is clickable for a full drill-down (agent evidence, the correlation table, the verdict, the gated remediations, or the failure error). An interactive knowledge graph of services, dependencies, teams and incidents. A system-architecture view of how it all connects.
A Multi Cloud, AI & Observability Architect specializing in cloud-native observability solutions with a deep understanding of distributed systems and modern DevOps practices. Key contributions in automating infrastructure operations, improving self-service workflows for developers, extending platform functionalities and accelerating software delivery velocity through effective toolchain and workflow enhancements.
