Kubernetes incidents usually start the same way: an alert fires, someone checks events, runs a few describe and logs commands, builds a hypothesis, and decides whether it's safe to take action. Although this process is familiar to most SREs, it is still largely manual, time-consuming, and often difficult to document consistently. In our team, we started asking a simple question: how much of that investigation could be automated while still keeping humans in control of critical decisions? This talk presents a two-agent AIOps architecture designed around that idea. A collector monitors Kubernetes Warning events and triggers a read-only diagnostic agent. Through Kubernetes tools exposed by MCP, the agent gathers evidence, investigates affected resources, analyzes possible causes, and generates a structured incident report that an operator can review. The key design decision was to separate diagnosis from remediation. The diagnostic agent can investigate and suggest actions, but it cannot make changes. The operator remains responsible for deciding whether a proposed remediation is appropriate. Only after explicit approval does a second agent perform the corrective action. Every step is tracked. The system validates the cluster state after remediation, records the outcome, and preserves reports, decisions, failures, and state transitions to provide a complete audit trail.
Marcelo Capilé is an SRE and Infrastructure Architect at IBM with more than 15 years of experience in IT. His background includes Linux, AIX, automation, cloud infrastructure, observability, Kubernetes, and Red Hat OpenShift, with a strong focus on solving operational problems in complex environments. He is a Certified Kubernetes Administrator (CKA) and a Red Hat Certified Architect (RHCA), specializing in OpenShift. Marcelo also contributed to the IBM Redbook Implementation Guide for Kubernetes and Red Hat OpenShift, covering the implementation of IBM Storage Scale Container Native (CNSA), and is a contributing author of Jornada Kubernetes, a collaborative Brazilian book about Kubernetes. He enjoys turning operational challenges into practical solutions and sharing what he learns through technical communities, mentoring, writing, and collaborative projects.
