Your on-call rotation doesn't page you when an AI agent gives a customer bad advice, but it should. As LLM-powered agents move from chatbot novelty into production systems that make decisions, recommend actions, and interact directly with users, SRE practices built around traditional services need to evolve to account for non-deterministic, natural-language behavior. In this talk, I’ll walk through a live AI agent, an intentionally “broken” education assistant, to demonstrate what agent observability looks like in practice. We’ll follow distributed traces across the LLM call and its downstream tool calls, see an automated evaluator catch a harmful response before it reaches a real audience, and trace a deliberate service outage end-to-end through the agent’s error-handling path. I’ll then close the loop the way an SRE would: a simple thumbs-up/down control on every agent response feeds directly into a conversation-rating signal. A dissatisfied user is no longer just a qualitative complaint. It becomes a queryable, traceable signal that can be correlated with the trace that produced it. Finally, we’ll explore what familiar SRE concepts such as SLOs, error budgets, and incident response begin to look like when a language model becomes part of the production service. Attendees will leave with a practical framework for instrumenting and operating AI agents as production dependencies, and a healthy skepticism toward the idea that AI agents somehow don't need SREs.
Wei-Chin Call is a Staff Emerging Products Manager at Grafana Labs, where she helps shape the company's next generation of agentic, full-stack observability products. She joined Grafana Labs as a Senior Solutions Engineer, partnering with customers on technical proof-of-concepts and driving adoption of the Grafana Cloud stack. Prior to Grafana Labs, Wei-Chin managed a Solutions Engineering team at Cribl and held technical evangelism and sales engineering leadership roles at AppDynamics, where she built sales enablement curricula and led cloud-native training initiatives. She holds certifications in OpenTelemetry (OTCA), Kubernetes (CKAD, CKA), and Prometheus (PCA).
