AI systems are moving from demos into production, but operating them reliably requires more than the traditional CPU, memory, uptime, and error-rate signals. As LLMs, RAG systems, vector databases, AI agents, and GPU-backed inference services become part of real customer-facing environments, SRE teams are being asked to support a new class of workload. These systems do not always fail like traditional web applications. A request can return HTTP 200 while the user still experiences slow token generation, poor retrieval quality, broken tool calls, model rate limits, fallback failures, or unexpected cost spikes. This talk looks at AI workloads through an SRE lens. Instead of focusing on whether AI will replace SREs, it explores the opposite problem: what happens when AI itself becomes the production system that SRE teams must keep reliable? We will walk through the reliability path of a modern AI application, including the API layer, prompt gateway, model server, vector database, external model provider, GPU infrastructure, queues, fallbacks, and agent/tool-call workflows. From there, we will map traditional SRE concepts such as SLIs, SLOs, observability, capacity planning, progressive rollout, rollback, and incident response to AI-specific production risks. The session will cover practical AI workload indicators such as Time to First Token, end-to-end inference latency, tokens per second, queue depth, GPU saturation, retrieval success, fallback success, rate-limit errors, and cost per request. We will also discuss why CPU and memory alone are often not enough for scaling LLM inference, and why AI systems need visibility across infrastructure health, model-serving behaviour, retrieval quality, dependency health, and user experience. Attendees will leave with a simple production readiness framework for AI workloads: define the user journey, choose meaningful SLIs, set realistic SLOs, observe the full inference path, plan for capacity and cost, design safe fallbacks, prepare rollback strategies, and treat AI systems as critical production services rather than impressive demos.
Trupti is a Senior Site Reliability Engineer and Co-Founder of OpsCortex Labs, focused on Kubernetes, SRE, observability, cloud infrastructure, and AIOps. She creates practical engineering content that simplifies production reliability concepts using diagrams, real-world examples, and incident-focused explanations. Her work has reached 100K+ engineers across the DevOps and SRE community, with 1M+ impressions on technical content. She holds certifications including CKA, AWS, Linux, and HashiCorp, and is passionate about helping engineers understand complex production systems with clarity, strong fundamentals, and practical automation.
