SREday is a worldwide series of community events for engineers who build, ship and run modern software systems. Across cities around the world, we bring together people working in reliability, cloud, DevOps, observability and production engineering to share real-world experience, connect with their local community and explore how these disciplines are evolving in the age of AI.
Companies presenting:
AJ Meyers, Carlson BCM, Dalton AI, Datadog, Experian, Gatling, Glitnir Ticketing, Grainger, Guild Systems, Imubit, Kidar, Mirror Panel, Modal, Nadir, Qovery, Red Hat, Solo.io, The New York Times, TheRobotAge, Tiger Data, WWT
An AI SRE is most useful when it can act as a first responder on every alert, forming hypotheses, querying telemetry, and weighing evidence before an engineer steps in. This makes investigation quality per unit of inference cost a central design goal. In this talk, we present Scout 1.0, Datadog’s first AI model built specifically to investigate production issues. We describe how we post-trained Scout using reinforcement learning in investigation simulations built from historical incidents, along with a few lessons learned along the way. To measure investigation quality, we evaluate Scout on AlertEval, our internal benchmark that pairs real alerts and archived telemetry with root-cause explanations written by engineers. We find that Scout pushes out the cost/quality Pareto frontier: its medium-effort mode matches leading general-purpose models at 3–8× lower estimated inference cost, while its highest-effort mode reaches state-of-the-art accuracy. These gains also carry over to the third-party SREGym benchmark. Together, these results suggest that targeted post-training can make domain-specialized models a cost-effective alternative to general-purpose frontier models.... Read more
Modern AI agents don’t fail like traditional services, they fail across chains of tool calls, retrieval steps, and model decisions. There's no shortage of AI agents to pick from, but once you deploy an agent, the next hard problem is operability: What happened? What took so long? Which tool caused an error? Did the agent exfiltrate data or misuse permissions? In this session, you’ll see a practical observability architecture for AI agents using OpenTelemetry and MLflow. From how to correlate a user request with every tool invocation, to enforcing consistent trace context across agent to orchestrator to tool servers, and deriving agent-specific signals (overall success rate, step latency, retries, “time to first action,” and token/tool cost). We’ll run a live demo of a real end-to-end flow with production checklists for dashboards, alerts, redactions, and AgentOps guardrails that make agents trustworthy.... Read more
AI has dramatically reduced the complexity of starting a new code project. Today, you don't read the docs first - you ask your coding agent. In this talk, we'll show how this applies to performance testing. How developers with zero Gatling experience can write realistic load tests in minutes, using AI as their entry point instead of documentation, and why this changes how teams adopt performance testing altogether.... Read more
Agents need to be isolated for what they can and can't access (files/folders, production envs, LLMs, MCP tools, runtimes, etc. - think autonomous and standard Agents) and at the same time, scale up and down based on load, if/how the Agent is being used, and how it's being used. By default, Agents have access to every file, LLM, command, and authN/Z implementation that you do. At the same time, you're running Agents in a scalable, performant, and reliable way with a proven out orchestration/clustering platform. Example: You want to run 200+ Agents, but only spread across 6-8 k8s Pods. The Agents still perform as well as expected while keeping the underlying infrastructure costs (both resources and actual financial costs) low. In this session, you will learn about the OSS project that makes this happen - Agent Substrate, with key topics including: Build on top of k8s primitives (clustering of resources for performant sandboxed Agents). Implement a specialized and efficient Control/Management Plane that's built for the world of agentic. How Substrate Actors running Agents are 90% more efficient (proven in benchmarks) for cost optimization in comparison to Agents in k8s Pods. Efficient and much smaller footprints for running Agents via Actors.... Read more
With more and more tools abstracting from a developer's workflows, understanding how git visualization tools help - not simply using it - is more important than ever. In this talk, we take a look at the history of git workflows, the basics of git, and the renaissance of understanding version control in a world gone mad for “vibe coding”.... Read more
A cheaper model does not always mean a cheaper run. Routing between models, trimming context, splitting work across agents, or adding tools and skills mid-run can look like smart optimizations. But when those changes disrupt prompt-cache reuse, they can turn discounted, reusable context into input you pay to process again—undermining the savings you expected. OpenAI Developers This talk explores why prompt caching deserves to be an architectural consideration, not an afterthought, in production LLM systems. We’ll examine how caching interacts with model routing, context management, subagents, and dynamic tool loading. Through concrete examples, we’ll compare preserving a warm cache with starting fresh, explore when breaking the cache is worth the trade-off, and show how to make these decisions based on the economics of an entire run rather than the price of an individual request. Attendees will leave with practical patterns for cache-aware workflows and an observability checklist covering cache reuse, cache-read and write costs, latency, and cost per successfully completed task. The takeaway: don’t optimize away your biggest savings. Optimize the whole run—not just the token count.... Read more
Over the past three years, I've collaborated with hundreds of SRE teams as they've integrated AI into their workflows. What has truly evolved - and what remains the same? This talk will provide practical insights and observations from the forefront of AI adoption in infrastructure. I'll discuss: The evolving skill set: What SREs need to learn today and which skills stand the test of time AI in incident management: Identifying where AI agents add value, where they fall short, and how to distinguish between the two AI SRE: What's real, and what's just hype... Read more
Claude Code, Codex, and other coding agents are becoming long-running, tool-using systems that read repositories, execute commands, modify code, call external services, and make decisions with limited human supervision. Yet most teams observe them primarily through token counts and final pull requests. This talk applies familiar SRE practices to agentic development. We’ll instrument a coding-agent workflow with OpenTelemetry, examine traces of tool calls, failures, retries, tests and code changes, and look at what those traces reveal about reliability that the final output hides. We’ll cover practical failure patterns, useful signals, and where traditional observability concepts translate well - and where they don’t.... Read more
Moving a Lambda into a service looks straightforward: wrap the handler in an API or a queue consumer and deploy. That’s what we did at first when we migrated seven Python data-ingestion Lambdas to k8s, and the early tests passed, including concurrent deliveries. As we moved more customers over and left the services running for days, code that had worked on Lambda for years started breaking. Liveness probes timed out, pods kept running out of memory during DLQ replay, tmp files collided, connections dropped and credentials expired. I’ve organised the talk around four promises we relied on by using Lambda: blocking was free, every delivery had the place to itself, nothing lived long enough to go bad, and the outside world saw a different you. We relied both on Lambda’s documented behaviour and on what we’d observed while running our application there. Moving to services forced us to revisit how we managed concurrency, connections and credentials once deliveries shared resources and processes lived longer. I’ll use evidence from the migration to explain each promise, including how fixing the blocking calls made the memory problem easier to trigger. I’ll also cover what to test alongside a rollout, including replay, sustained load and credential rotation. I’ll finish with a checklist for identifying what your application depends on its runtime to do, and what you’ll need to account for when that runtime changes.... Read more
As engineering teams accelerate software delivery with AI, documentation struggles to keep pace. Outdated runbooks, API references, and technical guides can create operational risks, slow incident response, and undermine platform reliability. What if documentation could evolve alongside the code itself? This talk explores how AI agents can detect documentation drift, identify changes in code and infrastructure, and automatically propose updates as part of the engineering workflow. We'll examine integrations with tools like Jira, Confluence, and GitHub through Model Context Protocol (MCP), along with automated validation, CI/CD integration, human approval, and governance. Through a practical example, attendees will learn how to approach documentation as a continuously maintained, verifiable component of software delivery rather than an afterthought.... Read more
AI agents can write migrations, but production changes need guardrails. This live demonstration walks through an agent-assisted PostgreSQL workflow using Tiger Cloud and the Tiger CLI: provisioning an isolated development service, generating versioned migrations, and testing correctness and performance before human-approved production deployment. We’ll cover scoped access, automated checks, lock and rewrite detection, deployment monitoring, and rollback criteria. Attendees will leave with a practical blueprint for accelerating database delivery without giving agents unchecked authority over production.... Read more
Growing alerts, new sources, growing volumes of telemetry. This talk shows an SRE Coworker that is always-on to correlate and detect subtle anomalies across sources, investigate complex hypotheses, and surface root causes automatically. Attend this session to discover how your team can wake up to zero alerts and fully explained incidents while you sleep.... Read more
17:00
Wrap up
Scan each other's QR codes & head to a nearby pub!
You can observe every component of an enterprise and still not understand the health of the enterprise. SRE practice tells us to set objectives around what users care about, but those objectives rarely connect to a governed account of what the enterprise actually does. The missing object isn’t another metric, dashboard or SLO. It’s the business capability: the enduring definition of what the enterprise must be able to do, independent of the systems that deliver it. This talk sets out how a capability-based topology, running from business capability through applications and data to infrastructure, can become the organizing foundation for service health, semantic analysis and AI-driven diagnostics. It compares three approaches: adding capabilities to existing observability tools, making the capability model the organizing structure for service health, and using AI agents to engineer observability against topologies that change constantly. The argument throughout is that capabilities are the layer worth designing for, because they remain stable as the technology beneath them changes.... Read more
As AI agents move from experimentation into production, they are increasingly capable of executing commands, accessing enterprise systems, modifying infrastructure, and making operational decisions without continuous human supervision. But what happens when an autonomous agent makes the wrong decision at 2 AM? Who detects it, who has authority to intervene, and how do we stop the action before it becomes a production incident? Traditional SRE practices provide powerful mechanisms for observability, reliability, incident response, and recovery. Autonomous AI introduces additional challenges involving agent identity, dynamic authorization, decision accountability, and runtime enforcement. This session introduces a practical architecture for bringing AI governance directly into production operations. We'll explore how OpenTelemetry, agent registries, identity-aware policies, authorization gateways, human approval workflows, runtime monitoring, and automated kill switches can work together to make autonomous systems safer and more reliable. Through a realistic production incident scenario, we'll examine how an AI agent's proposed action can be evaluated, authorized, blocked, escalated, and documented before unintended consequences occur. Attendees will leave with actionable architectural patterns for connecting SRE practices with enforceable AI governance. The central message: Observing an AI agent is not the same as controlling it. Reliable autonomous systems require both.... Read more
While the industry chases agent reliability through smarter models, most agent failures trace to one design choice: letting the model decide what happens next. I built a software factory where the model only writes the code, while a deterministic controller drives Claude Code and Codex workers through the entire workflow, grounded by executable gates and engineering directives, with every transition, retry budget, and terminal owned by a versioned graph. I'll show the architecture, the failure classes it eliminates by construction, and what adversarial battle-testing taught me about where agent autonomy actually breaks.... Read more
New to TPUs? This talk starts with what a Cloud TPU is, how it differs from a GPU, and how to get one on Google Cloud. Then it moves to a live demo: an MCP server that lets an AI agent find TPU capacity, create a TPU VM, and start vLLM on it. We send the server a few requests, watch it answer, and tear everything down so nothing keeps billing. You leave with the steps to run your first model on a TPU.... Read more
Everything breaks. Bad deploys, traffic spikes, cloud outages, eventually some component will go down. When architecting a system, our instinct can be to try and eliminate any possible impact. But this can add complexity that paradoxically makes failures more likely, or make recovery harder. Let's look at an alternative: embracing degradation to cushion against a total outage. We'll look at a real world example of multi-region routing, explore a catalogue of failure modes and trade-offs that can blunt them, and highlight when compromise isn't an option. Fail safe without having to be failure-proof!... Read more
While observability is the first step towards building observable systems, monitoring is what enables us to action on the telemetry collected in a more automated way. Previous approaches to monitoring have relied primarily on infrastructure and service metrics, but modern approaches have embraced the idea of monitoring based on metrics that reflect the user experiences that consider business impact more accurately. In this, we're enabled to drive clarity over reliability targets, effectively prioritize reliability improvements, and hold ourselves accountable to customers. This talk will elaborate on those differences and review traditional and new approaches to monitoring, including the role of Service Level Objectives (SLOs). Additionally, we’ll cover the following: Observability vs Monitoring Metrics and the types of Metrics & Monitors The benefits of an SLO-based approach to monitoring Effective Accountability with Error Budget Policies... Read more
You build it, you run it. Fifteen years of SRE gospel, and almost nobody questions it. The rule was never about writing code. It was about a loop. Whoever wrote it understands it, so whoever wrote it carries the pager. Understanding was the point. The pager was just enforcement. Agents cut that loop. The code still ships. The author is gone. I take the assumption apart, one joint at a time. No human author, so who holds the pager. Nobody who can explain it, so the author cannot answer the 3am question. Review cannot keep up with the output, so your gate is a rubber stamp. Then the fixes that do not work, and why each one folds. Human in the loop. Make the agent better. Keep agent code out of prod. Then the reframe. If writing it no longer means understanding it, reliability has to come from somewhere else. Operability stops being something you hope the author cared about, and becomes something you check for. One real incident carries the argument. A single S3 copy call, a 5 GiB limit nobody checked, a migration that never sealed, and a cluster crash-looping while the log line said migrating objects: . None of it was hard. Nobody could see it. The principle is not dead. The assumption under it is.... Read more
Modern observability platforms make it easy to collect browser telemetry. The real challenge is deciding what not to collect. At Grainger, our digital platforms process approximately 1.5 billion requests per day across multiple brands, countries, and customer-facing applications. As our Real User Monitoring (RUM) adoption expanded, we discovered that a significant portion of our telemetry was generated by bots, synthetic monitoring solutions, automated testing platforms, internal tooling, and duplicate browser events. While these events increased observability costs, they provided little value when understanding real customer experiences. This session shares our journey modernizing browser observability during a Datadog RUM migration and transforming RUM from a "collect everything" model into a scalable, business-focused observability platform. Attendees will learn how we: Identified and filtered bot, synthetic, and automated traffic before it reached our observability platform Reduced browser event ingestion by nearly 80% (7.81 million to 1.67 million events) while maintaining visibility into real user experiences Improved session accuracy by addressing split-session behavior, cookie inconsistencies, and telemetry fragmentation Implemented intelligent filtering across edge, application, and browser layers to reduce noise and improve signal quality Balanced engineering observability, business analytics, compliance requirements, and platform costs Established a repeatable framework for determining which browser telemetry delivers operational value and which should be excluded Beyond the technical implementation, this talk explores the broader challenge facing modern SRE and observability teams: collecting the right data rather than simply collecting more data. The result was a cleaner signal, faster incident investigations, reduced observability spend, and greater confidence that our dashboards, alerts, and analytics reflect real customer behavior—not automated traffic. Whether you use Datadog, OpenTelemetry, New Relic, Dynatrace, Elastic, or another observability platform, you'll leave with practical techniques and architectural patterns to navigate RUM without limits.... Read more
We keep spare infrastructure running for the worst day of the year. It consumes resources every other day. We wanted to know how much standby capacity we actually need to buy the recovery objective we promise. We built the same Kubernetes service with three disaster recovery strategies: pilot light, warm standby, and hot standby. Then we failed the primary region and measured what happened. The tradeoff was visible on both sides. Keeping more capacity ready reduced recovery time, but increased energy use while waiting for an incident. Keeping less capacity idle saved energy, but made recovery depend on provisioning, autoscaling, cache warming, and downstream capacity exactly when the system was already under stress. We'll compare each strategy using recovery time, p99 latency, failed requests, error budget burn, and workload energy measured during both idle and recovery periods. We will also look at where scaling from a smaller standby stops being fast enough to satisfy the RTO. We'll finish with a practical method for sizing standby capacity from an RTO instead of assuming that more redundancy is always better.... Read more
"The fix that closes an incident is rarely the fix that prevents the next one." Finding the root cause feels like the finish line, but it's really the starting point for the work that decides whether an incident becomes a one-time story or a recurring pattern. Problem management and Change management are where that decision actually gets made — yet they are often treated as paperwork, instead of the discipline that turns a lucky catch into a durable fix. In this talk I will walk through: How a single root cause can quietly hide an entire class of risk across a codebase. Why the obvious fix is sometimes the shallow one. How the guardrails built into change management — pausing risky rollouts, tightening review for a category of change, automating remediation — determine whether a fix actually sticks. I will share a simple framework for telling apart "fixed the symptom", "fixed the incident" and "fixed the system" plus the follow-up questions that stop a team from closing a problem record too early. Attendees will leave with a sharper way to evaluate their own post-incident fixes, and a short list of questions to ask before marking a problem "resolved".... Read more
16:30
Wrap up
Scan each other's QR codes & head to a nearby pub!
The New York Times Building
5620 8th Avenue, 47th Floor,
New York, NY 10018, United States
Sponsors & Partners
Want to become a sponsor? Get in touch!
Cedric Clyburn
Red Hat
Observing AI Agents in Production
Abstract
Modern AI agents don’t fail like traditional services, they fail across chains of tool calls, retrieval steps, and model decisions. There's no shortage of AI agents to pick from, but once you deploy an agent, the next hard problem is operability: What happened? What took so long? Which tool caused an error? Did the agent exfiltrate data or misuse permissions?
In this session, you’ll see a practical observability architecture for AI agents using OpenTelemetry and MLflow. From how to correlate a user request with every tool invocation, to enforcing consistent trace context across agent to orchestrator to tool servers, and deriving agent-specific signals (overall success rate, step latency, retries, “time to first action,” and token/tool cost). We’ll run a live demo of a real end-to-end flow with production checklists for dashboards, alerts, redactions, and AgentOps guardrails that make agents trustworthy.
Bio
Cedric Clyburn (@cedricclyburn), Senior Developer Advocate at Red Hat, is an enthusiastic software developer with a background in Kubernetes, DevOps, and container tools. Focused on open-source software, he both contributes (e.g., Podman, vLLM) and enjoys speaking, with prior experience at Devoxx, WeAreDevelopers, The Linux Foundation, and more. Cedric also spends (too much) time creating video and written content helping developers learn new topics in emerging technologies, with over 2M+ views online. He’s based in New York City and is an organizer of the local Kubernetes Community Day.
Paul-Henri Pillet
Gatling
Don't RTFM, use AI instead for your first Gatling tests!
Abstract
AI has dramatically reduced the complexity of starting a new code project. Today, you don't read the docs first - you ask your coding agent. In this talk, we'll show how this applies to performance testing. How developers with zero Gatling experience can write realistic load tests in minutes, using AI as their entry point instead of documentation, and why this changes how teams adopt performance testing altogether.
Bio
Paul-Henri Pillet is the CEO of Gatling, one of France’s most successful open source technology companies and a leading provider of high performance load testing solutions. Since joining Gatling in 2014, he has helped grow the platform into a globally trusted tool used by hundreds of thousands of organizations to ensure the scalability and reliability of their applications. Based in Paris, Paul-Henri combines a background in business, public policy, and entrepreneurship, with degrees from HEC Paris and Freie Universität Berlin.
Michael Levan
Solo.io
Sandboxed Agents: Building Isolated, Governed and Scalable AI
Abstract
Agents need to be isolated for what they can and can't access (files/folders, production envs, LLMs, MCP tools, runtimes, etc. - think autonomous and standard Agents) and at the same time, scale up and down based on load, if/how the Agent is being used, and how it's being used.
By default, Agents have access to every file, LLM, command, and authN/Z implementation that you do. At the same time, you're running Agents in a scalable, performant, and reliable way with a proven out orchestration/clustering platform.
Example: You want to run 200+ Agents, but only spread across 6-8 k8s Pods. The Agents still perform as well as expected while keeping the underlying infrastructure costs (both resources and actual financial costs) low.
In this session, you will learn about the OSS project that makes this happen - Agent Substrate, with key topics including:
Build on top of k8s primitives (clustering of resources for performant sandboxed Agents).
Implement a specialized and efficient Control/Management Plane that's built for the world of agentic.
How Substrate Actors running Agents are 90% more efficient (proven in benchmarks) for cost optimization in comparison to Agents in k8s Pods.
Efficient and much smaller footprints for running Agents via Actors.
Bio
Michael Levan translates technical complexity into practical value. He is a Principal AI Architect and Forward Deployed Engineer in the AI and Platform Engineering space who spends his time working with organizations around the globe on technical implementation and strategy. Michael is also a Microsoft MVP, 4x published author, podcast host, international public speaker, AAIF Ambassador, CNCF Ambassador, and was part of the Kubernetes v1.28 and v1.31 Release Team.
PJ Hagerty
WWT
Why Dev Tools Still Matter
Abstract
With more and more tools abstracting from a developer's workflows, understanding how git visualization tools help - not simply using it - is more important than ever. In this talk, we take a look at the history of git workflows, the basics of git, and the renaissance of understanding version control in a world gone mad for “vibe coding”.
Bio
PJ Hagerty is a well-known figure in the tech industry, particularly within the developer relations and DevOps communities. He is recognized for his work as a developer advocate, community builder, writer, and speaker. PJ founded DevRelate.io, a company focused on helping tech organizations build communities and enhance developer relations. He has a strong presence in the open-source and tech communities, often speaking at conferences and producing content on topics such as software development, DevOps, and community engagement.
Dor Amir
Nadir
Cache Is King: The Hidden Cost of LLM Optimization
Abstract
A cheaper model does not always mean a cheaper run. Routing between models, trimming context, splitting work across agents, or adding tools and skills mid-run can look like smart optimizations. But when those changes disrupt prompt-cache reuse, they can turn discounted, reusable context into input you pay to process again—undermining the savings you expected. OpenAI Developers
This talk explores why prompt caching deserves to be an architectural consideration, not an afterthought, in production LLM systems. We’ll examine how caching interacts with model routing, context management, subagents, and dynamic tool loading. Through concrete examples, we’ll compare preserving a warm cache with starting fresh, explore when breaking the cache is worth the trade-off, and show how to make these decisions based on the economics of an entire run rather than the price of an individual request.
Attendees will leave with practical patterns for cache-aware workflows and an observability checklist covering cache reuse, cache-read and write costs, latency, and cost per successfully completed task. The takeaway: don’t optimize away your biggest savings. Optimize the whole run—not just the token count.
Bio
Dor Amir is the founder of Nadir, an LLM infrastructure company focused on making AI systems more efficient. Previously, he worked on machine learning and personalization at Dropbox, Amazon, Guesty, and Fiverr. His work focuses on production AI systems, LLM routing, agents, and reducing the cost and latency of running AI at scale.
Romaric Philogene
Qovery
SRE in the Era of AI: Lessons Learned from Hundreds of Teams I've met Over 3 Years
Abstract
Over the past three years, I've collaborated with hundreds of SRE teams as they've integrated AI into their workflows. What has truly evolved - and what remains the same? This talk will provide practical insights and observations from the forefront of AI adoption in infrastructure. I'll discuss:
The evolving skill set: What SREs need to learn today and which skills stand the test of time
AI in incident management: Identifying where AI agents add value, where they fall short, and how to distinguish between the two
AI SRE: What's real, and what's just hype
Bio
Romaric Philogene is CEO & Co-founder of Qovery, the Agentic infrastructure platform powering hundreds of companies like Alan, Talkspace, Zoom..With 20+ years in infrastructure engineering, he started his career when cloud didn't exist - managing physical servers in HFT, getting 2 AM pages, and building runbooks from scratch. Prior to Qovery (founded 2020), he was CEO of two other companies self funded and sold to major brands in ERP for automotive.
Elliot Gardiner
Mirror Panel Inc.
Production Has Observability. Your Coding Agent Doesn’t
Abstract
Claude Code, Codex, and other coding agents are becoming long-running, tool-using systems that read repositories, execute commands, modify code, call external services, and make decisions with limited human supervision. Yet most teams observe them primarily through token counts and final pull requests. This talk applies familiar SRE practices to agentic development. We’ll instrument a coding-agent workflow with OpenTelemetry, examine traces of tool calls, failures, retries, tests and code changes, and look at what those traces reveal about reliability that the final output hides. We’ll cover practical failure patterns, useful signals, and where traditional observability concepts translate well - and where they don’t.
Bio
Elliot Gardiner is a software engineer and former principal SRE with 13 years of experience building and operating large-scale systems. He is the founder of Code Telemetry, an observability platform for AI coding agents, and co-founder of Mirror Panel, a synthetic research platform focused on measuring and improving the reliability of AI-generated audience insight. His current work sits at the intersection of AI systems, observability, developer infrastructure, and reliability.
Denys Zhak
Imubit
What AWS Lambda Was Hiding: Runtime Assumptions Exposed by a Migration
Abstract
Moving a Lambda into a service looks straightforward: wrap the handler in an API or a queue consumer and deploy. That’s what we did at first when we migrated seven Python data-ingestion Lambdas to k8s, and the early tests passed, including concurrent deliveries. As we moved more customers over and left the services running for days, code that had worked on Lambda for years started breaking. Liveness probes timed out, pods kept running out of memory during DLQ replay, tmp files collided, connections dropped and credentials expired.
I’ve organised the talk around four promises we relied on by using Lambda: blocking was free, every delivery had the place to itself, nothing lived long enough to go bad, and the outside world saw a different you. We relied both on Lambda’s documented behaviour and on what we’d observed while running our application there. Moving to services forced us to revisit how we managed concurrency, connections and credentials once deliveries shared resources and processes lived longer.
I’ll use evidence from the migration to explain each promise, including how fixing the blocking calls made the memory problem easier to trigger. I’ll also cover what to test alongside a rollout, including replay, sustained load and credential rotation. I’ll finish with a checklist for identifying what your application depends on its runtime to do, and what you’ll need to account for when that runtime changes.
Bio
Denys Zhak is a software engineer with nine years of experience in backend engineering and distributed systems, focused on production reliability. At Imubit, he builds cloud and on-prem software for industrial machine-learning deployments. He led the six-month migration of seven Python ingestion Lambdas into services described in this talk. Outside work, he contributes to open source. Much of that work has been on Astral’s Ruff and ty. He writes about production engineering at denyszhak.com.
Michael Posso
TheRobotAge
Your Code Deploys. Your Docs Drift: Can AI Agents Fix That?
Abstract
As engineering teams accelerate software delivery with AI, documentation struggles to keep pace. Outdated runbooks, API references, and technical guides can create operational risks, slow incident response, and undermine platform reliability.
What if documentation could evolve alongside the code itself?
This talk explores how AI agents can detect documentation drift, identify changes in code and infrastructure, and automatically propose updates as part of the engineering workflow. We'll examine integrations with tools like Jira, Confluence, and GitHub through Model Context Protocol (MCP), along with automated validation, CI/CD integration, human approval, and governance.
Through a practical example, attendees will learn how to approach documentation as a continuously maintained, verifiable component of software delivery rather than an afterthought.
Bio
Michael Posso is a technology leader, engineer, and educator with over 15 years of experience in web development, cloud platforms, and digital products. He currently serves as Vice President at JPMorgan Chase, focusing on product engineering, go-to-market (GTM) strategy, and AI platform adoption. Previously, he led web engineering at AECOM, developing enterprise web experiences for global audiences. He is also the founder of The Robot Age, where he explores robotics, human-robot interaction, and the future of intelligent systems.
Giovanni Martinez
Tiger Data
Agents Can Write Migrations. They Shouldn’t Ship Them.
Abstract
AI agents can write migrations, but production changes need guardrails. This live demonstration walks through an agent-assisted PostgreSQL workflow using Tiger Cloud and the Tiger CLI: provisioning an isolated development service, generating versioned migrations, and testing correctness and performance before human-approved production deployment. We’ll cover scoped access, automated checks, lock and rewrite detection, deployment monitoring, and rollback criteria. Attendees will leave with a practical blueprint for accelerating database delivery without giving agents unchecked authority over production.
Bio
Giovanni Martinez is a Database Support Engineer at Tiger Data and an AWS Community Builder with more than 10 years of PostgreSQL experience. He specializes in database performance, production reliability, cloud-native PostgreSQL, TimescaleDB, and agent-assisted database operations. Through The Postgres Guy and his open-source iqtoolkit project, Giovanni creates practical tools and educational content that help engineers troubleshoot, optimize, and operate PostgreSQL systems safely.
AJ Meyers
AJ Meyers
Always On SRE CoWorker
Abstract
Growing alerts, new sources, growing volumes of telemetry. This talk shows an SRE Coworker that is always-on to correlate and detect subtle anomalies across sources, investigate complex hypotheses, and surface root causes automatically. Attend this session to discover how your team can wake up to zero alerts and fully explained incidents while you sleep.
Bio
AJ Meyers is a technology strategist, architect, and leader who helps businesses turn advances in AI, data, and emerging technologies into opportunities for growth and competitive advantage. His career spans business strategy, product development, organizational leadership, and enterprise operations, combining technical depth with experience building teams and directing large-scale technology investments.
Currently a Principal Solutions Architect at Elastic, AJ brings extensive experience across leading technology companies and consultancies, including Teradata, FICO, Deloitte Digital, and EY.
Raymond Carlson
Carlson BCM
Observing the Enterprise, Not Just the Stack
Abstract
You can observe every component of an enterprise and still not understand the health of the enterprise. SRE practice tells us to set objectives around what users care about, but those objectives rarely connect to a governed account of what the enterprise actually does. The missing object isn’t another metric, dashboard or SLO. It’s the business capability: the enduring definition of what the enterprise must be able to do, independent of the systems that deliver it.
This talk sets out how a capability-based topology, running from business capability through applications and data to infrastructure, can become the organizing foundation for service health, semantic analysis and AI-driven diagnostics. It compares three approaches: adding capabilities to existing observability tools, making the capability model the organizing structure for service health, and using AI agents to engineer observability against topologies that change constantly.
The argument throughout is that capabilities are the layer worth designing for, because they remain stable as the technology beneath them changes.
Bio
Raymond Carlson is the founder and principal of Carlson BCM, a business capability architecture practice for insurance and financial services. Over three decades he has led design authority on enterprise transformations across insurance, banking and capital markets, connecting strategy, operations and technology to a single governed model of what the enterprise must be able to do. His recent work applies that model to observability and service health, anchoring enterprise topology, operational data and AI-driven diagnostics to the business capabilities they support. He is the author of the Carlson BCM framework and publishes the weekly newsletter Carlson BCM Insights.
Marcelo Lorenzetti
Guild Systems
Who Stops the AI Agent at 2 AM?
Abstract
As AI agents move from experimentation into production, they are increasingly capable of executing commands, accessing enterprise systems, modifying infrastructure, and making operational decisions without continuous human supervision.
But what happens when an autonomous agent makes the wrong decision at 2 AM? Who detects it, who has authority to intervene, and how do we stop the action before it becomes a production incident?
Traditional SRE practices provide powerful mechanisms for observability, reliability, incident response, and recovery. Autonomous AI introduces additional challenges involving agent identity, dynamic authorization, decision accountability, and runtime enforcement.
This session introduces a practical architecture for bringing AI governance directly into production operations. We'll explore how OpenTelemetry, agent registries, identity-aware policies, authorization gateways, human approval workflows, runtime monitoring, and automated kill switches can work together to make autonomous systems safer and more reliable.
Through a realistic production incident scenario, we'll examine how an AI agent's proposed action can be evaluated, authorized, blocked, escalated, and documented before unintended consequences occur.
Attendees will leave with actionable architectural patterns for connecting SRE practices with enforceable AI governance.
The central message: Observing an AI agent is not the same as controlling it. Reliable autonomous systems require both.
Bio
Marcelo Lorenzetti is Chief AI Officer and Principal AI Architect at Guild Systems, specializing in enterprise AI architecture, agentic AI governance, cloud infrastructure, and operational resilience. With extensive experience leading enterprise technology architecture and AI transformation initiatives in financial services and other regulated environments, he focuses on turning AI governance principles into enforceable production controls. His work spans autonomous agent orchestration, identity and authorization, observability, runtime policy enforcement, operational risk, and AI economics. Marcelo also serves on the New York Law School AI Blue Ribbon Committee and is an author and speaker on enterprise AI, responsible innovation, and technology governance. His approach bridges strategic governance and practical engineering, helping organizations deploy autonomous AI systems that are reliable, accountable, and operationally controllable.
Marcelo Labre
Kidar
Agentic Engineering: The Agent Writes the Code, Not the Process
Abstract
While the industry chases agent reliability through smarter models, most agent failures trace to one design choice: letting the model decide what happens next. I built a software factory where the model only writes the code, while a deterministic controller drives Claude Code and Codex workers through the entire workflow, grounded by executable gates and engineering directives, with every transition, retry budget, and terminal owned by a versioned graph. I'll show the architecture, the failure classes it eliminates by construction, and what adversarial battle-testing taught me about where agent autonomy actually breaks.
Bio
Marcelo is Founder of Kidar AI, an AI agency helping investment firms and capital allocators turn frontier AI into governed operating systems that maximize ROI. Previously, he co-founded and led Viridios AI, bringing transparency to climate markets through an AI platform. He guided the company through Series A/B fundraising and its acquisition by Bloomberg in 2024. He is also a mentor at C10 Labs, a startup advisor, and an AI researcher at the Advanced Institute for Artificial Intelligence (AI2). His career spans organizations such as Morgan Stanley, Standard Bank, OCBC Bank, and ING Bank across London, Singapore, and New York. He has taught master’s-level machine learning at NUS and carbon finance at London Business School. Marcelo holds a PhD in Mathematical Finance from Imperial College London and a Master’s in Finance from London Business School.Marcelo is also deeply involved in the venture investing ecosystem in the US, as a VC Partner at Advantary Capital Partners, a Mentor at C10 Labs and an advisor of early stage startups. He is also a Member of the Advanced Institute for Artificial Intelligence (AI2), where he explores topics in state-of-the-art Generative AI as an AI researcher. His career developed in the financial industry with roles at Morgan Stanley, Standard Bank, OCBC Bank and ING Bank in London, Singapore, and New York. He has also held academic positions, teaching Masters-level courses in machine learning and carbon finance at the National University of Singapore (NUS) and London Business School (LBS), respectively.
Marcelo holds a PhD in Mathematical Finance from Imperial College London and a Master's in Finance from London Business School, bringing a powerful blend of engineering rigor and financial acumen to his work.
William McLean
Glitnir Ticketing
Inference on Cloud TPU
Abstract
New to TPUs? This talk starts with what a Cloud TPU is, how it differs from a GPU, and how to get one on Google Cloud. Then it moves to a live demo: an MCP server that lets an AI agent find TPU capacity, create a TPU VM, and start vLLM on it. We send the server a few requests, watch it answer, and tear everything down so nothing keeps billing. You leave with the steps to run your first model on a TPU.
Bio
CTO @ Glitnir Ticketing | GDE AI+Cloud ☁️ | AWS Community Builder AI | Google Cloud Ambassador | 9x Google Cloud Certified
Tom Elliott
Modal
Degraded is better than down
Abstract
Everything breaks. Bad deploys, traffic spikes, cloud outages, eventually some component will go down. When architecting a system, our instinct can be to try and eliminate any possible impact. But this can add complexity that paradoxically makes failures more likely, or make recovery harder.
Let's look at an alternative: embracing degradation to cushion against a total outage. We'll look at a real world example of multi-region routing, explore a catalogue of failure modes and trade-offs that can blunt them, and highlight when compromise isn't an option. Fail safe without having to be failure-proof!
Bio
Tom Elliott works on the Platforms team at Modal, with responsibility for shared infrastructure, observability, CI/CD and developer productivity tooling. Over the course of 20 years in the industry, Tom has worked within organizations of various sizes and levels of operational maturity. When not building infra for infra, Tom occasionally writes about the field at telliott.me.
Lesley Cordero
The New York Times
Actionable Observability
Abstract
While observability is the first step towards building observable systems, monitoring is what enables us to action on the telemetry collected in a more automated way. Previous approaches to monitoring have relied primarily on infrastructure and service metrics, but modern approaches have embraced the idea of monitoring based on metrics that reflect the user experiences that consider business impact more accurately. In this, we're enabled to drive clarity over reliability targets, effectively prioritize reliability improvements, and hold ourselves accountable to customers.
This talk will elaborate on those differences and review traditional and new approaches to monitoring, including the role of Service Level Objectives (SLOs). Additionally, we’ll cover the following:
Observability vs Monitoring
Metrics and the types of Metrics & Monitors
The benefits of an SLO-based approach to monitoring
Effective Accountability with Error Budget Policies
Bio
Lesley Cordero is currently a Staff Software Engineer, Tech Lead at The New York Times. She has spent the majority of her career on edtech teams as an engineer, including Google for Education and other edtech startups. In her current role, she is focused on resilience engineering, observability, shared platforms, and building excellent teams by setting reliability vision & strategy across The Times, improving our observability footprint, and cultivating culture that builds with the most vulnerable employees in mind first. She shows care for others by holding them accountable to the best versions of themselves – and by buying them the occasional bubble tea.
Itamar Knafo
Dalton AI
You Build It, You Run It Is Dead: Reliability When the Author Is an Agent
Abstract
You build it, you run it. Fifteen years of SRE gospel, and almost nobody questions it. The rule was never about writing code. It was about a loop. Whoever wrote it understands it, so whoever wrote it carries the pager. Understanding was the point. The pager was just enforcement.
Agents cut that loop. The code still ships. The author is gone.
I take the assumption apart, one joint at a time. No human author, so who holds the pager. Nobody who can explain it, so the author cannot answer the 3am question. Review cannot keep up with the output, so your gate is a rubber stamp.
Then the fixes that do not work, and why each one folds. Human in the loop. Make the agent better. Keep agent code out of prod.
Then the reframe. If writing it no longer means understanding it, reliability has to come from somewhere else. Operability stops being something you hope the author cared about, and becomes something you check for.
One real incident carries the argument. A single S3 copy call, a 5 GiB limit nobody checked, a migration that never sealed, and a cluster crash-looping while the log line said migrating objects: <nil>. None of it was hard. Nobody could see it.
The principle is not dead. The assumption under it is.
Bio
Itamar Knafo is the co-founder and CEO of Dalton AI. Previously, has spent five years building and leading software, DevOps, observability, and operations teams. He is a hands-on engineer with a deep background of cloud infrastructure, and open source, and now focuses on applying AI to production engineering.
Anand Mattah Chenna Kesavalu
Grainger
Navigate RUM Without Limits: Cutting 80% of Browser Telemetry Noise While Improving Observability
Abstract
Modern observability platforms make it easy to collect browser telemetry. The real challenge is deciding what not to collect.
At Grainger, our digital platforms process approximately 1.5 billion requests per day across multiple brands, countries, and customer-facing applications. As our Real User Monitoring (RUM) adoption expanded, we discovered that a significant portion of our telemetry was generated by bots, synthetic monitoring solutions, automated testing platforms, internal tooling, and duplicate browser events. While these events increased observability costs, they provided little value when understanding real customer experiences.
This session shares our journey modernizing browser observability during a Datadog RUM migration and transforming RUM from a "collect everything" model into a scalable, business-focused observability platform.
Attendees will learn how we:
Identified and filtered bot, synthetic, and automated traffic before it reached our observability platform
Reduced browser event ingestion by nearly 80% (7.81 million to 1.67 million events) while maintaining visibility into real user experiences
Improved session accuracy by addressing split-session behavior, cookie inconsistencies, and telemetry fragmentation
Implemented intelligent filtering across edge, application, and browser layers to reduce noise and improve signal quality
Balanced engineering observability, business analytics, compliance requirements, and platform costs
Established a repeatable framework for determining which browser telemetry delivers operational value and which should be excluded
Beyond the technical implementation, this talk explores the broader challenge facing modern SRE and observability teams: collecting the right data rather than simply collecting more data.
The result was a cleaner signal, faster incident investigations, reduced observability spend, and greater confidence that our dashboards, alerts, and analytics reflect real customer behavior—not automated traffic.
Whether you use Datadog, OpenTelemetry, New Relic, Dynatrace, Elastic, or another observability platform, you'll leave with practical techniques and architectural patterns to navigate RUM without limits.
Bio
Anand is a Staff Software Engineer and Platform SRE at Grainger, where he serves as the enterprise subject matter expert for CDN, edge security, bot management, observability, and web performance. He is responsible for reliability and performance initiatives supporting more than 1,000 services and APIs across Grainger's digital platforms.
Anand has over 20 years of experience in software engineering, site reliability engineering, and cloud infrastructure. He has led large-scale implementations involving Datadog, Akamai, DataDome, AWS, and enterprise observability platforms. His work includes browser observability optimization, bot mitigation, edge security modernization, and platform reliability engineering. Anand is also the author of a several guest blogs at leading vendors including Datadog RUM [https://www.datadoghq.com/blog/grainger-rum-costs-from-bot-traffic/]
He holds numerous industry certifications including AWS Machine Learning Speciality, AWS Solutions Architect Professional, AWS Certified Generative AI Developer - Professional, Datadog Certified Professional, Github Copilot, Terraform Associate, Splunk certifications, Akamai Professional, Gremlin Certified Chaos Engineer and Google GenAI credentials etc.,
A frequent conference speaker, Anand has presented at multiple Grainger technology conferences on observability, bot management, and security engineering.
Manan Chawda & Nikunj Doshi
Experian
Paying for the Worst Day: Sizing Standby from RTO
Abstract
We keep spare infrastructure running for the worst day of the year. It consumes resources every other day. We wanted to know how much standby capacity we actually need to buy the recovery objective we promise.
We built the same Kubernetes service with three disaster recovery strategies: pilot light, warm standby, and hot standby. Then we failed the primary region and measured what happened.
The tradeoff was visible on both sides. Keeping more capacity ready reduced recovery time, but increased energy use while waiting for an incident. Keeping less capacity idle saved energy, but made recovery depend on provisioning, autoscaling, cache warming, and downstream capacity exactly when the system was already under stress.
We'll compare each strategy using recovery time, p99 latency, failed requests, error budget burn, and workload energy measured during both idle and recovery periods. We will also look at where scaling from a smaller standby stops being fast enough to satisfy the RTO.
We'll finish with a practical method for sizing standby capacity from an RTO instead of assuming that more redundancy is always better.
Bio
Manan Chawda is a software engineer at Experian focused on distributed systems, cloud-native platforms, and production reliability. He has built backend and data infrastructure across AdTech and FinTech and holds an M.S. in Computer Science from New York University. Nikunj Doshi makes Solutions simpler, secured, reliable and accessible. Nikunj Doshi is an accomplished IT Architect with over 10+ years of professional experience and more than a decade of deep specialization in Open Source technologies, Cloud Architecture, Artificial Intelligence, and Cybersecurity. His career reflects a rare combination of technical excellence, research contribution, global project leadership, and sustained community impact.
Shaalmali Raychaudhury
Sr Site Reliability Engineer
Beyond the Fix — Why Problem and Change Management Decide If Incidents Repeat
Abstract
"The fix that closes an incident is rarely the fix that prevents the next one." Finding the root cause feels like the finish line, but it's really the starting point for the work that decides whether an incident becomes a one-time story or a recurring pattern. Problem management and Change management are where that decision actually gets made — yet they are often treated as paperwork, instead of the discipline that turns a lucky catch into a durable fix.
In this talk I will walk through:
How a single root cause can quietly hide an entire class of risk across a codebase.
Why the obvious fix is sometimes the shallow one.
How the guardrails built into change management — pausing risky rollouts, tightening review for a category of change, automating remediation — determine whether a fix actually sticks.
I will share a simple framework for telling apart "fixed the symptom", "fixed the incident" and "fixed the system" plus the follow-up questions that stop a team from closing a problem record too early.
Attendees will leave with a sharper way to evaluate their own post-incident fixes, and a short list of questions to ask before marking a problem "resolved".
Bio
Shaalmali Raychaudhury is a Sr Site reliability engineer with 13+ years of experience, focused on the part of the incident lifecycle where lasting reliability is built: problem management and change management. She previously spoke on hidden downstream failures, the quiet ways one system's degradation surfaces as someone else's incident. Her current focus is on what separates a fix that closes a ticket from one that closes the risk, and how teams build guardrails that outlast the person who wrote them.
Afshin Rostamizadeh
Datadog
KeynoteScout: Training a Specialized Model for SRE Investigations
Abstract
An AI SRE is most useful when it can act as a first responder on every alert, forming hypotheses, querying telemetry, and weighing evidence before an engineer steps in. This makes investigation quality per unit of inference cost a central design goal.
In this talk, we present Scout 1.0, Datadog’s first AI model built specifically to investigate production issues. We describe how we post-trained Scout using reinforcement learning in investigation simulations built from historical incidents, along with a few lessons learned along the way.
To measure investigation quality, we evaluate Scout on AlertEval, our internal benchmark that pairs real alerts and archived telemetry with root-cause explanations written by engineers. We find that Scout pushes out the cost/quality Pareto frontier: its medium-effort mode matches leading general-purpose models at 3–8× lower estimated inference cost, while its highest-effort mode reaches state-of-the-art accuracy. These gains also carry over to the third-party SREGym benchmark.
Together, these results suggest that targeted post-training can make domain-specialized models a cost-effective alternative to general-purpose frontier models.
Bio
Afshin Rostamizadeh is a Staff Research Scientist at Datadog, where he leads a team working on AI for observability, including Scout, a specialized model for SRE investigations. Before joining Datadog, he spent many years as a research scientist at Google Research in New York, working on active learning, distillation, kernel methods, and learning theory. He is a co-author of the MIT Press textbook Foundations of Machine Learning. Afshin earned his BS in Electrical Engineering and Computer Science from UC Berkeley and his PhD in Computer Science from NYU’s Courant Institute, then completed a postdoc at UC Berkeley.