
An AI SRE is most useful when it can act as a first responder on every alert, forming hypotheses, querying telemetry, and weighing evidence before an engineer steps in. This makes investigation quality per unit of inference cost a central design goal.
In this talk, we present Scout 1.0, Datadog’s first AI model built specifically to investigate production issues. We describe how we post-trained Scout using reinforcement learning in investigation simulations built from historical incidents, along with a few lessons learned along the way.
To measure investigation quality, we evaluate Scout on AlertEval, our internal benchmark that pairs real alerts and archived telemetry with root-cause explanations written by engineers. We find that Scout pushes out the cost/quality Pareto frontier: its medium-effort mode matches leading general-purpose models at 3–8× lower estimated inference cost, while its highest-effort mode reaches state-of-the-art accuracy. These gains also carry over to the third-party SREGym benchmark.
Together, these results suggest that targeted post-training can make domain-specialized models a cost-effective alternative to general-purpose frontier models.
Afshin Rostamizadeh is a Staff Research Scientist at Datadog, where he leads a team working on AI for observability, including Scout, a specialized model for SRE investigations. Before joining Datadog, he spent many years as a research scientist at Google Research in New York, working on active learning, distillation, kernel methods, and learning theory. He is a co-author of the MIT Press textbook Foundations of Machine Learning. Afshin earned his BS in Electrical Engineering and Computer Science from UC Berkeley and his PhD in Computer Science from NYU’s Courant Institute, then completed a postdoc at UC Berkeley.