Most of the quality tooling we have added for AI-written code looks at the diff. Code review, static analysis, and pre-merge checks all ask whether the change looks correct. None of them answer the question a reliability engineer actually cares about, which is whether the running application still works.
When a person wrote the code, that mattered less. They knew what they had changed and what else it might touch. When an agent writes the code, nobody knows. The pull request looks clean, the checks pass, and the first honest signal arrives after the deploy, often when a user hits it.
This talk makes the case for moving verification to the other side of the deploy event, and shows what that looks like in practice. A testing agent opens the live build in a real browser, exercises it the way a user would, and hands the failure back to the coding agent with enough context to act on it: the failing step, a screenshot, a DOM snapshot, and a root-cause hypothesis. The repair happens before a human is involved.
Yunhao will close with what TestSprite measured across CoderCup, a public benchmark where frontier coding agents built the same application under identical conditions, including a pattern the team calls feature retention: working features quietly disappearing as later edits land in a codebase whose author does not remember what it built last week.
Yunhao Jiao is a Yale University graduate with a master's degree and spent nearly five years at AWS as a Senior Software Engineer, where he helped build the Contract Tests framework for AWS CloudFormation, ensuring resource types operated as expected throughout their lifecycle. He has been active in NLP research since 2015 and published his first AI paper as lead author in 2017. Yunhao founded TestSprite and participates in the Techstars Miami and YC China accelerator programs.
