What happens when your core logging framework has a critical exploit, rollback is a security risk, and you can’t test the new stack until it hits production? You burn the ships. In this talk, I share the high-stakes engineering story of a complete, one-way logging migration across thousands of live clusters. Forced by an active vulnerability to bypass traditional safety nets we had to rely on napkin-math resource estimations, synthetic traffic testing, table top exercises of what could go wrong, and a terrifyingly literal "test in production" rollout. I will share the exact strategies we used to pull off this zero-rollback cutover without plunging our systems into telemetry blindness. You’ll leave with concrete patterns for calculating production risk under extreme constraints, validating data pipelines on the fly, and failing forward.
Hilliary Lipsig is an Engineering Leader, Author, and Senior Principal Site Reliability Engineer at Red Hat, where she focuses on Azure Red Hat OpenShift. Known for leading follow-the-sun SRE teams, she is also the host of the popular YouTube livestream GitOps Guide to the Galaxy. Outside of her work in reliability engineering, Hilliary serves as Secretary on the Board of Directors for the Gender Education Network and is an active voice in the open-source and Kubernetes communities.
