Automation promises speed but speed without judgment is how a routine storage migration corrupts a live production database in minutes. While reducing EBS storage across a 142-server fleet targeting roughly 8TB of savings worth an estimated $8,000–$15,000 one database server had quietly fallen outside the disaster recovery checklist. When the automation ran, that single blind spot let a live rsync corrupt roughly 700GB of production data. This talk pulls back the curtain on the reliability engineering behind the recovery: reading metrics and logs in real time to confirm system health, why keeping the old volume until verification turned near-disaster into a clean rollback, and why the real failure wasn’t the script it was a gap in DR coverage nobody caught until production found it. You’ll leave with a principle every SRE should tattoo on their runbooks: disaster recovery has to cover every system by default, not just the ones you remembered because automation is only ever as safe as the recovery plan standing behind it.
Gabriel Okiri is a Senior Platform, DevOps & Cloud Security Engineer with 5+ years of experience spanning financial services, logistics, and AI. My work centers on infrastructure automation, cloud reliability, and recovering production systems when things go sideways because they always eventually do. I have also built AI agents and MCP (Model Context Protocol) integrations that help DevOps teams offload repetitive troubleshooting so they can focus on the judgment calls that actually matter.
