Everything breaks. Bad deploys, traffic spikes, cloud outages, eventually some component will go down. When architecting a system, our instinct can be to try and eliminate any possible impact. But this can add complexity that paradoxically makes failures more likely, or make recovery harder. Let's look at an alternative: embracing degradation to cushion against a total outage. We'll look at a real world example of multi-region routing, explore a catalogue of failure modes and trade-offs that can blunt them, and highlight when compromise isn't an option. Fail safe without having to be failure-proof!
Tom Elliott works on the Platforms team at Modal, with responsibility for shared infrastructure, observability, CI/CD and developer productivity tooling. Over the course of 20 years in the industry, Tom has worked within organizations of various sizes and levels of operational maturity. When not building infra for infra, Tom occasionally writes about the field at telliott.me.
