Pre-production environments are often left running 24/7, leading to significant cloud waste—but safely automating their shutdown is more complex than it seems. Naive scheduling approaches can easily break dependencies, introduce inconsistent states, or make recovery unreliable. In this talk, I’ll present a real-world implementation of a scheduled, event-driven orchestration system on AWS that safely scales down and restores an entire pre-production environment. The solution uses EventBridge, Step Functions, and Lambda to coordinate state transitions across EC2, RDS, EBS, and application services. A key focus of this system is safe, state-aware execution. Each operation validates the current resource state before acting, ensuring that actions are effectively idempotent and safe to re-run. The workflow enforces dependency-aware sequencing and uses check/wait patterns to avoid acting on transitional or stale states, reducing the risk of unsafe shutdowns and restarts. Rather than relying on blind automation, this approach treats infrastructure changes as explicit, observable workflows, where failures are surfaced clearly and recovery paths are controlled. I’ll also share lessons learned around handling partial executions, the limitations of manual retry strategies, and why idempotency must be considered beyond individual steps. This session will provide practical insights into building reliable, cost-efficient infrastructure automation, along with design patterns that can be applied to any AWS-based environment.
I am a Site Reliability Engineer with experience in AWS and cloud-native environments, focusing on platform engineering and infrastructure automation. I have worked extensively with containerized systems using Docker and Kubernetes, along with building and operating CI/CD pipelines, and maintaining infrastructure using Terraform. My work centers on operating and improving existing systems, building reliable infrastructure, enhancing automation workflows, and handling real-world failure scenarios in distributed environments.
