SREday is a worldwide series of community events for engineers who build, ship and run modern software systems. Across cities around the world, we bring together people working in reliability, cloud, DevOps, observability and production engineering to share real-world experience, connect with their local community and explore how these disciplines are evolving in the age of AI.
The next evolution of incident response isn’t faster alerts, it’s autonomous resolution. Join ilert CEO Birol Yildiz as he shows how AI SRE agents now diagnose and remediate outages without waking anyone up. Learn how these systems combine observability data, deployment context, and code intelligence to restore services in minutes and hand over clean incident reports instead of 3 a.m. pages.... Read more
This is a tongue in cheek talk about the pitfalls of letting AI Agents interact with highly critical systems and sensitive data. With new capabilities also come new vulnerabilities. With help from OWASP LLM Top 10 we’ll walk through three real-world failure scenarios: Agents making catastrophic but 'innocent' mistakes Customer Data leaking into AI Lab training data Prompt injections enabling deliberate exfiltration... Read more
We, tech people, have internalized the concept of reliability so deeply that we don't need an explanation for why it's bad to have services failing in production. It doesn't matter what your software is doing — whether it controls train schedules, allows people to make money transfers, or serves funny pictures online. We have invented an entire language to talk about reliability. We say things like, "This service runs with three nines availability," "Mean-Time-To-Repair for the website is 25 minutes," or "This month, we consumed 80% of our error budget." And yet, it's a common struggle to convince business stakeholders to prioritize technical improvements. In the past seven years, I've been responsible for products and services used by millions of people. I have been on call for years and have spent many nights resolving production incidents. From that experience, I learned how to make reliability a shared priority and prove, with data, the value of technical improvements. In this session, I'll share techniques you can use to improve the reliability of your services, convince business stakeholders of the importance of reliability, and drive positive change in your organization.... Read more
Modern observability is built on a flawed foundation: three siloed pillars - logs, metrics, and traces - each powered by different engines with separate query models, storage formats, and operational costs. Users are forced to manually correlate across systems, accept duplication, or pay high SaaS bills. But what if observability is just a data problem? One that needs a general-purpose solution instead of purpose-built compromises? This talk argues that true observability requires fast, high-cardinality queries over unsampled data at scale and at low cost. Traditional search and metrics engines were not designed for this, but column-oriented databases are. We introduce ClickStack, a fully open source, OpenTelemetry-native observability stack built around ClickHouse. It provides fast, flexible querying and efficient storage, enabling real-time visibility without compromise.... Read more
Modern SRE teams drown in alerts that show up after damage is done. This talk is a practical blueprint for going proactive: predicting incidents before they page you, and triggering safe, reversible auto-remediation.... Read more
Do you still think that traces are those debug level logs to enable when something goes wrong? Do you think that this modern observability thingy is just another buzzword for other languages like javascript or go? Do you believe that "std::cout" is the best way to debug everything? Well, I do agree up to some point. But it does not scale that easily. For modern, distributed systems, you will need to adopt modern observability. The path seems too complicated. But you are not alone! In this talk we'll demystify how to simply create metrics and distributed traces from a bunch of logs by using the OpenTelemetry standard and libraries. Metrics included too! After describing the basics, Jose will also share an example he built to monitor many transactions in production without impacting performance. After this talk, you will be able to talk with higher confidence about OpenTelemetry, Jaeger, distributed tracing, Prometheus or VictoriaMetrics.... Read more
The Terraform plugin cache, unfortunately, does not support concurrent Terraform inits. This is a massive efficiency and performance loss for those who use Terraform at a big enough scale, since we're left with the following choices: - (A) Disable caching, and download gigabytes of providers off the Terraform Registry on every single init. - (B) Serialize all terraform init runs so they can safely share the cache, reducing the Terraform pipeline's throughput (and infuriating developers, been there, done that). We didn't like our options here, so we got creative. One day, everything clicked: OverlayFS! OverlayFS is a Linux filesystem which combines the contents of multiple read-only directories and one writable layer on top, into a single volume, effectively implementing a sort of read-through cache within your filesystem. If you've ever used a live linux distribution, this is what those use. Containers use OverlayFS. CoreOS used OverlayFS to provide ephemeral /etc directories. You rarely see such a low-level tool this high up the stack, so what is doing here? We tweaked our workflow so that our plugin cache is mounted once per init using OverlayFS, which makes the centralized cache read-only (and thus, concurrent-safe) and gives Terraform a writable overlay where it can write whatever providers were missing. We then added a final, non-blocking step to feed back those new providers to the central cache (in a serial fashion using filesystem locking), effectively implementing a write-back cache. After testing it on the lab, we promoted it to our production pipeline. After a couple rounds of plans to fill up the cache, we saw stunning results. Plan times dropped by -61%. Our concurrent plan capacity 10x'd because we were no longer saturating both the network (pulling) and the disk (writing providers). And after all, we didn't add much complexity to the setup anyway. We're using long-standing, reliable, kernel tech (overlayfs and flock) to address a shortcoming of a higher level tool like Terraform.... Read more
15:00
Wrap up
Scan each other's QR codes & head to a nearby pub!
Agents in Production: The Best Ways to Shoot Yourself in the Foot
Abstract
This is a tongue in cheek talk about the pitfalls of letting AI Agents interact with highly critical systems and sensitive data. With new capabilities also come new vulnerabilities.
With help from OWASP LLM Top 10 we’ll walk through three real-world failure scenarios:
Agents making catastrophic but 'innocent' mistakes
Jannes Wolf is a Senior Software Engineer at MaibornWolff GmbH in Hamburg, where he builds modern, scalable software solutions. Over the past several years, he’s advanced from junior to senior roles, gaining deep expertise in full-stack and frontend development. Before joining MaibornWolff, Jannes honed his craft at bilettix GmbH and MEN@NET GmbH, contributing to various web and application projects. His journey reflects a steady focus on clean code, user-centered design, and continuous learning within dynamic engineering environments.
Dominik Rehbock is the Co-Founder of Hyground, an AI-powered incident resolution platform for DevOps and SRE teams. With a strong background in strategy, technology, and entrepreneurship, Dominik previously led strategic initiatives as CEO of Axiom-Insights GmbH and Strategy Director at Innolith, and served as a Management Consultant at Bain & Company. His career spans industries from energy storage to biotech and sustainable agriculture, united by a consistent focus on innovation, scalability, and operational excellence. At Hyground, he’s driving the future of intelligent automation in IT reliability and infrastructure management.
Maxim Schepelin
Booking.com
How to set SLOs, drive improvements, and make friends with business stakeholders
Abstract
We, tech people, have internalized the concept of reliability so deeply that we don't need an explanation for why it's bad to have services failing in production. It doesn't matter what your software is doing — whether it controls train schedules, allows people to make money transfers, or serves funny pictures online.
We have invented an entire language to talk about reliability. We say things like, "This service runs with three nines availability," "Mean-Time-To-Repair for the website is 25 minutes," or "This month, we consumed 80% of our error budget."
And yet, it's a common struggle to convince business stakeholders to prioritize technical improvements.
In the past seven years, I've been responsible for products and services used by millions of people. I have been on call for years and have spent many nights resolving production incidents. From that experience, I learned how to make reliability a shared priority and prove, with data, the value of technical improvements.
In this session, I'll share techniques you can use to improve the reliability of your services, convince business stakeholders of the importance of reliability, and drive positive change in your organization.
Bio
Maxim Schepelin has been an engineering leader in various industries including game dev, e-commerce, and travel building teams capable of launching products from idea to world-scale. With over a decade of experience leading engineering teams, Maxim has mastered the art and science of management, as well as how to exit both Vim and Emacs. Maxim continues exploring ways to build effective engineering teams that deliver value at a sustainable pace.
Vlad Seliverstov
ClickHouse
How We Built ClickStack - an open source, open telemetry native Observability stack
Abstract
Modern observability is built on a flawed foundation: three siloed pillars - logs, metrics, and traces - each powered by different engines with separate query models, storage formats, and operational costs. Users are forced to manually correlate across systems, accept duplication, or pay high SaaS bills. But what if observability is just a data problem? One that needs a general-purpose solution instead of purpose-built compromises? This talk argues that true observability requires fast, high-cardinality queries over unsampled data at scale and at low cost. Traditional search and metrics engines were not designed for this, but column-oriented databases are. We introduce ClickStack, a fully open source, OpenTelemetry-native observability stack built around ClickHouse. It provides fast, flexible querying and efficient storage, enabling real-time visibility without compromise.
Bio
Vlad Seliverstov leads the internal observability team at ClickHouse, overseeing the monitoring of ClickHouse Cloud, which handles over 100 petabytes of data and 500 trillion events. With more than a decade of SRE experience at Datadog, Dropbox, and Facebook, Vlad now focuses on building and scaling ClickHouse’s in-house observability platform.
Rajat Gupta
Paymenttools
Predict, Don’t Page: AI for Incident Forecasting and Safe Auto-Remediation
Abstract
Modern SRE teams drown in alerts that show up after damage is done. This talk is a practical blueprint for going proactive: predicting incidents before they page you, and triggering safe, reversible auto-remediation.
Bio
I’m a Senior Engineering Manager at Paymenttools in Berlin, leading platform teams across SRE and Security. I focus on reliability, observability, Kubernetes, and policy as code, and I drive green-field work from idea to production. Lately, I’ve been applying GenAI to predict incidents and enable safe, reversible auto-remediation. I like clear processes, data-backed decisions, and practical solutions, and I write for peers to share what works and what doesn’t.
Jose Gomez-Selles
VictoriaMetrics
Modern Observability with OpenTelemetry in C++
Abstract
Do you still think that traces are those debug level logs to enable when something goes wrong? Do you think that this modern observability thingy is just another buzzword for other languages like javascript or go? Do you believe that "std::cout" is the best way to debug everything? Well, I do agree up to some point. But it does not scale that easily.
For modern, distributed systems, you will need to adopt modern observability. The path seems too complicated. But you are not alone! In this talk we'll demystify how to simply create metrics and distributed traces from a bunch of logs by using the OpenTelemetry standard and libraries. Metrics included too!
After describing the basics, Jose will also share an example he built to monitor many transactions in production without impacting performance. After this talk, you will be able to talk with higher confidence about OpenTelemetry, Jaeger, distributed tracing, Prometheus or VictoriaMetrics.
Bio
Jose is the Product Lead for VictoriaMetrics Cloud. With a focus on Observability and Sustainability, his work has been deeply related to the OpenTelemetry project, distributed tracing and power monitoring projects.
His expertise has been built from previous gigs as a Software Architect, Tech Lead and Product Owner in the Telecommunications industry, all the way from the software programming trenches where agile ways of working where a sound CI, testing and observability best practices have presented themselves as the main principles that drive every successful project.
With a scientific background in Physics and a PhD in Computational Materials Engineering, curiosity, openness and a pragmatic view are always expected. Beyond the boardroom, he is a C++ enthusiast and a creative force: contributing symphonic touches as a keyboardist in metal bands, when he is not playing video games or lowering lap times at the simracing cockpit. He also loves to spend time teaching Physics to Vet students in the Complutense University of Madrid, as an Associate Professor.
Ricard Bejarano
Cisco
Speeding up Terraform caching with OverlayFS
Abstract
The Terraform plugin cache, unfortunately, does not support concurrent Terraform inits. This is a massive efficiency and performance loss for those who use Terraform at a big enough scale, since we're left with the following choices: - (A) Disable caching, and download gigabytes of providers off the Terraform Registry on every single init. - (B) Serialize all terraform init runs so they can safely share the cache, reducing the Terraform pipeline's throughput (and infuriating developers, been there, done that).
We didn't like our options here, so we got creative.
One day, everything clicked: OverlayFS!
OverlayFS is a Linux filesystem which combines the contents of multiple read-only directories and one writable layer on top, into a single volume, effectively implementing a sort of read-through cache within your filesystem. If you've ever used a live linux distribution, this is what those use. Containers use OverlayFS. CoreOS used OverlayFS to provide ephemeral /etc directories.
You rarely see such a low-level tool this high up the stack, so what is doing here?
We tweaked our workflow so that our plugin cache is mounted once per init using OverlayFS, which makes the centralized cache read-only (and thus, concurrent-safe) and gives Terraform a writable overlay where it can write whatever providers were missing. We then added a final, non-blocking step to feed back those new providers to the central cache (in a serial fashion using filesystem locking), effectively implementing a write-back cache.
After testing it on the lab, we promoted it to our production pipeline. After a couple rounds of plans to fill up the cache, we saw stunning results. Plan times dropped by -61%. Our concurrent plan capacity 10x'd because we were no longer saturating both the network (pulling) and the disk (writing providers). And after all, we didn't add much complexity to the setup anyway. We're using long-standing, reliable, kernel tech (overlayfs and flock) to address a shortcoming of a higher level tool like Terraform.
Bio
Ricard is a Lead Site Reliability Engineer at Cisco ThousandEyes' SRE team. Ricard is responsible for the Terraform pipeline of 600+ developers, managing over 110k cloud resources.
Birol Yildiz
ilert
KeynoteWhen Incidents Fix Themselves: AI SRE in action
Abstract
The next evolution of incident response isn’t faster alerts, it’s autonomous resolution. Join ilert CEO Birol Yildiz as he shows how AI SRE agents now diagnose and remediate outages without waking anyone up. Learn how these systems combine observability data, deployment context, and code intelligence to restore services in minutes and hand over clean incident reports instead of 3 a.m. pages.
Bio
Birol Yildiz is the Co-founder and CEO of ilert, adeptly steering the company with a rare combination of technical and product expertise. His prior experience includes a significant role as Chief Product Owner for Big Data products at REWE Digital. With a strong foundation in computer science, Birol bridges the gap between developer and product strategist, constantly striving to innovate and provide customer-centric solutions at ilert.
