While working with our customers, we are seeing that teams are shipping AI features fast but skipping the instrumentation that tells them what's actually happening in production.
AI applications fail differently from traditional services. Token exhaustion, retrieval quality drift, and provider outages don't show up in HTTP status codes. But how much instrumentation do you actually need, and when does the effort stop paying off?
To distill what we've learned from instrumenting customer AI applications, we built a reference RAG app and applied four levels of instrumentation: zero-code auto instrumentation, manual OTel spans, AI-native telemetry libraries, and an external AI gateway. Each level adds visibility but also adds effort, from zero lines of code to a production grade telemetry pipeline setup.
This talk presents a practical maturity model. For each level, we show: what you can now see, what failure modes you can catch, what's still invisible, and what it costs in engineering time. We use real traces and dashboards from a running system to ground the comparison.
Along the way, we bust some practical myths: why standard percentile calculations silently produce garbage for RAG similarity scores, and why "cheaper" models can end up costlier than premium ones in specific situations.
Whether you're deciding where to start or evaluating whether the next level of investment is worth it, you'll leave with a clear framework mapping effort to observability payoff, and a working open-source repo to validate it yourself.
Saurabh Hirani is a Principal SRE at One2N with 20+ years of experience in infrastructure, operations, and reliability engineering. He has worked with startups and enterprises to build production-grade monitoring systems with a focus on automation and minimalism.
Previously, he led customer success teams for observability products and architected resilient telemetry ingestion pipelines for JioHotstar during major IPL events. Currently, he works with customers to instrument and observe their AI applications in production.
