As AI systems scale in complexity and impact, observability becomes essential—not just for performance monitoring, but for ensuring quality, safety, and trust. In Azure AI Foundry, evaluators are the backbone of this observability layer, enabling continuous, automated assessments across the AI lifecycle. This session provides a deep technical dive into the evaluator framework in Azure AI Foundry, covering: 🔍 Evaluator Categories and Use Cases RAG Evaluators (Retrieval-Augmented Generation) Retrieval: Measures relevance of retrieved documents. Groundedness / Groundedness Pro: Assesses factual alignment with context. Relevance & Completeness: Evaluates how well the response answers the query and covers all necessary information. Agent Evaluators Intent Resolution: Accuracy in understanding user intent. Task Adherence: Measures how well the agent completes assigned tasks. Tool Call Accuracy: Validates correct tool invocation and usage. General Purpose Evaluators Fluency: Natural language quality and readability. Coherence: Logical consistency and flow. QA: Comprehensive quality checks for question-answering tasks. 📊 Operational Observability Real-time metrics: latency, token usage, error rates. Visualizing evaluator outputs in the Foundry Observability Dashboard. Connecting evaluations to traces for root cause analysis. By the end of this session, attendees will understand how to design, implement, and scale evaluator-driven observability in AI Foundry to build robust, transparent, and production-grade AI systems.
Senior Azure Cloud & AI Consultant at Microsoft
