Most AI sales agents don’t underperform because of weak models — they underperform because no one is measuring the right things in production. Accuracy benchmarks look fine, yet buyers disengage. The gap? We’re not evaluating conversational quality at the interaction level. This talk is a technical deep dive into building human-like AI sales agents using LLM evals, production telemetry, and observability frameworks like LangSmith (and similar tooling). We’ll show how to move beyond static prompt tuning and into measurable, iterative system optimization. You’ll learn how to: Design multi-turn evals that measure flow, intent alignment, and adaptive verbosity Use pairwise and win-rate evaluations to detect regressions in human-likeness Instrument production systems to track AI detection signals (verbosity spikes, context resets, over-structuring) Leverage tools like LangSmith to trace conversations, label failure modes, and connect evals to real revenue outcomes Close the loop between offline evaluation and live performance using production conversation data We’ll cover concrete metrics such as tokens-per-resolved-intent, clarification efficiency ratio, and state-progress efficiency — and how to operationalize them inside your LLM stack. If you’re building AI agents in high-stakes sales environments, this session provides a practical framework for turning evals and observability into a competitive advantage — and systematically transforming scripted bots into adaptive, revenue-driving systems.
Lior Mechlovich is the CTO and co-founder of salespeak.ai. Before Salespeak, he was the co-founder of amberflo.io and led engineering teams at AWS and Informatica.
