Synthetic traffic offers cheaper evaluation for agent search
Software engineer Doug Turnbull has introduced a method for evaluating AI agent search through simulated user sessions rather than LLM-generated quality scores. Findings published on November 12, 2026, reported a 70% cost reduction compared with LLM-as-a-judge baselines, alongside improved sensitivity to search regressions.
The approach generates realistic query patterns and measures task success through click-through rates, dwell time, query reformulation and abandonment. Simulations cover factual lookups and complex research tasks, allowing teams to compare candidate search configurations using behavioral signals rather than model judgments.
LLM-based evaluators can favor results based on their position and assign inconsistent scores to the same query, complicating regression detection. Their API costs also grow with evaluation volume. In early testing, synthetic traffic identified quality degradations in A/B tests that model judges had rated as neutral or improved, including failures involving ambiguous queries and domain-specific terminology.
Turnbull released an open-source reference implementation with code examples and benchmarking data. The method targets teams building AI agent interfaces and autonomous research systems, providing interpretable metrics intended to connect search evaluation more closely with production behavior and business objectives.