Vals AI plans independent tests of model routers

Vals AI is developing an independent evaluation of model routers, which select which AI model handles each request based on quality, cost, and latency. The evaluation will measure live, end-to-end agent performance across finance, coding, legal, and tax work, with results due on a public leaderboard soon.

Performance differences across models create an opportunity for routing. Vals reports that five different models lead its eight benchmarks, with no model placing in the top three across all of them. A theoretical oracle that selects the perfect model for each task in hindsight would outperform the best single model by 14 points while costing about 40% less. That calculation illustrates potential gains, not demonstrated router performance.

Long-running workflows make those gains harder to capture. A cheaper model can introduce errors that require additional calls, while switching models can sacrifice prompt-cache savings. Evaluations based on isolated public questions can miss these effects and face risks from saturated benchmarks or prior exposure to test data.

Vals plans to consult each router on every model call while keeping prompts, tools, and graders fixed. Products will run with their default model pools and settings, with configurations documented. Task outcomes, cost, cache usage, and completion time will be compared with the best and cheapest individual models in each router’s pool, using the oracle as a reference. The Vals Index includes mostly private, expert-written benchmarks to reduce contamination risk.