ServiceNow releases EVA framework for voice agent evaluation

ServiceNow has released EVA, an open-source framework for evaluating conversational voice agents across full multi-turn spoken interactions. The framework measures two dimensions: EVA-A for whether an agent completes a task correctly and faithfully, and EVA-X for whether the exchange feels natural, concise, and appropriate for voice dialogue.

EVA uses a bot-to-bot architecture in which conversational AIs speak over a live WebSocket connection, avoiding human listeners and text replays. The system supports cascade pipelines that combine STT, LLM, and TTS components, as well as speech-to-speech and audio-native models. Its metrics engine scores conversations using recordings, transcripts, and tool call logs.

The release includes 213 enteprise scenarios across 3 domains, covering airline customer service, healthcare HR service delivery, and enterprise IT service management. It also reports results for 12 cascade and audio-native systems, along with perturbation tools for background noise, user accents, simulated connection degradation, and combined audio conditions.

EVA includes installation guidance, CLI workflows, Docker support, Streamlit tools for configuration and analysis, validation checks, and documentation for metrics, datasets, and experiment setup. The project requires Python 3.11-3.13 when using the recommended uv-based setup.