Atropos Health measures clinical summaries with Arize AX

Atropos Health is working to close the evidence gap in healthcare by helping physicians access observational studies on demand. The company conducts hundreds of studies each month, and summarizing each study can take about 10 minutes per study. To reduce that workload, Atropos Health built Auto Summary, an LLM-based tool that generates summaries for physicians to review and edit.

As Auto Summary moved through production use, the team needed a clearer view of how it performed at scale. Individual physician edits were not enough to understand quality across many studies, so Atropos Health began building a measurement framework in Arize AX to monitor the tool more systematically.

The team started with an iterative approach rather than waiting for a perfect evaluation framework. Its next step is to narrow monitoring to the one or two metrics that matter most, explore LLM-as-a-judge for a more accurate read on performance, and build broader internal support for the measurement process.