METR tests expose gaps in AI research judgment
AI evaluation nonprofit METR has expanded its time horizon benchmark to include longer and more judgment-heavy tasks. Its findings suggest that capabilities on readily verifiable work continue to improve exponentially, while models struggle with assignments requiring subjective assessment. The original benchmark found that every four months, the length of tasks frontier models could successfully complete doubled, with difficulty measured by human completion time.
In a task asking models to design evaluations for METR’s research, none produced results employees considered useful. Models also fell below human performance when scoring philosophical arguments, though METR employees disagree about that task’s relevance to automating AI research and development. Some newer systems, including Opus 5.5, produced evaluation designs rated “borderline above the bar,” suggesting improvement without reaching practical usefulness.
The findings leave unresolved whether advances in coding and other automatically checkable tasks will transfer to open-ended research. Choosing promising questions and experiments could remain a bottleneck. OpenAI has also reported weaker performance on real-world employee assignments than on the easier, more verifiable tasks in METR’s original suite.
Measurement limitations complicate forecasts. Most human completion times in the updated benchmark are estimates because direct testing would be expensive and slow. Judgment-heavy tasks remain scarce, and philosophical argument scores largely reflect a single evaluator’s preferences. Researchers welcome broader testing but caution that the evidence cannot yet establish reliable trends for automated research.