o3 points beyond bigger models

François Chollet created ARC-AGI in 2019 as a benchmark meant to challenge the industry’s reliance on scaling model size. The test used simple colored-grid puzzles that a child could solve, but that required identifying abstract patterns rather than leaning on memorized training examples. For four years, larger models made little progress: GPT-3 scored close to zero, and GPT-4o scored 5%.

OpenAI’s o3 changed the trajectory in December 2024, reaching 87.5% on the same benchmark while belonging to the same model class as the system that had scored 5% the month before. The key shift was longer inference: the older model spent a few hundred tokens deciding on an answer, while o3 spent five and a half billion. The high-compute run was reportedly trained on a large portion of the public training set, so the result was not pure inference, but it still pointed to a different path for progress beyond simply making models bigger.