Researchers challenge company-led views of AI use

AI companies regularly publish usage reports for products such as Claude and ChatGPT, but researchers say those accounts rely on data the companies choose to disclose and lack independent corroboration. The AI Observatory, co-led by Stanford researcher Anka Reuel and MIT Media Lab graduate Shayne Longpre, aims to provide a public platform for analyzing real conversations with popular models collected with user consent.

The project found that usage varies by model and over time, with more personal and sensitive behavior than major company reports typically show. Applying Anthropic’s work-focused filtering approach would have excluded 48% of conversations in the Observatory dataset, including higher shares of health, relationships, adult or illicit topics, harassment and hate, and sexual content than in Anthropic’s analysis.

The researchers aggregated 85,633 conversational turns across 24,521 conversations from 5,000 users interacting with 52 different models, including ChatGPT, Gemini, Claude, and Grok, between 2023 and 2025. They found Grok and Gemini were used more often for information retrieval, Anthropic for coding, Gemini for social and roleplay exchanges, and ChatGPT for homework assistance; misinformation tended to concentrate on Grok.

The dataset remains far smaller than company-held data and likely underrepresents sensitive use because it draws on voluntarily shared conversations. Researchers said broader independent access is needed because proprietary chat data leaves policymakers and researchers making consequential decisions from partial company narratives.