Newer language models shed many familiar flaws
Many problematic behaviours in large language models fade as companies release more capable systems, according to an analysis that has not been peer reviewed. Researchers found that the latest models rarely, if ever, cite nonexistent sources. Nearly half of the behaviours tested had largely disappeared from newer models, with those effects lasting an average of about 16 months after their initial publication.
The researchers identified 50 widely cited behaviours and retested the original prompts on 20 LLMs. Improvements included more reliable counting of letters in words and less overthinking of simple maths problems. Progress was uneven: disproportionately describing Black women as resilient persisted for longer than three years. Meanwhile, models became more inclined to praise users’ preferences, favour their own writing over work by people or other models, and struggle with generating truly random numbers.
Better performance in newer systems does not mean the consequences of earlier weaknesses have disappeared. Columbia University health-care AI researcher Maxim Topaz found that hallucinated citations in academic papers have risen drastically since 2023, probably because more researchers are using LLMs and often rely on older models. The findings suggest that assessments of AI capabilities need updating while real-world risks remain dependent on which systems people use.