Bad AI advice can inflate confidence

New research from Valerio Capraro of the University of Milan-Bicocca and coauthors finds that incorrect AI guidance can make people more confident while reducing their willingness to say “I don’t know.” In experiments built around obscure movie-detail questions that the tested model answered incorrectly almost every time, AI access reduced accuracy while making users more willing to commit to answers. The paper has not yet been peer reviewed.

Participants without AI withheld an answer on 36% of the questions, while those who could ask the model declined only 6% of the time. In a replication, the rates were 44% without AI and 3% with AI. Across conditions, people working alone answered correctly 27.5% of the time, compared with 9.2% for those with AI access. Capraro said accuracy declined by a factor of three while confidence rose by a factor of 2.5 when incorrect AI advice was available.

The researchers suggest AI may lower users’ metacognitive threshold, making them feel they know enough to answer even without adequate evidence. The effect persisted when advice appeared automatically. Small financial rewards and penalties made participants more cautious and more accurate, but did not remove the gap between AI users and people working alone.

Capraro said systems should augment human judgment rather than replace it, especially as search engines, writing tools and workplace apps add automatic suggestions. He also warned that children with constant access to instant answers may miss the learning value of doubt, which helps people examine evidence and ask how an answer is justified.