Cyberpediaby Brain

Automation bias: why we stop checking an AI that is often right

Why people end up accepting, without rereading, the answers of an AI that rarely gets things wrong, and which simple habits keep the checking going.

On this page
  1. 1In 30 seconds
  2. 2What it is
  3. 3Straight answers
  4. 4How it works
  5. 5What you do
  6. 6Pitfalls
  7. 7Sources
  8. 8Cite this article

In 30 seconds

Where it happens
In any work where an AI proposes a text, a figure, a sorting or a recommendation that you then approve.
What happens
People can follow a wrong piece of machine advice even though they first had the right answer[1].
The reflex
Before approving, reread part of the answer, look for its source and ask yourself what happens if it is wrong.
If it is already done
Reread now the result you reused without checking it, especially if it commits you to a client or an official figure, and report any error to the person who received it.

What it is

An AI assistant that gives good answers most of the time gradually changes the way people read it. Researchers call this drift automation bias: the tendency to follow a machine's suggestions without examining them.

A pathology study illustrates the risk: AI improves overall performance there, and yet it causes initially correct assessments to be reversed[1].

Straight answers

Does this really concern skilled people?

Yes. In an online experiment published in 2024, 28 pathology experts trained in the job had to estimate percentages of tumour cells. The AI improved their overall performance, but in 7% of cases an initially correct assessment was reversed by wrong advice from the machine[1].

Does time pressure make the problem worse?

In that same study, time pressure did not make the bias appear more often. It does seem to have made it more severe, though: participants relied more on the system's negative consultations, and their performance dropped afterwards[1].

Is a warning from elsewhere enough to make us doubt?

Researchers studied this question in three experiments run in the Netherlands and posted online in 2021. They look at excessive trust in algorithmic advice, even when other sources send warning signals, and at the selective adoption of that advice when it matches stereotypes[2].

Does the level of AI knowledge change anything?

Yes. A preregistered experiment from 2023, run on 9,000 adults in 9 countries, tested an identification task. People who know the least about AI are slightly more inclined to distrust it. Automation bias then appears at still-low levels of knowledge, and levels off when knowledge of AI becomes very high[3]. The difficulty of the task and the competence attributed to the tool also play a part[3].

Who is most exposed?

It depends on the attitude toward AI. In an experiment published in 2025 with 2,784 participants, those who distrusted AI were better at spotting errors. Those who were favourable to it showed excessive reliance on its suggestions[4].

Does a human + AI team do better than the AI alone?

Not necessarily. The authors of a study published in 2025 point out that human + AI teams have been found to perform worse than the AI alone, with automation bias given as the reason: humans follow the AI's recommendations even when they are wrong[5].

What does the AI Act say?

For high-risk AI systems, Article 14 requires that the people in charge of oversight can remain aware of the "possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)"[6]. They must also be able to decide not to use the system, or to disregard, override or reverse its output[6]. A legal analysis from 2025 considers that the emphasis on providers does not sufficiently address design and context as causes of the bias[7].

How it works

The authors of the pathology study define the bias as uncritically following automated indications[1].

The way the work is organised also matters. In the experiment with 2,784 participants, requiring a correction for every flagged error reduced engagement and increased the acceptance of wrong suggestions[4]. The authors conclude that success also depends on who reviews the AI's outputs and how the review is organised[4].

What you do

These habits draw on Microsoft recommendations aimed at teams building applications on Azure OpenAI[8], together with the findings of the studies above.

  • Reread part of the answer before approving it, and edit it instead of accepting it as it is. Microsoft recommends that teams design the tool so that users review and edit outputs before accepting them[8].
  • Look for the source. When a tool cites its references, you can see where the generated content comes from[8]. If the tool gives none, look for the information elsewhere.
  • Check the figures first. Microsoft gives the example of a system that is less accurate with numbers: it recommends flagging them and inviting users to verify them or consult external sources[8].
  • Test code before accepting it. For code suggestions, Microsoft reminds the developer to review and test them[8].
  • Ask yourself what happens if the answer is wrong. Microsoft notes that the risk linked to inaccuracies is higher for a summary of doctors' notes than for a summary of online articles[8].
  • Keep responsibility for the final content. Microsoft recommends reminding users that they are accountable for the content they approve[8].

Pitfalls

  • Believing the tool makes checking unnecessary. In the pathology study, overall performance improved with AI and the bias existed all the same[1].
  • Believing all support features have the same effect. In a study of 108 participants on a diabetes care decision, AI confidence levels, text explanations and performance visualisations improved results. Human feedback and questions asked by the AI encouraged reflection, but often reduced performance by increasing cognitive effort[5].
  • Thinking the law settles the matter. The AI Act requires that high-risk systems be provided so that the people in charge of oversight are able to remain aware of the bias[6], but the legal analysis cited above questions whether it is advisable to regulate the risk directly rather than only requiring awareness[7].

Sources

  1. arXiv, “Automation Bias in AI-Assisted Medical Decision-Making under Time Pressure in Computational Pathology”, 1 November 2024. https://arxiv.org/abs/2411.00998 (accessed 14 November 2026)
  2. arXiv, “Human-AI Interactions in Public Sector Decision-Making: "Automation Bias" and "Selective Adherence" to Algorithmic Advice”, 3 March 2021. https://arxiv.org/abs/2103.02381 (accessed 14 November 2026)
  3. arXiv, “Bending the Automation Bias Curve: A Study of Human and AI-based Decision Making in National Security Contexts”, 28 June 2023. https://arxiv.org/abs/2306.16507 (accessed 14 November 2026)
  4. arXiv, “Bias in the Loop: How Humans Evaluate AI-Generated Suggestions”, 10 September 2025. https://arxiv.org/abs/2509.08514 (accessed 14 November 2026)
  5. arXiv, “Engaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-Making”, 28 January 2025. https://arxiv.org/abs/2501.16627 (accessed 14 November 2026)
  6. European Commission, “AI Act Service Desk - Article 14: Human oversight”. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14 (accessed 14 November 2026)
  7. arXiv, “Automation Bias in the AI Act: On the Legal Implications of Attempting to De-Bias Human Oversight of AI”, 14 February 2025. https://arxiv.org/abs/2502.10036 (accessed 14 November 2026)
  8. Microsoft, “Overview of responsible AI practices for Azure OpenAI models”. https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/overview (accessed 14 November 2026)