What is automation bias?
Make room for a second opinion.

Automation bias is a decision pattern, not a model defect.

Automation bias describes errors that happen when people give an automated recommendation too much weight. They may follow incorrect advice, or miss an important signal because the system did not call attention to it. A classic review of human–automation research ↗ describes overreliance as misuse when it weakens monitoring or biases a decision.

The idea predates generative AI. Earlier work studied alerts, decision aids, and control systems; it does not establish how often the same pattern occurs in every chatbot or AI product. For generative systems, the practical question is still clear: can the person see enough evidence, time, and authority to challenge the answer?

That differs from an AI system producing a wrong answer ↗. A model can be wrong without anyone relying on it. Automation bias describes the human–system interaction that lets a wrong or incomplete suggestion pass unchecked.

Putting a person in the loop does not guarantee a meaningful review.

A final “approve” button can become a formality when a person is rushed, sees no contrary evidence, or expects the system to be right. A confident explanation may look reassuring, but it does not prove that the recommendation is correct or that the person had a fair chance to assess it. For the difference between an explanation and evidence, see our guide to explainable AI ↗.

In a controlled 2021 experiment with 199 participants, Buçinca, Malaya, and Gajos tested a simulated AI recommendation on a meal-planning task. Asking people to decide before seeing the suggestion, delaying it, or making it available on request reduced overreliance on incorrect suggestions compared with simple explanation designs. The interventions did not eliminate the problem, and people rated the more effortful designs less favorably. Read the experiment ↗

This is a useful interface study, not a universal recipe: it used one task and a simulated model. Its strongest lesson is that showing an explanation is not the same as helping someone evaluate a recommendation. For a wider view of when combined teams perform well, read how to evaluate human–AI collaboration ↗.

Make disagreement possible, useful, and proportionate to the stakes.

Meaningful oversight requires more than a human name on the workflow. The reviewer needs relevant information, enough time, the skill to assess the case, and authority to change the outcome. The NIST AI Risk Management Framework ↗ offers a broader structure for governing, mapping, measuring, and managing AI risks across a system’s use.

Measure whether people catch bad advice—not whether they like the interface.

1. Compare the right baselines. Evaluate human-only, AI-only, and human-plus-AI outcomes on the same representative cases.
2. Include deliberate errors and omissions. Test when the AI is right, wrong, uncertain, and silent. Check whether reviewers identify each condition.
3. Record the decision process. Measure acceptance of wrong recommendations, missed signals, justified overrides, review time, and downstream corrections.
4. Check who benefits. Compare results by relevant experience, task familiarity, and other context that could affect people’s ability to review the suggestion.
5. Revisit the real workflow. Test after repeated use and under the time, information, and workload conditions people actually face.

Automation should make work more capable without turning review into theatre. Design the decision around what a person can verify, then test that the check holds when the recommendation is inconvenient to question.