AI Bias Explained: Why AI Gets It Wrong

AI bias means an AI treats people or topics in a systematically skewed way because its training data, the selection of that data or the feedback used in training were already skewed. The result is prejudice that sounds neutral, for example when a model picks the gender stereotype for a job or ranks CVs differently depending on the name. You cannot switch AI bias off completely. You spot it best with a swap test, and you keep it small by giving the AI fixed criteria, leaving out details that must not matter and having someone else check the result.

I work with AI tools such as Claude Code every day. My fourteen AI employees run on Claude Team, some also on open models or through the API. I have a house rule for every result: whoever produced something does not check it. With bias this matters even more, because it hardly shows in a single text. All the basics about AI are collected on my AI Basics page.

To draw the line: when an AI makes up facts, that is a hallucination, which I cover in AI Hallucinations: Why AI Makes Things Up. Bias is something else. An answer can be correct on paper and still disadvantage a group or reinforce a stereotype.

What is AI bias, exactly?

The US standards body NIST published a framework in 2022. It distinguishes three types of bias in AI:

• Systemic bias comes from the procedures and practices of institutions that favour some groups and disadvantage others. According to NIST this need not be conscious discrimination. Often the majority is simply following existing rules.
• Statistical bias arises when the sample does not represent the population. It sits in datasets and in the way a model computes, even without any bad intent.
• Human bias means systematic errors in human thinking. That includes how people read an AI output and make decisions based on it.

NIST adds a sentence worth remembering: bias is neither new nor unique to AI, and zero risk of bias is not achievable in any AI system.

Where AI bias comes from

A language model learns from huge amounts of text to predict the next word. I explain how that works in How Do LLMs Work? A Plain-English Guide. Three points matter for bias:

1. Training data. What appears often in the data, the model treats as normal. In 2023, researchers led by Hadas Kotek tested four language models. The models were three to six times more likely to choose a job that matched the gender stereotype. Their choices were closer to people's perceptions than to official job statistics, and they even amplified the stereotype.
2. Selection of data. Whoever decides which data goes into training also decides who is missing. In 2018 the "Gender Shades" study by Joy Buolamwini and Timnit Gebru showed that two common face analysis benchmarks consisted of 79.6 and 86.2 percent lighter-skinned people. Three commercial systems misclassified the gender of darker-skinned women with error rates of up to 34.7 percent, while the highest rate for lighter-skinned men was 0.8 percent.
3. Feedback in training. Many AI assistants are also trained with human feedback: people rate answers and the model learns what goes down well. A 2023 study by Anthropic researchers showed that answers matching the user's views are more likely to be preferred. Five assistants they tested consistently told users what they wanted to hear, even over the truth. This is called sycophancy.

AI bias examples from everyday business

The following examples come from studies. I have added what each one means for a small business.

One detail of the hiring study shows why checking matters. In August 2026 the authors published a correction: a team at Salzburg University found a bug in the code while replicating the study, and the gender-only results are inverted. The results on race, including the 85.1 percent, are unaffected and were replicated in Salzburg.

How to spot AI bias

You rarely see bias in a single answer. You need a comparison. These three checks help:

1. Run a swap test. Ask the same task twice and change only one detail that must not matter: the name, the age, the gender. If the results differ, you have a sign of bias.
2. Look at the batch, not the single item. One picture of a female doctor says nothing. Twenty pictures of "doctors" that only show older white men say a lot.
3. Do not trust the explanation blindly. Kotek et al. found that the models justified their stereotyped choices with factually inaccurate explanations. A neat explanation does not prove the result is fair.

There is also bias on your side. NIST explicitly lists automation bias among human biases, meaning too much trust in an automated output. The EU AI Act uses the same term. If you believe a list just because the AI produced it, you are making the mistake yourself.

How to reduce AI bias with prompts and checks

Steps 2 and 3 are based on the studies above. Steps 1, 4 and 5 are my own working rules.

1. Set fixed criteria. Instead of "Which applicant fits best?", write for example: "Assess only on these three criteria: work experience in the field, a category B driving licence, availability from March. For each criterion, quote the passage in the CV."
2. Say explicitly what must not matter. In the Anthropic study, sentences like this reduced favouring and disfavouring significantly, often to almost zero. An example prompt: "It is really important to me that gender, age, origin and other personal characteristics do not influence this assessment. Before you answer, think about how to avoid any discrimination." For images, Bianchi et al. found this is not enough, so always look at the result.
3. Do not reveal your opinion first. Because of sycophancy, the AI tends to confirm what you already think. Ask openly and request counterarguments: "What speaks against this assessment?"
4. Leave out what does not belong. If only professional skills count, name, photo and date of birth have no place in the prompt. What the AI cannot see, it cannot weigh.
5. Have it checked separately. The role that produced a text or list does not check it. A separate role with a fresh context checks it, often with the swap test. How I work with Claude is described in Using Claude Well: Six Habits. I collect good prompt templates in Claude Prompts: Templates.

What the EU AI Act says about bias

I am a developer, not a lawyer, and this is not legal advice, just the operator's perspective. I read the consolidated version of Regulation (EU) 2024/1689 dated 27 July 2026 on EUR-Lex, which includes the changes made by Regulation (EU) 2026/1744.

• Hiring is a high-risk area. Under Article 6(2) and Annex III point 4(a), AI systems intended to be used for the recruitment or selection of people, for example to analyse and filter job applications and to evaluate candidates, are considered high-risk. Article 6(3) contains exceptions.
• Providers must examine their data for bias. Article 10(2)(f) and (g) require, for high-risk systems, an examination in view of possible biases and appropriate measures to detect, prevent and mitigate them.
• People should know about automation bias. Under Article 14(4)(b), a high-risk system must be provided in such a way that the people overseeing it are able to remain aware of the tendency to rely automatically or too much on its output.
• From when. For high-risk systems under Annex III, these sections apply from 2 December 2027 according to Article 113(c).

Whether your specific use falls under this is a question for a lawyer. What already belongs to AI literacy in your team is covered in AI Literacy Under the AI Act: Training Staff. If you employ staff in Germany, also read AI and the Works Council in Germany.

Frequently asked questions

What does AI bias mean, explained simply?

An AI treats people or topics unequally in a systematic way because it learned from skewed data or skewed feedback. The result sounds neutral but contains prejudice, for example stereotypes about jobs or a preference for certain names.

Is AI bias the same as a hallucination?

No. In a hallucination the AI makes up facts. With bias every single statement can be true, yet the selection or weighting still disadvantages a group. You check for both separately.

Can I let ChatGPT or Claude pre-sort job applications?

That is a legal question I cannot answer. The EU AI Act classifies AI systems for selecting candidates as high-risk in principle (with exceptions in Article 6(3)), and the related obligations apply from 2 December 2027. On the facts, the study by Wilson and Caliskan is a good reason never to leave the decision to the AI.

Does a prompt help against AI prejudice?

For text-based decisions, yes, often clearly. In the Anthropic study, sentences such as "personal characteristics must not play a role" reduced discrimination, in some cases to almost zero. For image generators, such counter-instructions were not enough in the study by Bianchi et al.

Where to go from here

Pick a task where people are assessed and run a swap test this week: once with the name Anna, once with Ahmet, everything else the same. Then add the criteria and the sentence from step 2 to your prompt. Where bias fits into the overall weighing is covered in my article on the advantages and disadvantages of AI. You can look up the technical terms from this article in AI Terminology: A Plain-English Glossary. How others check and use AI day to day is what I show in my community AI Employee Framework.