Cyberpediaby Brain
ENDEFR

Algorithmic bias and data stereotypes

An AI repeats the stereotypes in its training data while looking neutral. What employees check, and what the AI Act provides.

In 30 seconds

Where it happens
CV screening, scoring, candidate assessment, text or image generation, anywhere AI ranks or describes people.
What happens
The tool reproduces gaps present in its data, and its answer seems neutral.
The reflex
Never let an AI alone decide a matter that affects people, compare the results, report the gaps and keep a human validation.
If it is already done
Tell the person in charge of the tool or the relevant contact, and have a person review the decision.

What it is

An AI tool learns from data produced by people, in organizations and a society that have their own habits and their own inequalities. What it draws from that data can reproduce stereotypes, without anyone intending it. NIST, the American standards body, writes that biases « can lead to harmful impacts regardless of intent »1.

The result arrives in a numerical or ranked form, which looks objective. This page explains where these gaps come from and what an employee does when using AI to sort or score people.

Straight answers

Why does an AI reproduce stereotypes?

Because it learns from data that contain them. NIST distinguishes three categories of bias in AI: systemic, statistical and human2. Systemic biases come from the procedures and practices of institutions that advantage some groups and disadvantage others. This requires no conscious prejudice: it is enough that the majority follows the existing rules and norms2. These biases are found in the datasets used in AI2.

Can data be accurate and still biased?

Yes. The CNIL describes two cases. The training data may be non-representative. They may also be well representative of the « real world » and still reflect a discriminatory character, for example the reproduction of pay gaps between women and men3. NIST points the same way: even when representative, datasets can present entrenched historical and systemic biases2.

Why does the answer look neutral?

NIST notes that biases are widespread in technological processes and can cause harm regardless of intent1. The tool does not show where its answer comes from. The European Commission points out that it is often impossible to know why an AI system made a decision, which makes it hard to know whether a person was unfairly disadvantaged, for example in a hiring decision4.

Is removing sex or origin from the data enough?

No, according to NIST. Developers sometimes exclude protected attributes, but this does not solve the problem, because the information can be inferred in other ways2.

Is a human reviewing the output enough to correct the error?

Not automatically. NIST calls the idea that a human can effectively and objectively oversee an algorithmic decision system a problematic assumption. It adds that humans bring their own cognitive biases to the use of these systems2. Human validation therefore requires a method, described below.

1How it works1 min

NIST sorts the sources of bias into three families2.

  • Systemic biases: they stem from institutions and history, and are present in data and in practices throughout the AI life cycle2.
  • Statistical and computational biases: they appear when the sample does not represent the population2. One example given: data collected through online questionnaires or taken from social media represent only the users of one platform, but are sometimes used to train tools deployed in much broader contexts2.
  • Human biases: these are systematic errors of reasoning, present in the decisions of institutions, groups and individuals, including in the use of an application once deployed2.

For text, NIST observes that natural language processing datasets often differ from their real uses, which can lead to discrimination and performance gaps2. For facial recognition, it cites gender classification: its accuracy varies by age and ethnic group, and a limited sample, or one not representative of a group, gives lower accuracy for that group2.

2What you do1 min
  • Do not leave a decision that affects people (recruitment, assessment) to an AI alone. It prepares, a person decides.
  • Compare the results. Submit to the tool similar cases that differ only in one personal characteristic and see whether the answer changes. In a case of technology experimentation, the CNIL required a systematic analysis of the system's errors and elements of reflection on the risks of discrimination3.
  • Ask who is not in the data. NIST poses the question « Who is counted, and who is not counted? » and notes that indigenous populations, women and people with disabilities are consistently underrepresented in datasets2.
  • Report the gaps you notice to the person responsible for the tool in the company, with the examples.
  • Keep a human validation that looks at the result with a critical mind and can contradict it. NIST reports that researchers recommend that users of an automated output « consider a deliberate and modest approach »2.
3Pitfalls1 min
  • Overconfidence. NIST calls « automation complacency » the habit of relying too much on automated systems2. It also explains that experts may use AI to ease the pressure of more subjective processes, in the name of the presumed objectivity of automation, and use it as cover for their own implicit biases2.
  • Non-representative data. A tool trained on data that do not reflect the people it assesses makes more mistakes about them2.
  • Average groups applied to individuals. NIST calls « ecological fallacy » the act of deducing something about a person from the group they belong to2.
  • A façade of control. NIST judges that, without significant procedural and cultural support, optimistic expectations about the role of the human overseer do not hold up in practice2.
  • The recruitment example. The CNIL inspected an organization that automatically assessed video CVs recorded by candidates. It found discriminatory biases: the system, meant to qualify soft skills, did not take into account the diversity of accents3.
  • A known case under a title. NIST's reference list cites a Reuters article dated October 2018 titled « Amazon scraps secret AI recruiting tool that showed bias against women »2. The sources cited here do not describe this case in detail.
4What European law says1 min

The European regulation on AI is Regulation (EU) 2024/16894. It classifies uses by level of risk. AI tools for employment and worker management, such as CV-screening software, are among the high-risk use cases. So is the grading of exams in education4.

Among the obligations for high-risk systems are high-quality datasets to minimize the risks of discriminatory results, logging of activity, detailed documentation and appropriate human oversight measures4.

On the timetable, the Commission states that the regulation entered into force on 1 August 2024. The rules for certain high-risk areas, including employment, will apply from 2 December 20274.

On the method side, NIST's AI risk management framework is intended for voluntary use and aims to better incorporate trustworthiness considerations into the design, development, use and evaluation of AI systems5.

Fact sheet

Identifier
CP-274
Wing
4. Artificial intelligence
Brain skill
Responsible AI Use & EU AI Act (use_responsibly)
Brain reflex
air-007
Diagram
G2 · Visual vs Technical Indicators
Languages
de, en, fr
Sources
5 sources checked on 3 November 2026
Sources5 sources
  1. NIST, “Towards a Standard for Identifying and Managing Bias in Artificial Intelligence”, 15 March 2022. https://www.nist.gov/publications/towards-standard-identifying-and-managing-bias-artificial-intelligence (accessed 3 November 2026)
  2. NIST, “Towards a Standard for Identifying and Managing Bias in Artificial Intelligence”. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf (accessed 3 November 2026)
  3. CNIL, “IA : comment être en conformité avec le RGPD ?”, 5 April 2022. https://www.cnil.fr/fr/intelligence-artificielle/ia-comment-etre-en-conformite-avec-le-rgpd (accessed 3 November 2026)
  4. European Commission, “Législation sur l’IA”. https://digital-strategy.ec.europa.eu/fr/policies/regulatory-framework-ai (accessed 3 November 2026)
  5. NIST, “AI Risk Management Framework”. https://www.nist.gov/itl/ai-risk-management-framework (accessed 3 November 2026)

Cite this article

Brain, “Algorithmic bias and data stereotypes”, Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes/

BibTeX 
@misc{brain_cp274_en_2026,
  author = {Brain},
  title = {Algorithmic bias and data stereotypes},
  year = {2026},
  howpublished = {Cyberpedia},
  url = {https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes/},
  note = {CC BY 4.0}
}
CC BY 4.0 licence