--- id: CP-274 slug: algorithmic-bias-training-data-stereotypes lang: en title: Algorithmic bias and data stereotypes description: An AI repeats the stereotypes in its training data while looking neutral. What employees check, and what the AI Act provides. canonical_url: https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes/ markdown_url: https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes.md publisher: Brain (https://startbrain.io) license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) cite_as: Brain, "Algorithmic bias and data stereotypes", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes/ (CC BY 4.0) wing: 4 theme: use_responsibly reflex: air-007 schema: G2 updated: "2026-11-03" translations: en: https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes/ fr: https://startbrain.io/cyberpedia/fr/articles/biais-algorithmiques-stereotypes-donnees/ de: https://startbrain.io/cyberpedia/de/articles/algorithmische-verzerrungen-stereotype-trainingsdaten/ es: https://startbrain.io/cyberpedia/es/articles/sesgos-algoritmicos-estereotipos-datos/ sources: - "n": 1 title: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence publisher: NIST url: https://www.nist.gov/publications/towards-standard-identifying-and-managing-bias-artificial-intelligence published: "2022-03-15" accessed: "2026-11-03" - "n": 2 title: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence publisher: NIST url: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf accessed: "2026-11-03" - "n": 3 title: "IA : comment être en conformité avec le RGPD ?" publisher: CNIL url: https://www.cnil.fr/fr/intelligence-artificielle/ia-comment-etre-en-conformite-avec-le-rgpd published: "2022-04-05" accessed: "2026-11-03" - "n": 4 title: Législation sur l’IA publisher: European Commission url: https://digital-strategy.ec.europa.eu/fr/policies/regulatory-framework-ai accessed: "2026-11-03" - "n": 5 title: AI Risk Management Framework publisher: NIST url: https://www.nist.gov/itl/ai-risk-management-framework accessed: "2026-11-03" --- # Algorithmic bias and data stereotypes An AI tool learns from data produced by people, in organizations and a society that have their own habits and their own inequalities. What it draws from that data can reproduce stereotypes, without anyone intending it. NIST, the American standards body, writes that biases « can lead to harmful impacts regardless of intent » [1]. The result arrives in a numerical or ranked form, which looks objective. This page explains where these gaps come from and what an employee does when using AI to sort or score people. ## In 30 seconds - Where it happens: CV screening, scoring, candidate assessment, text or image generation, anywhere AI ranks or describes people. - What happens: the tool reproduces gaps present in its data, and its answer seems neutral. - The reflex: never let an AI alone decide a matter that affects people, compare the results, report the gaps and keep a human validation. - If it is already done: tell the person in charge of the tool or the relevant contact, and have a person review the decision. ## Straight answers ### Why does an AI reproduce stereotypes? Because it learns from data that contain them. NIST distinguishes three categories of bias in AI: systemic, statistical and human [2]. Systemic biases come from the procedures and practices of institutions that advantage some groups and disadvantage others. This requires no conscious prejudice: it is enough that the majority follows the existing rules and norms [2]. These biases are found in the datasets used in AI [2]. ### Can data be accurate and still biased? Yes. The CNIL describes two cases. The training data may be non-representative. They may also be well representative of the « real world » and still reflect a discriminatory character, for example the reproduction of pay gaps between women and men [3]. NIST points the same way: even when representative, datasets can present entrenched historical and systemic biases [2]. ### Why does the answer look neutral? NIST notes that biases are widespread in technological processes and can cause harm regardless of intent [1]. The tool does not show where its answer comes from. The European Commission points out that it is often impossible to know why an AI system made a decision, which makes it hard to know whether a person was unfairly disadvantaged, for example in a hiring decision [4]. ### Is removing sex or origin from the data enough? No, according to NIST. Developers sometimes exclude protected attributes, but this does not solve the problem, because the information can be inferred in other ways [2]. ### Is a human reviewing the output enough to correct the error? Not automatically. NIST calls the idea that a human can effectively and objectively oversee an algorithmic decision system a problematic assumption. It adds that humans bring their own cognitive biases to the use of these systems [2]. Human validation therefore requires a method, described below. ## How it works NIST sorts the sources of bias into three families [2]. - Systemic biases: they stem from institutions and history, and are present in data and in practices throughout the AI life cycle [2]. - Statistical and computational biases: they appear when the sample does not represent the population [2]. One example given: data collected through online questionnaires or taken from social media represent only the users of one platform, but are sometimes used to train tools deployed in much broader contexts [2]. - Human biases: these are systematic errors of reasoning, present in the decisions of institutions, groups and individuals, including in the use of an application once deployed [2]. For text, NIST observes that natural language processing datasets often differ from their real uses, which can lead to discrimination and performance gaps [2]. For facial recognition, it cites gender classification: its accuracy varies by age and ethnic group, and a limited sample, or one not representative of a group, gives lower accuracy for that group [2]. ## What you do - Do not leave a decision that affects people (recruitment, assessment) to an AI alone. It prepares, a person decides. - Compare the results. Submit to the tool similar cases that differ only in one personal characteristic and see whether the answer changes. In a case of technology experimentation, the CNIL required a systematic analysis of the system's errors and elements of reflection on the risks of discrimination [3]. - Ask who is not in the data. NIST poses the question « Who is counted, and who is not counted? » and notes that indigenous populations, women and people with disabilities are consistently underrepresented in datasets [2]. - Report the gaps you notice to the person responsible for the tool in the company, with the examples. - Keep a human validation that looks at the result with a critical mind and can contradict it. NIST reports that researchers recommend that users of an automated output « consider a deliberate and modest approach » [2]. ## Pitfalls - Overconfidence. NIST calls « automation complacency » the habit of relying too much on automated systems [2]. It also explains that experts may use AI to ease the pressure of more subjective processes, in the name of the presumed objectivity of automation, and use it as cover for their own implicit biases [2]. - Non-representative data. A tool trained on data that do not reflect the people it assesses makes more mistakes about them [2]. - Average groups applied to individuals. NIST calls « ecological fallacy » the act of deducing something about a person from the group they belong to [2]. - A façade of control. NIST judges that, without significant procedural and cultural support, optimistic expectations about the role of the human overseer do not hold up in practice [2]. - The recruitment example. The CNIL inspected an organization that automatically assessed video CVs recorded by candidates. It found discriminatory biases: the system, meant to qualify soft skills, did not take into account the diversity of accents [3]. - A known case under a title. NIST's reference list cites a Reuters article dated October 2018 titled « Amazon scraps secret AI recruiting tool that showed bias against women » [2]. The sources cited here do not describe this case in detail. ## What European law says The European regulation on AI is Regulation (EU) 2024/1689 [4]. It classifies uses by level of risk. AI tools for employment and worker management, such as CV-screening software, are among the high-risk use cases. So is the grading of exams in education [4]. Among the obligations for high-risk systems are high-quality datasets to minimize the risks of discriminatory results, logging of activity, detailed documentation and appropriate human oversight measures [4]. On the timetable, the Commission states that the regulation entered into force on 1 August 2024. The rules for certain high-risk areas, including employment, will apply from 2 December 2027 [4]. On the method side, NIST's AI risk management framework is intended for voluntary use and aims to better incorporate trustworthiness considerations into the design, development, use and evaluation of AI systems [5]. ## Sources 1. NIST, "Towards a Standard for Identifying and Managing Bias in Artificial Intelligence", 15 March 2022. https://www.nist.gov/publications/towards-standard-identifying-and-managing-bias-artificial-intelligence (accessed 3 November 2026) 2. NIST, "Towards a Standard for Identifying and Managing Bias in Artificial Intelligence". https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf (accessed 3 November 2026) 3. CNIL, "IA : comment être en conformité avec le RGPD ?", 5 April 2022. https://www.cnil.fr/fr/intelligence-artificielle/ia-comment-etre-en-conformite-avec-le-rgpd (accessed 3 November 2026) 4. European Commission, "Législation sur l’IA". https://digital-strategy.ec.europa.eu/fr/policies/regulatory-framework-ai (accessed 3 November 2026) 5. NIST, "AI Risk Management Framework". https://www.nist.gov/itl/ai-risk-management-framework (accessed 3 November 2026) --- Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/algorithmic-bias-training-data-stereotypes/ You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.