--- id: CP-279 slug: model-collapse-ai-slop-web-pollution lang: en title: Model collapse and AI slop on the web description: How AIs trained on AI-written text lose diversity, and how to check generated content before using it at work. canonical_url: https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution/ markdown_url: https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution.md publisher: Brain (https://startbrain.io) license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) cite_as: Brain, "Model collapse and AI slop on the web", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution/ (CC BY 4.0) wing: 4 theme: understand_ai reflex: air-003 schema: G2 updated: "2026-11-03" translations: en: https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution/ fr: https://startbrain.io/cyberpedia/fr/articles/derive-modele-pollution-web-contenu-synthetique/ de: https://startbrain.io/cyberpedia/de/articles/modellkollaps-ki-inhalte-web-verschmutzung/ es: https://startbrain.io/cyberpedia/es/articles/deriva-modelos-contaminacion-web-contenido-sintetico/ sources: - "n": 1 title: "The Curse of Recursion: Training on Generated Data Makes Models Forget" publisher: arXiv url: https://arxiv.org/abs/2305.17493 published: "2023-05-27" accessed: "2026-11-03" - "n": 2 title: "The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text" publisher: arXiv url: https://arxiv.org/abs/2311.09807 published: "2023-11-16" accessed: "2026-11-03" - "n": 3 title: Self-Consuming Generative Models Go MAD publisher: arXiv url: https://arxiv.org/abs/2307.01850 published: "2023-07-04" accessed: "2026-11-03" - "n": 4 title: LLM04:2025 Data and Model Poisoning publisher: OWASP url: https://genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning/ accessed: "2026-11-03" - "n": 5 title: LLM09:2025 Misinformation publisher: OWASP url: https://genai.owasp.org/llmrisk/llm092025-misinformation/ accessed: "2026-11-03" - "n": 6 title: LLM03:2025 Supply Chain publisher: OWASP url: https://genai.owasp.org/llmrisk/llm032025-supply-chain/ accessed: "2026-11-03" - "n": 7 title: Guidelines for secure AI system development publisher: NCSC (UK) url: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development published: "2023-11-27" accessed: "2026-11-03" - "n": 8 title: "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" publisher: NIST url: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence published: "2024-07-26" accessed: "2026-11-03" --- # Model collapse and AI slop on the web Writing assistants produce texts that other models, or other employees, then reuse. When these generated texts are used to train new models, researchers have found irreversible defects. This effect is called « Model Collapse », translated here as model drift [1]. This page explains the mechanism without going into equations, then gives simple habits: check a piece of information, cross-check it, spot doubtful content and never reuse a generated text as it stands. ## In 30 seconds - Where it happens: in text and image generation tools, and in the online content used to train them. - What it changes for you: an answer can look credible and be wrong, or be only an impoverished version of existing content. - The reflex: do not reuse any generated text without cross-checking it against a reliable source that you opened yourself. - If it is already done: reread the document, find the source of each fact, correct it, and warn the recipients if an error has gone out. ## Straight answers ### What is model drift? It is the degradation of a model trained on content produced by other models. Researchers found that using generated content for training causes irreversible defects: the « tails » of the original distribution, that is, the rare cases, disappear. They named this effect « Model Collapse » and observed it in variational autoencoders, Gaussian mixture models and large language models [1]. ### Is the phenomenon limited to text? No. Another team studied generative image models in loops where each generation is trained on the outputs of the previous one. Without enough fresh real data in each generation, the quality (precision) or the diversity (recall) of future models gradually decreases. The authors call this state « Model Autophagy Disorder » (MAD), by analogy with mad cow disease [3]. ### What about language models? A study accepted at NAACL 2024 (Findings) measured lexical, syntactic and semantic diversity after successive fine-tuning on synthetic text, in English. It reports a steady decline in the diversity of outputs over the iterations, particularly marked for tasks that call for creativity [2]. ### Can a generated text be wrong without the tool « lying »? Yes. OWASP describes hallucination: the model fills gaps with statistical patterns, without understanding the content, and produces answers that seem right but have no foundation. Biases in the training data and incomplete information also contribute to it [5]. ### Can training data be deliberately booby-trapped? Yes. OWASP calls data poisoning the manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases. The risk is high with external sources, which may contain unverified or malicious content [4]. ## How it works A language model learns from large amounts of text collected on the Web. If a growing share of that text comes from models, the next one learns partly from copies. The authors of the first study argue that this risk must be taken seriously if we are to keep benefiting from training on data collected at large scale on the Web. They add that data from genuine human interactions with the systems will become more valuable as generated content spreads in the data collected from the Internet [1]. The other work establishes that without enough fresh real data in each generation, quality or diversity decreases [3]. The pages consulted do not quantify the share of synthetic content currently on the Web, and this page gives no figure. ## What you do - Treat a generated answer as a draft. OWASP names overreliance: users integrate incorrect data into important decisions without sufficient review [5]. - Cross-check with reliable external sources. OWASP recommends cross-referencing the model's outputs with those sources and providing for human review and fact-checking for sensitive information [5]. - Find the origin of each fact, figure or quotation before writing it into a document. If you find no source, remove the fact. - Check that a reference exists before using it. OWASP cites a case where ChatGPT fabricated fake legal cases, with serious consequences in court [5]. - For code, do not take a suggested library without checking it. Assistants suggest third-party libraries that are unsafe or do not exist [5]. - Spot doubtful content: text with no author or source, claims with no support, contradictions with a source you know. These criteria are common-sense markers of this page, not a method drawn from the sources. - Flag generated content when you share it. OWASP recommends clearly labelling AI-generated content and informing people about reliability limits [5]. ## Pitfalls - A fluent answer is not a verified answer: a model's misinformation appears credible [5]. - A source that reuses a generated text is not a second opinion. Two pages that say the same thing can come from the same model. - The models themselves can come from outside. OWASP notes that there is currently no strong assurance about the provenance of published models: model cards give none about their origin [6]. - A poisoned model can behave normally until a trigger changes its behavior, which makes the defect hard to detect [4]. - A tool's terms of use can allow your data to be used for training. OWASP flags this risk when the terms and the privacy policy are unclear [6]. Do not enter sensitive information in a tool not approved by your company. - An AI error binds the organization even without an attacker. OWASP cites the Air Canada chatbot, whose erroneous information to travelers led to disruption and legal complications, the company having been successfully sued [5]. On the organization side, NIST offers a cross-sector profile for generative AI, a companion to AI RMF 1.0, for voluntary use [8]. The UK NCSC guidelines, published on 27 November 2023, cover the design, development, deployment and operation of AI systems, including supply chain security [7]. ## Real cases ### 27 May 2023: the « curse of recursion » Ilia Shumailov and five coauthors submit to arXiv a paper that asks what will happen to future GPT models when large language models produce much of the text online. Their answer: generated content used in training causes irreversible defects. They show that the effect can occur in several families of models [1]. ### 4 July 2023: model autophagy disorder Sina Alemohammad, Richard G. Baraniuk and six coauthors submit to arXiv an analysis of three families of self-consuming loops on image models. Their conclusion: without enough fresh real data, precision or diversity decreases from generation to generation [3]. ### 16 November 2023: the decline of linguistic diversity Yanzhu Guo and coauthors submit to arXiv recursive fine-tuning experiments in English. The diversity of outputs drops with each iteration, especially for creative tasks [2]. ## Sources 1. arXiv, "The Curse of Recursion: Training on Generated Data Makes Models Forget", 27 May 2023. https://arxiv.org/abs/2305.17493 (accessed 3 November 2026) 2. arXiv, "The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text", 16 November 2023. https://arxiv.org/abs/2311.09807 (accessed 3 November 2026) 3. arXiv, "Self-Consuming Generative Models Go MAD", 4 July 2023. https://arxiv.org/abs/2307.01850 (accessed 3 November 2026) 4. OWASP, "LLM04:2025 Data and Model Poisoning". https://genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning/ (accessed 3 November 2026) 5. OWASP, "LLM09:2025 Misinformation". https://genai.owasp.org/llmrisk/llm092025-misinformation/ (accessed 3 November 2026) 6. OWASP, "LLM03:2025 Supply Chain". https://genai.owasp.org/llmrisk/llm032025-supply-chain/ (accessed 3 November 2026) 7. NCSC (UK), "Guidelines for secure AI system development", 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 3 November 2026) 8. NIST, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", 26 July 2024. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence (accessed 3 November 2026) --- Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution/ You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.