Cyberpediaby Brain
ENDEFR

Model collapse and AI slop on the web

How AIs trained on AI-written text lose diversity, and how to check generated content before using it at work.

In 30 seconds

Where it happens
In text and image generation tools, and in the online content used to train them.
What it changes for you
An answer can look credible and be wrong, or be only an impoverished version of existing content.
The reflex
Do not reuse any generated text without cross-checking it against a reliable source that you opened yourself.
If it is already done
Reread the document, find the source of each fact, correct it, and warn the recipients if an error has gone out.

What it is

Writing assistants produce texts that other models, or other employees, then reuse. When these generated texts are used to train new models, researchers have found irreversible defects. This effect is called « Model Collapse », translated here as model drift1.

This page explains the mechanism without going into equations, then gives simple habits: check a piece of information, cross-check it, spot doubtful content and never reuse a generated text as it stands.

Straight answers

What is model drift?

It is the degradation of a model trained on content produced by other models. Researchers found that using generated content for training causes irreversible defects: the « tails » of the original distribution, that is, the rare cases, disappear. They named this effect « Model Collapse » and observed it in variational autoencoders, Gaussian mixture models and large language models1.

Is the phenomenon limited to text?

No. Another team studied generative image models in loops where each generation is trained on the outputs of the previous one. Without enough fresh real data in each generation, the quality (precision) or the diversity (recall) of future models gradually decreases. The authors call this state « Model Autophagy Disorder » (MAD), by analogy with mad cow disease3.

What about language models?

A study accepted at NAACL 2024 (Findings) measured lexical, syntactic and semantic diversity after successive fine-tuning on synthetic text, in English. It reports a steady decline in the diversity of outputs over the iterations, particularly marked for tasks that call for creativity2.

Can a generated text be wrong without the tool « lying »?

Yes. OWASP describes hallucination: the model fills gaps with statistical patterns, without understanding the content, and produces answers that seem right but have no foundation. Biases in the training data and incomplete information also contribute to it5.

Can training data be deliberately booby-trapped?

Yes. OWASP calls data poisoning the manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases. The risk is high with external sources, which may contain unverified or malicious content4.

1How it works1 min

A language model learns from large amounts of text collected on the Web. If a growing share of that text comes from models, the next one learns partly from copies. The authors of the first study argue that this risk must be taken seriously if we are to keep benefiting from training on data collected at large scale on the Web. They add that data from genuine human interactions with the systems will become more valuable as generated content spreads in the data collected from the Internet1.

The other work establishes that without enough fresh real data in each generation, quality or diversity decreases3. The pages consulted do not quantify the share of synthetic content currently on the Web, and this page gives no figure.

2What you do1 min
  • Treat a generated answer as a draft. OWASP names overreliance: users integrate incorrect data into important decisions without sufficient review5.
  • Cross-check with reliable external sources. OWASP recommends cross-referencing the model's outputs with those sources and providing for human review and fact-checking for sensitive information5.
  • Find the origin of each fact, figure or quotation before writing it into a document. If you find no source, remove the fact.
  • Check that a reference exists before using it. OWASP cites a case where ChatGPT fabricated fake legal cases, with serious consequences in court5.
  • For code, do not take a suggested library without checking it. Assistants suggest third-party libraries that are unsafe or do not exist5.
  • Spot doubtful content: text with no author or source, claims with no support, contradictions with a source you know. These criteria are common-sense markers of this page, not a method drawn from the sources.
  • Flag generated content when you share it. OWASP recommends clearly labelling AI-generated content and informing people about reliability limits5.
3Pitfalls1 min
  • A fluent answer is not a verified answer: a model's misinformation appears credible5.
  • A source that reuses a generated text is not a second opinion. Two pages that say the same thing can come from the same model.
  • The models themselves can come from outside. OWASP notes that there is currently no strong assurance about the provenance of published models: model cards give none about their origin6.
  • A poisoned model can behave normally until a trigger changes its behavior, which makes the defect hard to detect4.
  • A tool's terms of use can allow your data to be used for training. OWASP flags this risk when the terms and the privacy policy are unclear6. Do not enter sensitive information in a tool not approved by your company.
  • An AI error binds the organization even without an attacker. OWASP cites the Air Canada chatbot, whose erroneous information to travelers led to disruption and legal complications, the company having been successfully sued5.

On the organization side, NIST offers a cross-sector profile for generative AI, a companion to AI RMF 1.0, for voluntary use8. The UK NCSC guidelines, published on 27 November 2023, cover the design, development, deployment and operation of AI systems, including supply chain security7.

4Real cases1 min

27 May 2023: the « curse of recursion »

Ilia Shumailov and five coauthors submit to arXiv a paper that asks what will happen to future GPT models when large language models produce much of the text online. Their answer: generated content used in training causes irreversible defects. They show that the effect can occur in several families of models1.

4 July 2023: model autophagy disorder

Sina Alemohammad, Richard G. Baraniuk and six coauthors submit to arXiv an analysis of three families of self-consuming loops on image models. Their conclusion: without enough fresh real data, precision or diversity decreases from generation to generation3.

16 November 2023: the decline of linguistic diversity

Yanzhu Guo and coauthors submit to arXiv recursive fine-tuning experiments in English. The diversity of outputs drops with each iteration, especially for creative tasks2.

Fact sheet

Identifier
CP-279
Wing
4. Artificial intelligence
Brain skill
Understanding AI & LLMs (understand_ai)
Brain reflex
air-003
Diagram
G2 · Visual vs Technical Indicators
Languages
de, en, fr
Sources
8 sources checked on 3 November 2026
Sources8 sources
  1. arXiv, “The Curse of Recursion: Training on Generated Data Makes Models Forget”, 27 May 2023. https://arxiv.org/abs/2305.17493 (accessed 3 November 2026)
  2. arXiv, “The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text”, 16 November 2023. https://arxiv.org/abs/2311.09807 (accessed 3 November 2026)
  3. arXiv, “Self-Consuming Generative Models Go MAD”, 4 July 2023. https://arxiv.org/abs/2307.01850 (accessed 3 November 2026)
  4. OWASP, “LLM04:2025 Data and Model Poisoning”. https://genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning/ (accessed 3 November 2026)
  5. OWASP, “LLM09:2025 Misinformation”. https://genai.owasp.org/llmrisk/llm092025-misinformation/ (accessed 3 November 2026)
  6. OWASP, “LLM03:2025 Supply Chain”. https://genai.owasp.org/llmrisk/llm032025-supply-chain/ (accessed 3 November 2026)
  7. NCSC (UK), “Guidelines for secure AI system development”, 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 3 November 2026)
  8. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”, 26 July 2024. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence (accessed 3 November 2026)

Cite this article

Brain, “Model collapse and AI slop on the web”, Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution/

BibTeX 
@misc{brain_cp279_en_2026,
  author = {Brain},
  title = {Model collapse and AI slop on the web},
  year = {2026},
  howpublished = {Cyberpedia},
  url = {https://startbrain.io/cyberpedia/en/articles/model-collapse-ai-slop-web-pollution/},
  note = {CC BY 4.0}
}
CC BY 4.0 licence