---
id: CP-261
slug: pretraining-fine-tuning-rlhf-dpo-alignment
lang: en
title: How an AI model is trained and aligned
description: "Pretraining, fine-tuning, RLHF and DPO: the stages of building an AI assistant and what they change for how you use it at work."
canonical_url: https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment/
markdown_url: https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment.md
publisher: Brain (https://startbrain.io)
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
cite_as: Brain, "How an AI model is trained and aligned", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment/ (CC BY 4.0)
wing: 4
theme: understand_ai
reflex: air-001
schema: G3
updated: "2026-11-03"
translations:
  en: https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment/
  fr: https://startbrain.io/cyberpedia/fr/articles/pre-entrainement-fine-tuning-alignement-rlhf-dpo/
  de: https://startbrain.io/cyberpedia/de/articles/training-und-alignment-von-ki-modellen-pretraining-fine-tuning-rlhf-dpo/
  es: https://startbrain.io/cyberpedia/es/articles/preentrenamiento-fine-tuning-alineacion-rlhf-dpo/
sources:
  - "n": 1
    title: Attention Is All You Need
    publisher: arXiv
    url: https://arxiv.org/abs/1706.03762
    published: "2017-06-12"
    accessed: "2026-11-03"
  - "n": 2
    title: Language Models are Few-Shot Learners
    publisher: arXiv
    url: https://arxiv.org/abs/2005.14165
    published: "2020-05-28"
    accessed: "2026-11-03"
  - "n": 3
    title: Training language models to follow instructions with human feedback
    publisher: arXiv
    url: https://arxiv.org/abs/2203.02155
    published: "2022-03-04"
    accessed: "2026-11-03"
  - "n": 4
    title: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
    publisher: arXiv
    url: https://arxiv.org/abs/2204.05862
    published: "2022-04-12"
    accessed: "2026-11-03"
  - "n": 5
    title: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"
    publisher: arXiv
    url: https://arxiv.org/abs/2305.18290
    published: "2023-05-29"
    accessed: "2026-11-03"
  - "n": 6
    title: "Constitutional AI: Harmlessness from AI Feedback"
    publisher: arXiv
    url: https://arxiv.org/abs/2212.08073
    published: "2022-12-15"
    accessed: "2026-11-03"
  - "n": 7
    title: "LoRA: Low-Rank Adaptation of Large Language Models"
    publisher: arXiv
    url: https://arxiv.org/abs/2106.09685
    published: "2021-06-17"
    accessed: "2026-11-03"
  - "n": 8
    title: "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations"
    publisher: NIST
    url: https://csrc.nist.gov/pubs/ai/100/2/e2025/final
    published: "2025-03-24"
    accessed: "2026-11-03"
  - "n": 9
    title: Guidelines for secure AI system development
    publisher: NCSC (UK)
    url: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
    published: "2023-11-27"
    accessed: "2026-11-03"
  - "n": 10
    title: OWASP Top 10 for Large Language Model Applications
    publisher: OWASP
    url: https://owasp.org/projects/top-10-for-large-language-model-applications
    accessed: "2026-11-03"
---

# How an AI model is trained and aligned

An AI assistant does not come out ready-made from a single training run. It goes through several successive stages: a massive reading of texts, learning from examples of instructions, then tuning according to the preferences of evaluators.
For an employee, the interest is practical. Knowing how the answer is produced helps gauge how far to trust it, and decide what can be entrusted to it.

## In 30 seconds

- Where it happens: every time someone queries an AI assistant, whether public or provided by the company.
- What threatens these systems: the NIST report lists data poisoning, evasion and abuse, among others [8].
- The reflex: reread and check every answer before using it, and do not paste sensitive data into a tool without the company's approval.
- If it is already done: notify the security team or the AI contact person, stating the tool used and the nature of the data entered.

## Straight answers

### What are the stages of building an AI assistant?

There are three. Pretraining gives the model a general knowledge of language from very large text corpora. Supervised fine-tuning (SFT) teaches it to respond to instructions, from examples of good answers. Alignment then adjusts its behavior according to human preferences, with methods such as RLHF or DPO.

### What does pretraining do?

The Transformer architecture, presented in 2017, is based solely on attention mechanisms [1]. A common approach consists of pretraining a model on a large corpus of text, then fine-tuning it on a specific task [2]. GPT-3, for example, is a language model with 175 billion parameters, ten times more than any previous non-sparse model [2].

### What does supervised fine-tuning add?

In the case of InstructGPT, the researchers gathered prompts written by evaluators and prompts submitted through the OpenAI API. They collected demonstrations of the desired behavior, then used them to fine-tune GPT-3 by supervised learning [3]. For the employee, the point to remember: in the case of InstructGPT, prompts submitted through the OpenAI API served as the basis of the dataset [3].

### What are RLHF and DPO?

RLHF (reinforcement learning from human feedback) adds a second phase. The researchers collect rankings of the model's answers, then use them to fine-tune the supervised model further [3]. Evaluators compare answers and the system learns to produce the ones they prefer. Other authors applied this method to obtain helpful and harmless assistants, and state that this training improves results on almost all natural language processing evaluations [4].

DPO (Direct Preference Optimization) aims at the same goal more simply. Its authors describe RLHF as a complex and often unstable procedure, and propose a method that replaces it with a simple classification loss [5]. According to its authors, DPO aligns the model with preferences as well as existing methods, or better [5].

### Can an AI also supervise another AI?

Yes, this is the principle of Constitutional AI. The only human supervision is a list of rules or principles, and the model itself produces critiques and revisions of its answers [6]. For harmlessness, no human label identifies the harmful answers: a model evaluates which of two answers is better [6].

### Is a polite model a reliable model?

No. The InstructGPT authors note that larger models do not necessarily follow the user's intent better, and that they can produce answers that are false, toxic or simply unhelpful [3]. Even after alignment, InstructGPT « still makes simple mistakes » [3]. A courteous and confident tone therefore proves nothing about the accuracy of the content.

## How it works

Pretraining makes the model absorb a large corpus of text [2]. The DPO authors stress that this training is entirely unsupervised, which makes precise control of behavior difficult [5].

The next two stages serve to correct this. Fine-tuning teaches expected answers. Alignment adds human judgment, or that of another model, on the quality of several possible answers [3][6].

Fully fine-tuning a very large model is expensive. The LoRA method freezes the weights of the pretrained model and adds small trainable matrices. For GPT-3 175B, it reduces the number of parameters to train by 10,000 times and the GPU memory required by 3 times compared with full fine-tuning [7].

## What you do

- Check the answers that matter: figures, references, names, excerpts of rules. The model can produce false content [3].
- Do not paste sensitive data (customers, contracts, internal code, personal data) into a tool the company has not approved. In the case of InstructGPT, prompts submitted through the OpenAI API served as the basis of the dataset [3].
- Follow the company's usage policy: it states which tools are authorized and which data may go into them.
- Report odd behavior from the model or an answer that seems to reveal information that should not come out.

## Pitfalls

The NIST AI 100-2 E2025 report establishes a common vocabulary for attacks against machine learning systems. Its keywords include data poisoning, evasion, abuse and privacy compromise [8]. It also covers chatbots and large language models [8].

Pitfalls follow from these stages of building:

- Booby-trapped training or fine-tuning data can skew the model. The NIST report places data poisoning among the listed attacks [8].
- NIST also lists abuse and evasion among the keywords of its report on attacks [8].
- Confidential information can leak. The UK NCSC wants AI systems to work « without revealing sensitive data to unauthorised parties » [9]. The OWASP project on language model applications lists the most critical security vulnerabilities of these applications [10].

The NCSC also points out that AI systems have new vulnerabilities, to be considered in addition to the usual IT threats [9].

## Real cases

### March 2022: a small aligned model preferred to a large raw model

In the article published on 4 March 2022, the authors compare models in human evaluations. The answers of the 1.3 billion parameter InstructGPT model are preferred to those of GPT-3 (175 billion), with a hundred times fewer parameters [3]. The same work notes better truthfulness and less toxic content [3]. For an employee, the lesson is limited: alignment makes answers more consistent with what people prefer, but does not remove errors, which the authors still acknowledge [3].

## Sources

1. arXiv, "Attention Is All You Need", 12 June 2017. https://arxiv.org/abs/1706.03762 (accessed 3 November 2026)
2. arXiv, "Language Models are Few-Shot Learners", 28 May 2020. https://arxiv.org/abs/2005.14165 (accessed 3 November 2026)
3. arXiv, "Training language models to follow instructions with human feedback", 4 March 2022. https://arxiv.org/abs/2203.02155 (accessed 3 November 2026)
4. arXiv, "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback", 12 April 2022. https://arxiv.org/abs/2204.05862 (accessed 3 November 2026)
5. arXiv, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", 29 May 2023. https://arxiv.org/abs/2305.18290 (accessed 3 November 2026)
6. arXiv, "Constitutional AI: Harmlessness from AI Feedback", 15 December 2022. https://arxiv.org/abs/2212.08073 (accessed 3 November 2026)
7. arXiv, "LoRA: Low-Rank Adaptation of Large Language Models", 17 June 2021. https://arxiv.org/abs/2106.09685 (accessed 3 November 2026)
8. NIST, "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations", 24 March 2025. https://csrc.nist.gov/pubs/ai/100/2/e2025/final (accessed 3 November 2026)
9. NCSC (UK), "Guidelines for secure AI system development", 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 3 November 2026)
10. OWASP, "OWASP Top 10 for Large Language Model Applications". https://owasp.org/projects/top-10-for-large-language-model-applications (accessed 3 November 2026)

---

Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment/
You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.