Cyberpediaby Brain
ENDEFR

How an AI model is trained and aligned

Pretraining, fine-tuning, RLHF and DPO: the stages of building an AI assistant and what they change for how you use it at work.

In 30 seconds

Where it happens
Every time someone queries an AI assistant, whether public or provided by the company.
What threatens these systems
The NIST report lists data poisoning, evasion and abuse, among others8.
The reflex
Reread and check every answer before using it, and do not paste sensitive data into a tool without the company's approval.
If it is already done
Notify the security team or the AI contact person, stating the tool used and the nature of the data entered.

What it is

An AI assistant does not come out ready-made from a single training run. It goes through several successive stages: a massive reading of texts, learning from examples of instructions, then tuning according to the preferences of evaluators. For an employee, the interest is practical. Knowing how the answer is produced helps gauge how far to trust it, and decide what can be entrusted to it.

Straight answers

What are the stages of building an AI assistant?

There are three. Pretraining gives the model a general knowledge of language from very large text corpora. Supervised fine-tuning (SFT) teaches it to respond to instructions, from examples of good answers. Alignment then adjusts its behavior according to human preferences, with methods such as RLHF or DPO.

What does pretraining do?

The Transformer architecture, presented in 2017, is based solely on attention mechanisms1. A common approach consists of pretraining a model on a large corpus of text, then fine-tuning it on a specific task2. GPT-3, for example, is a language model with 175 billion parameters, ten times more than any previous non-sparse model2.

What does supervised fine-tuning add?

In the case of InstructGPT, the researchers gathered prompts written by evaluators and prompts submitted through the OpenAI API. They collected demonstrations of the desired behavior, then used them to fine-tune GPT-3 by supervised learning3. For the employee, the point to remember: in the case of InstructGPT, prompts submitted through the OpenAI API served as the basis of the dataset3.

What are RLHF and DPO?

RLHF (reinforcement learning from human feedback) adds a second phase. The researchers collect rankings of the model's answers, then use them to fine-tune the supervised model further3. Evaluators compare answers and the system learns to produce the ones they prefer. Other authors applied this method to obtain helpful and harmless assistants, and state that this training improves results on almost all natural language processing evaluations4.

DPO (Direct Preference Optimization) aims at the same goal more simply. Its authors describe RLHF as a complex and often unstable procedure, and propose a method that replaces it with a simple classification loss5. According to its authors, DPO aligns the model with preferences as well as existing methods, or better5.

Can an AI also supervise another AI?

Yes, this is the principle of Constitutional AI. The only human supervision is a list of rules or principles, and the model itself produces critiques and revisions of its answers6. For harmlessness, no human label identifies the harmful answers: a model evaluates which of two answers is better6.

Is a polite model a reliable model?

No. The InstructGPT authors note that larger models do not necessarily follow the user's intent better, and that they can produce answers that are false, toxic or simply unhelpful3. Even after alignment, InstructGPT « still makes simple mistakes »3. A courteous and confident tone therefore proves nothing about the accuracy of the content.

1How it works1 min

Pretraining makes the model absorb a large corpus of text2. The DPO authors stress that this training is entirely unsupervised, which makes precise control of behavior difficult5.

The next two stages serve to correct this. Fine-tuning teaches expected answers. Alignment adds human judgment, or that of another model, on the quality of several possible answers36.

Fully fine-tuning a very large model is expensive. The LoRA method freezes the weights of the pretrained model and adds small trainable matrices. For GPT-3 175B, it reduces the number of parameters to train by 10,000 times and the GPU memory required by 3 times compared with full fine-tuning7.

2What you do1 min
  • Check the answers that matter: figures, references, names, excerpts of rules. The model can produce false content3.
  • Do not paste sensitive data (customers, contracts, internal code, personal data) into a tool the company has not approved. In the case of InstructGPT, prompts submitted through the OpenAI API served as the basis of the dataset3.
  • Follow the company's usage policy: it states which tools are authorized and which data may go into them.
  • Report odd behavior from the model or an answer that seems to reveal information that should not come out.
3Pitfalls1 min

The NIST AI 100-2 E2025 report establishes a common vocabulary for attacks against machine learning systems. Its keywords include data poisoning, evasion, abuse and privacy compromise8. It also covers chatbots and large language models8.

Pitfalls follow from these stages of building:

  • Booby-trapped training or fine-tuning data can skew the model. The NIST report places data poisoning among the listed attacks8.
  • NIST also lists abuse and evasion among the keywords of its report on attacks8.
  • Confidential information can leak. The UK NCSC wants AI systems to work « without revealing sensitive data to unauthorised parties »9. The OWASP project on language model applications lists the most critical security vulnerabilities of these applications10.

The NCSC also points out that AI systems have new vulnerabilities, to be considered in addition to the usual IT threats9.

4Real cases1 min

March 2022: a small aligned model preferred to a large raw model

In the article published on 4 March 2022, the authors compare models in human evaluations. The answers of the 1.3 billion parameter InstructGPT model are preferred to those of GPT-3 (175 billion), with a hundred times fewer parameters3. The same work notes better truthfulness and less toxic content3. For an employee, the lesson is limited: alignment makes answers more consistent with what people prefer, but does not remove errors, which the authors still acknowledge3.

Fact sheet

Identifier
CP-261
Wing
4. Artificial intelligence
Brain skill
Understanding AI & LLMs (understand_ai)
Brain reflex
air-001
Diagram
G3 · System & Protocol Anatomy
Languages
de, en, fr
Sources
10 sources checked on 3 November 2026
Sources10 sources
  1. arXiv, “Attention Is All You Need”, 12 June 2017. https://arxiv.org/abs/1706.03762 (accessed 3 November 2026)
  2. arXiv, “Language Models are Few-Shot Learners”, 28 May 2020. https://arxiv.org/abs/2005.14165 (accessed 3 November 2026)
  3. arXiv, “Training language models to follow instructions with human feedback”, 4 March 2022. https://arxiv.org/abs/2203.02155 (accessed 3 November 2026)
  4. arXiv, “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback”, 12 April 2022. https://arxiv.org/abs/2204.05862 (accessed 3 November 2026)
  5. arXiv, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, 29 May 2023. https://arxiv.org/abs/2305.18290 (accessed 3 November 2026)
  6. arXiv, “Constitutional AI: Harmlessness from AI Feedback”, 15 December 2022. https://arxiv.org/abs/2212.08073 (accessed 3 November 2026)
  7. arXiv, “LoRA: Low-Rank Adaptation of Large Language Models”, 17 June 2021. https://arxiv.org/abs/2106.09685 (accessed 3 November 2026)
  8. NIST, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations”, 24 March 2025. https://csrc.nist.gov/pubs/ai/100/2/e2025/final (accessed 3 November 2026)
  9. NCSC (UK), “Guidelines for secure AI system development”, 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 3 November 2026)
  10. OWASP, “OWASP Top 10 for Large Language Model Applications”. https://owasp.org/projects/top-10-for-large-language-model-applications (accessed 3 November 2026)

Cite this article

Brain, “How an AI model is trained and aligned”, Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment/

BibTeX 
@misc{brain_cp261_en_2026,
  author = {Brain},
  title = {How an AI model is trained and aligned},
  year = {2026},
  howpublished = {Cyberpedia},
  url = {https://startbrain.io/cyberpedia/en/articles/pretraining-fine-tuning-rlhf-dpo-alignment/},
  note = {CC BY 4.0}
}
CC BY 4.0 licence