---
id: CP-260
slug: temperature-top-p-top-k-sampling
lang: en
title: Why AI never answers the same way twice
description: A language model can answer the same question differently. What temperature, Top-p and Top-k control, and what you do about it.
canonical_url: https://startbrain.io/cyberpedia/en/articles/temperature-top-p-top-k-sampling/
markdown_url: https://startbrain.io/cyberpedia/en/articles/temperature-top-p-top-k-sampling.md
publisher: Brain (https://startbrain.io)
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
cite_as: Brain, "Why AI never answers the same way twice", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/temperature-top-p-top-k-sampling/ (CC BY 4.0)
wing: 4
theme: understand_ai
reflex: air-042
schema: G3
updated: "2026-11-03"
translations:
  en: https://startbrain.io/cyberpedia/en/articles/temperature-top-p-top-k-sampling/
  fr: https://startbrain.io/cyberpedia/fr/articles/temperature-top-p-top-k-ia/
  de: https://startbrain.io/cyberpedia/de/articles/temperatur-top-p-top-k-ki/
  es: https://startbrain.io/cyberpedia/es/articles/temperatura-top-p-top-k-ia/
sources:
  - "n": 1
    title: Prompt engineering techniques - Microsoft Foundry
    publisher: Microsoft
    url: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering
    accessed: "2026-11-03"
  - "n": 2
    title: The Curious Case of Neural Text Degeneration
    publisher: arXiv
    url: https://arxiv.org/abs/1904.09751
    published: "2019-04-22"
    accessed: "2026-11-03"
  - "n": 3
    title: Locally Typical Sampling
    publisher: arXiv
    url: https://arxiv.org/abs/2202.00666
    published: "2022-02-01"
    accessed: "2026-11-03"
  - "n": 4
    title: Training language models to follow instructions with human feedback
    publisher: arXiv
    url: https://arxiv.org/abs/2203.02155
    published: "2022-03-04"
    accessed: "2026-11-03"
  - "n": 5
    title: "Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback"
    publisher: arXiv
    url: https://arxiv.org/abs/2305.14975
    published: "2023-05-24"
    accessed: "2026-11-03"
  - "n": 6
    title: Guidelines for secure AI system development
    publisher: NCSC (UK)
    url: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
    published: "2023-11-27"
    accessed: "2026-11-03"
---

# Why AI never answers the same way twice

Ask an AI assistant the same question twice and the two answers may differ, sometimes on a figure, sometimes on the substance. The machine is not necessarily wrong and is not hesitating: a language model looks for the most probable sequence of words given the text that precedes it [1], and the choice of the next word includes a share of chance that can be adjusted.

This article explains where that chance comes from, what the settings found in tools (temperature, Top-p, Top-k) mean, and what an employee does with this variability: compare, verify, and never take a single answer as proof.

## In 30 seconds

- Where it happens: in text generation tools, every time a question is asked. Microsoft states that the techniques on its page are not recommended for reasoning models (gpt-5, o-series) [1].
- What the model does: it chooses each word among probable candidates. A fluent, confident answer is therefore not a verified answer.
- The reflex: ask the question again, compare the answers, and check every fact and figure against the source before using it.
- If it is already done: if you reused a figure or a quotation without checking, find its source before circulating the document.

## Straight answers

### Why does AI give a different answer on each attempt?

Because the model does not retrieve a ready-made answer. Microsoft describes the principle this way: the model tries to produce the most probable sequence of words after the text it was given [1]. Depending on the temperature chosen, the output is more or less random [1], so it can change from one attempt to the next. The question stays the same, but the path followed is not the same.

### What does temperature control?

Temperature changes the model's output [1]. In Microsoft's documentation, it is set between 0 and 2. A higher value, 0.7 for example, makes the output more random and produces more divergent answers. A lower value, such as 0.2, makes it more focused and more concrete [1]. The same documentation says that a higher temperature suits writing a fictional story, and that a much lower temperature is recommended for a legal document [1].

### And Top-p?

Top-p (Top_probability in Microsoft's documentation) also controls how random the answer is, but in a different way from temperature [1]. Microsoft's general recommendation is to change one of the two parameters at a time, not both [1]. A 2019 paper proposes nucleus sampling: the text is drawn from the dynamic "nucleus" of the probability distribution, which leaves room for diversity while cutting off the unreliable tail of the distribution [2].

### And Top-k?

The sources cited here do not detail Top-k: one of them only compares it with nucleus sampling and with a third method, locally typical sampling, which according to its authors offers comparable quality while consistently reducing degenerate repetitions [3]. For the employee, it is enough to know that these settings exist.

### Why not simply remove the chance?

Because the text becomes poor. The 2019 paper finds that always choosing the most probable text produces text that is "bland and strangely repetitive", in other words dull and strangely repetitive [2]. It also notes that the decoding strategy alone can change the quality of the text greatly, from exactly the same model [2].

### Does one correct answer prove the AI is reliable?

No. Microsoft says so about its own techniques: even when well used, the model's answers still have to be validated, and a well-built prompt that worked in one case does not necessarily generalize to others [1].

## How it works

At each step, the model has several possible words, each with a probability. It does not always take the first one. Temperature and Top-p control how this choice widens or narrows [1]. With a low value, the answers resemble each other more. With a high value, they diverge.

These settings depend on the tool that queries the model. The decoding strategy changes the quality of the text produced by a single model [2].

Some models, including InstructGPT, are fine-tuned with human feedback [4]. The work that produced InstructGPT starts from the observation that larger models do not follow the user's intent better, and that they can produce false, toxic or simply unhelpful outputs [4]. Its authors note that the model improves truthfulness, but that it still makes simple mistakes [4]. A fine-tuned model therefore remains fallible.

## What you do

- Ask the question again two or three times and compare. If the answers contradict each other on a fact, a name or a figure, none of the versions can be taken as settled.
- Treat the answer as a draft. Check every fact and every figure in a reference document before using it in a letter or a report.
- Ask the model for its sources, then check them. Microsoft explains that when citations are requested, a false answer requires two errors by the model: the invented answer and the wrong citation [1]. This can reduce how often answers are false [1].
- Do not conclude that a tool is reliable because it answered well once. Test it on several cases, including cases it does not know.
- Reserve a low temperature setting for tasks that demand precision, and a higher setting for creative tasks, if the tool lets you choose [1].

## Pitfalls

### The plausible but invented answer

The model looks for the most probable continuation [1]. An answer can therefore be well phrased and false. Microsoft specifies that an instruction such as "write only true facts" or "invent nothing" is sometimes not enough to fix the problem [1].

### The confident tone

A 2023 study of models such as ChatGPT, GPT-4 and Claude compared two ways of measuring a model's confidence [5]. It finds that the confidence levels the model states itself are generally better calibrated than its internal probabilities on the TriviaQA, SciQ and TruthfulQA test sets, and often reduce expected calibration error by 50% in relative terms [5]. The study starts from one principle: the confidence of a reliable system should indicate the probability that the answer is correct, so that one can defer to an expert when confidence is low [5].

### Non-reproducible results in a business process

If a step in a process (sorting mail, for example) relies on a model, two runs can produce two different results on the same input. A quality check passed one day does not guarantee the next day's result, and a disputed result cannot necessarily be reproduced identically. The British national cybersecurity centre (NCSC) recommends guidelines so that AI systems work as intended, and considers that security must be a baseline requirement throughout the life of the system [6]. For the employee, the practical rule is to keep a person who reviews the result before it has an effect.

### Setting the temperature does not guarantee accuracy

Lowering the temperature makes answers more focused [1], not more true. A false answer can repeat identically on every attempt. Comparing several attempts helps spot instability, but only checking against the source establishes a fact.

## Sources

1. Microsoft, "Prompt engineering techniques - Microsoft Foundry". https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering (accessed 3 November 2026)
2. arXiv, "The Curious Case of Neural Text Degeneration", 22 April 2019. https://arxiv.org/abs/1904.09751 (accessed 3 November 2026)
3. arXiv, "Locally Typical Sampling", 1 February 2022. https://arxiv.org/abs/2202.00666 (accessed 3 November 2026)
4. arXiv, "Training language models to follow instructions with human feedback", 4 March 2022. https://arxiv.org/abs/2203.02155 (accessed 3 November 2026)
5. arXiv, "Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback", 24 May 2023. https://arxiv.org/abs/2305.14975 (accessed 3 November 2026)
6. NCSC (UK), "Guidelines for secure AI system development", 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 3 November 2026)

---

Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/temperature-top-p-top-k-sampling/
You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.