Context window: an AI model's working memory
A context window limits how much text an AI assistant reads at once. What it handles poorly, and simple habits to use it better.
On this page
In 30 seconds
- Where it happens
- In any conversation with an AI assistant, as soon as you paste a long document or give several instructions.
- What happens
- The model works within a text limit, and even inside that limit it can make poor use of information placed in the middle of a long text[3].
- The reflex
- Split the task, put the instruction at the start and repeat it at the end, then check the answer[5].
- If it's already done
- Reread the answer point by point with the original text in front of you.
What it is
When you ask an AI assistant a question, it answers from the information it is given. Google calls this information the context[2]. The amount of text the model can take into account at once is called the context window. Google compares it to short-term memory: a person can hold only a limited amount of information, and the same is true of generative models[2].
The window is measured in tokens, not pages. Recent models have made it much larger, but research shows that a large window does not guarantee reliable reading of everything it contains[3][4]. This page explains what an employee should take from this and the habits that limit errors.
Straight answers
What is a token?
A language model first splits text into units called tokens: words, groups of characters, or combinations of words and punctuation[1]. The splitting method varies by model. OpenAI's GPT models, for example, use subword splitting[1]. Microsoft notes that common multi-syllable words often fit in a single token, while rarer words are split into syllables[5].
What is the context window?
It is the maximum number of tokens the model can process[1]. This limit is often expressed as a combined window, covering input tokens (what you write) and output tokens (what the model produces) together[1]. The longer the input, the less room is left for the output.
Microsoft gives a worked example: with a window of 100 tokens and a nine-word sentence split word by word, the input takes nine tokens and 91 remain for the output. Split character by character, the same sentence takes 34 and only 66 remain[1].
How big is a window today?
According to Google, most generative models built in recent years could process only 8,000 tokens at a time, then more recent models accepted 32,000 or 128,000 tokens[2]. Google states that Gemini accepts 1 million tokens, which corresponds to 8 average-length English novels or 50,000 lines of code[2].
Does the model keep what I gave it in memory?
Examples placed in a prompt do not change the model in any lasting way: they only steer it for the current answer[5]. Google describes methods that were used to manage narrow windows: dropping old messages as new ones arrive, or replacing them with a summary when the window is close to full[2].
Does a large window prevent errors?
No. Researchers tested several models, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct and Mistral Instruct, on sentiment analysis and news article classification. They report that the models fall short on long sequences despite their large windows[4]. Google also acknowledges a limit: when looking for several specific pieces of information in a long context, the model does not reach the same accuracy as for a single one, and performance can vary widely depending on the context[2].
How it works
Microsoft points out that, whatever the prompt, the model answers with what it judges most likely based on its training data, and that it does not follow a separate processing path for questions[5].
The position of information matters. The "Lost in the Middle" study, submitted in July 2023, measured model performance on two tasks: answering a question from several documents, and retrieving a value associated with a key. The authors observe that performance is often best when the relevant information is at the beginning or the end of the supplied text, and that it degrades significantly when it is in the middle, including for models designed for long contexts[3].
Microsoft separately describes a recency bias: information at the end of a prompt may influence the answer more than information at the start[5]. Microsoft also states that the order of elements can change the result, including the order of the examples given to the model[5].
Length has a cost. Generative AI services often bill per token and apply a maximum number of tokens per minute[1]. For an organization that runs an assistant, OWASP lists among the risks the continuous sending of inputs that exceed the context window, which can cause excessive resource consumption and degrade the service[7]. OWASP recommends strict input validation so that inputs do not exceed reasonable size limits[7].
What you do
- Split the task. Microsoft states that language models often do better when the task is divided into smaller steps, for example extracting the relevant facts first, then writing the rest[5].
- Put the instruction at the start, repeat it at the end. Microsoft states that stating the task at the beginning of the prompt, before the context and examples, may help produce better outputs, and that repeating the instructions at the end of the prompt is worth trying[5]. Microsoft adds that its tests with ChatGPT and GPT-4 gave the same answer with or without the first technique[5].
- Give the model a way out. Microsoft suggests, for example, asking it to answer "not found" if the answer is not present, which may help the model avoid a wrong answer[5].
- Provide the reference text and ask for citations. Microsoft strongly recommends giving the model data to rely on when the answer must be based on reliable, up-to-date information[5]. Asking for citations may help reduce how often answers are wrong, and citations placed in the text, next to the claim, are better safeguards than citations at the end of the answer[5].
- Always check. Even with a well-written prompt, Microsoft states that the model's answers must be validated, because a prompt that worked well in one case does not necessarily generalize[5].
Pitfalls
The pasted document may contain instructions
When an assistant reads a website or a file, the content may contain instructions that the model interprets and that change its behavior: OWASP calls this indirect prompt injection[6]. In a scenario OWASP describes, a user asks a model to summarize a web page containing hidden instructions, which lead to the exfiltration of the private conversation[6]. OWASP recommends separating and clearly identifying untrusted content, and requiring human approval for privileged operations[6].
A large window gives false assurance
Google states that, for a single piece of information being sought, roughly 99% success can be reached, but that the result is not the same when several must be retrieved[2]. An answer may therefore look complete while omitting points from a long text.
The advice depends on the model
Microsoft warns that each model behaves differently and that its techniques do not apply the same way to all of them. These techniques are not recommended for reasoning models such as gpt-5 and the o series[5].
Sources
- Microsoft, “Understand tokens”. https://learn.microsoft.com/en-us/dotnet/ai/conceptual/understanding-tokens (accessed 14 November 2026)
- Google, “Long context | Gemini Enterprise Agent Platform | Google Cloud Documentation”. https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/long-context (accessed 14 November 2026)
- arXiv, “Lost in the Middle: How Language Models Use Long Contexts”, 6 July 2023. https://arxiv.org/abs/2307.03172 (accessed 14 November 2026)
- arXiv, “Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly”, 3 August 2024. https://arxiv.org/abs/2408.01866 (accessed 14 November 2026)
- Microsoft, “Prompt engineering techniques”. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering (accessed 14 November 2026)
- OWASP, “LLM01:2025 Prompt Injection”, 10 April 2024. https://genai.owasp.org/llmrisk/llm01-prompt-injection/ (accessed 14 November 2026)
- OWASP, “LLM10:2025 Unbounded Consumption”, 10 April 2024. https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/ (accessed 14 November 2026)
Cite this article
Brain, “Context window: an AI model's working memory”, Cyberpedia, 2026. https:/
BibTeX
@misc{brain_cp259_en_2026,
author = {Brain},
title = {Context window: an AI model's working memory},
year = {2026},
howpublished = {Cyberpedia},
url = {https://startbrain.io/cyberpedia/en/articles/llm-context-window/},
note = {CC BY 4.0}
}