---
id: CP-258
slug: llm-tokenization-explained
lang: en
title: LLM tokenization
description: A language model reads text fragments, not sentences. What this means for an employee, and the security pitfalls.
canonical_url: https://startbrain.io/cyberpedia/en/articles/llm-tokenization-explained/
markdown_url: https://startbrain.io/cyberpedia/en/articles/llm-tokenization-explained.md
publisher: Brain (https://startbrain.io)
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
cite_as: Brain, "LLM tokenization", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/llm-tokenization-explained/ (CC BY 4.0)
wing: 4
theme: understand_ai
reflex: air-005
schema: G3
updated: "2026-11-03"
translations:
  en: https://startbrain.io/cyberpedia/en/articles/llm-tokenization-explained/
  fr: https://startbrain.io/cyberpedia/fr/articles/tokenisation-llm-fragments/
  de: https://startbrain.io/cyberpedia/de/articles/tokenisierung-llm-textfragmente/
  es: https://startbrain.io/cyberpedia/es/articles/tokenizacion-llm-fragmentos/
sources:
  - "n": 1
    title: Neural Machine Translation of Rare Words with Subword Units
    publisher: arXiv
    url: https://arxiv.org/abs/1508.07909
    published: "2015-08-31"
    accessed: "2026-11-03"
  - "n": 2
    title: LLM01:2025 Prompt Injection
    publisher: OWASP
    url: https://genai.owasp.org/llmrisk/llm01-prompt-injection/
    published: "2024-04-10"
    accessed: "2026-11-03"
  - "n": 3
    title: "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"
    publisher: arXiv
    url: https://arxiv.org/abs/1808.06226
    published: "2018-08-19"
    accessed: "2026-11-03"
  - "n": 4
    title: Language Models are Few-Shot Learners
    publisher: arXiv
    url: https://arxiv.org/abs/2005.14165
    published: "2020-05-28"
    accessed: "2026-11-03"
  - "n": 5
    title: Guidelines for secure AI system development
    publisher: NCSC (UK)
    url: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
    published: "2023-11-27"
    accessed: "2026-11-03"
  - "n": 6
    title: Universal and Transferable Adversarial Attacks on Aligned Language Models
    publisher: arXiv
    url: https://arxiv.org/abs/2307.15043
    published: "2023-07-27"
    accessed: "2026-11-03"
  - "n": 7
    title: "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations"
    publisher: NIST
    url: https://csrc.nist.gov/pubs/ai/100/2/e2023/final
    accessed: "2026-11-03"
---

# LLM tokenization

When you write to an AI assistant, it feels as if it reads sentences the way a colleague would. The research this article relies on describes a different mechanism: rare words are encoded as sequences of units smaller than a word, called fragments (or tokens) here [1].

This article explains the idea in plain terms, then what an employee can take from it to word requests, check answers and avoid a few security pitfalls. The sources in the file describe how text is split and how attacks using confusing text work. They measure neither the cost of an exchange nor the context window size of any given product: those points are therefore not covered here.

## In 30 seconds

- Where it happens: every time you paste a text, a name or a document into an AI assistant, or ask it to summarize a page.
- What the attacker wants: to make the model read instructions you cannot see, because a prompt injection can be imperceptible to a human [2].
- The reflex: write precise instructions, check figures and quotations against the source, and never paste sensitive data into a tool that has not been approved.
- If it's already done: tell your security team, stating the tool used and the nature of the data pasted.

## Straight answers

### Does a language model read words the way we write them?

Not as they are. In 2015, researchers described machine translation models that work with a fixed vocabulary, even though translation is an open-vocabulary problem [1]. Their solution is to encode rare or unknown words as sequences of units smaller than a word [1].

### How is the text split?

The 2015 authors compare several segmentation techniques, including simple character-sequence models and a segmentation based on the "byte pair encoding" compression algorithm [1]. They measure a gain of 1.1 BLEU points for English to German and 1.3 for English to Russian, compared with a dictionary back-off method [1].

### Does it depend on the language?

The SentencePiece tool, presented in 2018 by Taku Kudo and John Richardson, describes itself as a language-independent subword tokenizer [3]. It learns its units directly from raw sentences, whereas existing tools assumed text already split into words [3]. It is released as open source under the Apache 2 license [3]. The sources cited here do not measure how far one language is served worse than another: that question stays open here.

### Why does a rare name or an invented word cause trouble?

The intuition of the 2015 paper is that proper names, compound words and loanwords translate better in pieces than as whole words [1]. Rare words are encoded as subword units [1]. The link with mistakes on rare names is this article's reading, not a result measured in the sources.

### Do the models have known limits?

The GPT-3 model, presented in 2020 with 175 billion parameters, receives its tasks and examples purely as text [4]. Its authors note that it succeeds at tasks such as unscrambling words or three-digit addition, but also that it still struggles on some datasets [4]. A correct result once therefore does not guarantee a correct result the next time.

## How it works

The sources give the following principle. The model works with a vocabulary of fixed size [1]. Rare words are rebuilt from several units [1]. The splitting tool can be trained on raw text, without prior splitting into words [3].

Instructions travel through the same channel as data. In the GPT-3 study, tasks and examples are "specified purely via text interaction with the model" [4]. According to OWASP, prompt injection refers to inputs that alter the model's behavior in unintended ways, even if they are imperceptible to a human, as long as the model reads them [2]. Putting the two points together is this article's reading, not a claim by OWASP.

## What you do

Write clear instructions. State the expected role, the task and the format of the answer. OWASP recommends that system designers specify the model's role, capabilities and limitations, and define a clear output format [2]. An employee can borrow the spirit of this in their own requests.

Ask for sources, then check them. OWASP advises asking for detailed reasoning and source citations, while validating the output with deterministic code [2]. For an employee, this means a manual check: open the cited source, reread the figure, verify that the quotation exists word for word. The GPT-3 authors themselves report datasets where the model still struggles [4].

Keep control of important actions. OWASP recommends human approval for privileged operations, to prevent unauthorized actions [2]. Do not let an assistant send a message or modify a file without review.

Do not paste sensitive data. OWASP lists the disclosure of sensitive information among the possible consequences of a prompt injection [2]. The UK's NCSC aims at AI systems that work without revealing sensitive data to unauthorized third parties [5]. At work, apply your organization's rule on authorized tools.

Keep external content separate from your instructions. When the assistant summarizes a web page or a file you received, that content may contain hidden instructions [2]. OWASP advises separating and clearly flagging untrusted content [2].

## Pitfalls

Invisible or confusing text. According to OWASP, a prompt injection does not need to be readable by a human, as long as the content is read by the model [2]. In a scenario it describes, a user has a web page summarized that contains hidden instructions: the model inserts an image linked to an address, and the private conversation is exfiltrated [2].

Meaningless strings of characters. OWASP describes an attacker who adds a seemingly absurd string of characters to a request. That string influences the answer and bypasses the safeguards [2]. Research provides an automated version: an adversarial suffix found with greedy and gradient-based search techniques [6].

Mixed languages and encodings. OWASP also cites the attacker who uses several languages or encodes instructions, for example in Base64 or with emojis, to evade filters [2].

Booby-trapped files. In another OWASP scenario, a résumé contains fragmented instructions. When a model evaluates the application, the pieces combine and produce a positive recommendation despite the actual content of the résumé [2]. In a third, a modified document in a repository used by a retrieval-augmented generation application skews the results [2].

Imperfect safeguards. OWASP states that retrieval-augmented generation and fine-tuning do not fully eliminate prompt injections [2]. It also considers it uncertain that a foolproof prevention method exists [2]. The NCSC points out that AI systems have new vulnerabilities, to be handled on top of classic threats [5]. NIST published a taxonomy of attacks on machine learning in January 2024, with a glossary aimed at non-expert readers [7].

## Real cases

### July 2023: a suffix that works on several assistants

On July 27, 2023, Andy Zou and his coauthors, including Nicholas Carlini, submit a method that automatically produces a suffix to append to a request [6]. This suffix aims to maximize the probability that the model answers affirmatively rather than refusing [6]. Trained on several requests and on two models (Vicuna-7B and 13B), it also triggers undesirable content in the public interfaces of ChatGPT, Bard and Claude, as well as in open models such as LLaMA-2-Chat, Pythia and Falcon [6].

## Sources

1. arXiv, "Neural Machine Translation of Rare Words with Subword Units", 31 August 2015. https://arxiv.org/abs/1508.07909 (accessed 3 November 2026)
2. OWASP, "LLM01:2025 Prompt Injection", 10 April 2024. https://genai.owasp.org/llmrisk/llm01-prompt-injection/ (accessed 3 November 2026)
3. arXiv, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing", 19 August 2018. https://arxiv.org/abs/1808.06226 (accessed 3 November 2026)
4. arXiv, "Language Models are Few-Shot Learners", 28 May 2020. https://arxiv.org/abs/2005.14165 (accessed 3 November 2026)
5. NCSC (UK), "Guidelines for secure AI system development", 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 3 November 2026)
6. arXiv, "Universal and Transferable Adversarial Attacks on Aligned Language Models", 27 July 2023. https://arxiv.org/abs/2307.15043 (accessed 3 November 2026)
7. NIST, "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations". https://csrc.nist.gov/pubs/ai/100/2/e2023/final (accessed 3 November 2026)

---

Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/llm-tokenization-explained/
You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.