What is a Token? - AI Glossary

A token is essentially a snippet of text that serves as a basic unit for the model to work with.

Think of them as the AI’s vocabulary building blocks. These aren’t always whole words; they can be parts of words, punctuation marks, or even just spaces. The process of converting raw text into these tokens is known as tokenization.

Why Text Needs to be Broken Down

When we humans read, we effortlessly process sentences, recognizing words and their meanings based on context, grammar, and our vast accumulated knowledge. Our brains are incredibly adept at handling the fluidity and complexity of language.

AI models, however, need a more structured approach. They work with numerical data. To process text, the words, sentences, and paragraphs we understand need to be converted into a format that the AI can understand and perform calculations on – typically numbers (vectors or embeddings, a topic for another day, but know they are derived from tokens).

Imagine trying to teach a computer to read by just giving it a giant string of letters. It wouldn’t know where one word ends and another begins, or how punctuation affects meaning. It needs discrete units.

This is where tokenization comes in. It’s the essential first step where raw text is chopped up into a sequence of tokens. Each unique token is then assigned a unique identification number (an ID). The AI model then learns to process sequences of these token IDs, understanding the relationships between them to comprehend and generate language.

Without tokenization, text would just be an unbroken stream of characters, impossible for current AI architectures to process efficiently or effectively for language tasks.

What Is a Token, Really?

So, we know a token is a basic unit of text for AI. But what does that actually look like? It’s often surprising to beginners that a token isn’t always a single, neat word.

Let’s look at some examples using a hypothetical tokenizer similar to those used by modern LLMs:

Consider the sentence: “Hello there, how are you?”

A simple tokenizer might break this into: ["Hello", " ", "there", ",", " ", "how", " ", "are", " ", "you", "?"]

Here, spaces and punctuation are treated as separate tokens. This might seem odd to us, but it helps the model understand the structure and rhythm of language, and how punctuation affects meaning.

Now consider a more complex sentence: “Tokenization is fascinating!”

A modern subword tokenizer (we’ll discuss these soon) might break this differently: ["Token", "ization", " is", " fascin", "ating", "!"]

Notice how “Tokenization” was split into “Token” and “ization”. “Fascinating” was split into “fascin” and “ating”. Also, ” is” starts with a space. This is very common in many tokenizers – they include a space at the beginning of words that follow a space, helping the model reconstruct the original text accurately.

Key Takeaways about what tokens can be:

Whole Words: Many common words like “the”, “a”, “is”, “cat” will be single tokens.
Parts of Words (Subwords): Longer or less common words might be split into smaller, frequently appearing subword units (like “ing”, “ed”, “pre”, “ation”). This is crucial for handling a vast vocabulary without having a token for every single possible word, including rare or newly invented ones.
Punctuation: Commas, periods, question marks, exclamation points, etc., are almost always separate tokens.
Symbols and Emojis: These are also typically treated as individual tokens.
Spaces: As seen, spaces can be tokens themselves or be prepended to the following word token.

The total collection of unique tokens that a specific AI model’s tokenizer knows is called its vocabulary. This vocabulary can be huge – often tens of thousands or even hundreds of thousands of unique tokens.

Tokenization: How Text Becomes Tokens

Tokenization isn’t a single, simple process. Over time, researchers have developed different strategies to convert text into tokens, each with its own strengths and weaknesses.

Let’s look at the main types:

Word-Based Tokenization:
- Idea: Simply split text based on spaces and sometimes punctuation.
- Example: “Hello world!” -> ["Hello", "world", "!"]
- Pros: Conceptually simple.
- Cons:
  - Doesn’t handle variations well (e.g., “run”, “running”, “ran” might all be different tokens).
  - Struggles with punctuation attached to words (“hello,” vs. “hello”).
  - Leads to a massive vocabulary because every unique word form needs its own token.
  - Has trouble with unknown words (out-of-vocabulary words) – words the model has never seen during training.
Character-Based Tokenization:
- Idea: Treat each character (letter, number, symbol) as a token.
- Example: “cat” -> ["c", "a", "t"]
- Pros: Very small vocabulary (just the set of possible characters). Excellent at handling any text, including misspellings and rare words, as it can spell them out character by character.
- Cons:
  - Sequences of tokens become very long (e.g., a long word is many tokens).
  - Models have to work harder to learn the meaning of words because they only see characters, not word-level units. This can make learning long-range dependencies difficult.
Subword Tokenization (The Modern Standard):
- Idea: Find a balance between word-based and character-based. Split text into tokens that are often whole words for common words, but use smaller, frequently occurring subword units for rarer or longer words.
- Example: We saw this with “Tokenization is fascinating!” -> ["Token", "ization", " is", " fascin", "ating", "!"]
- Pros:
  - Manages a reasonable vocabulary size – much smaller than word-based, larger than character-based.
  - Can handle unseen words by breaking them down into known subword units (e.g., “unbelievable” might become “un”, “believ”, “able”).
  - Generally works very well for training large language models on vast amounts of text.
- Common Methods:
  - Byte Pair Encoding (BPE): Originally used for data compression, adapted for text. It works by starting with individual characters and iteratively merging the most frequent pairs of characters (or character sequences) into new tokens until a desired vocabulary size is reached or no more merges are beneficial.
  - WordPiece: Used by models like BERT and many others. Similar to BPE but merges based on the probability of the merge forming a word.
  - SentencePiece: Developed by Google, used by models like T5 and LaMDA. It treats the input as a raw stream of characters (including spaces) and learns tokens that allow it to reconstruct the original input exactly, including whitespace. It’s language-agnostic.

Most modern, powerful LLMs (like those behind ChatGPT, Bard/Gemini, Claude) rely heavily on subword tokenization, particularly variants of BPE or SentencePiece. This allows them to have large vocabularies that cover common words efficiently while still being able to represent virtually any word or character sequence they encounter.

The specific tokenization algorithm and the size of the token vocabulary are design choices made by the model creators, often based on research and experimentation to achieve the best balance of performance, efficiency, and handling of diverse text.

Why Tokens Matter So Much in AI

Now that we know what tokens are and how they are created, let’s delve into why they are so fundamental to how LLMs work and interact with us. Their importance cannot be overstated, influencing everything from the model’s understanding to its limitations and even its cost.

The Basis for Understanding and Generating Text: At their core, LLMs are statistical models that learn the probability distribution of token sequences based on the massive amounts of text they are trained on. When you give a model an input prompt (a sequence of tokens), it uses its training to predict the most likely next token to follow that sequence. It does this repeatedly, one token at a time, to generate its response. Think of it like a highly sophisticated autocomplete that generates not just the next word, but potentially entire paragraphs or pages, token by token. The model has learned complex patterns, grammar, facts, and even reasoning abilities by seeing how tokens relate to each other in its training data. Every output from an LLM is built token by token. This is why you sometimes see the text appearing word by word or even subword by subword when an AI is generating a response – it’s literally predicting and outputting the tokens one after the other.
Defining the Context Window (The Model’s “Memory”): This is one of the most critical roles of tokens. AI models, particularly transformers (the architecture behind most modern LLMs), have a limited capacity to attend to or “remember” the input text and the conversation history. This capacity is measured in tokens. This limit is known as the context window. If a model has a context window of 8,000 tokens, it means that for any given prediction it makes (generating the next token), it can effectively look back at and consider the information contained in the last 8,000 tokens of the conversation or input text. Why is this important?
- Understanding Long Conversations: A larger context window allows the model to remember more of what was said earlier in a conversation, leading to more coherent and contextually relevant responses over longer interactions.
- Processing Long Documents: If you ask an AI to summarize a document, its ability to do so depends on whether the entire document (once tokenized) fits within its context window. If it doesn’t, the model might only be able to process the beginning or end of the document, potentially missing crucial information in the middle.
- Maintaining Narrative Cohesion: For creative writing tasks, a larger context window helps the model keep track of characters, plot points, and settings established earlier in the story, maintaining consistency.

Research efforts are continuously pushing the boundaries of context window sizes. Models have evolved from having context windows of a few thousand tokens to tens of thousands (e.g., 32k, 100k, or even over 1 million tokens in some experimental setups), dramatically increasing their ability to handle and reason over large amounts of information. This progress is often reported alongside model updates, highlighting the token limit as a key capability metric.

Influencing the Cost of AI Services: For users accessing powerful LLMs through APIs (Application Programming Interfaces), the cost is frequently calculated based on the number of tokens processed. This typically includes both the input tokens (your prompt) and the output tokens (the AI’s response). Providers like OpenAI, Anthropic, and Google Cloud AI Platform often publish their pricing tiers based on tokens. For example, a model might cost $0.001 per 1,000 input tokens and $0.002 per 1,000 output tokens. These numbers are hypothetical but illustrate the concept. Why charge by tokens? Processing tokens is computationally expensive. The larger the context window and the longer the input/output sequences, the more processing power (and thus cost) is required. Billing by tokens provides a granular way to measure the computational resources consumed by each user query.

Understanding token costs can help users write more concise prompts and manage the length of AI-generated responses to optimize their usage expenses. For developers building applications on top of these models, careful management of token usage is crucial for cost control.

Impact on Model Efficiency and Performance: The choice of tokenization and the vocabulary size directly impact how efficiently a model can be trained and how well it performs on specific tasks.
- Efficiency: A smaller vocabulary requires less memory and computational power for the model to process each token. However, if tokens are too small (like single characters), sequences become very long, increasing the computational burden. Subword tokenization aims for a sweet spot.
- Performance: An effective tokenization method ensures that semantically related pieces of words or common phrases are represented efficiently. Poor tokenization can split meaningful units, making it harder for the model to learn relationships. Researchers analyze token distributions and their impact on model accuracy for various language understanding and generation tasks.
For instance, if a tokenizer consistently splits a common technical term into multiple unrelated subwords, the model might struggle to recognize and understand that term compared to a tokenizer that treats it as a single token or a small, consistent sequence of tokens. Research papers often discuss how improvements in tokenization contributed to performance gains in new model architectures.

Tokens in Action: A Deeper Example

Let’s take the sentence: “Artificial intelligence is transforming the world.”

How might a modern subword tokenizer break this down?

Original Sentence: Artificial intelligence is transforming the world.

Possible Tokenization: ["Artificial", " intelligence", " is", " trans", "forming", " the", " world", "."]

“Artificial” and “intelligence” might be whole tokens as they are common.
” is” starts with a space.
“transforming” is split into “trans” and “forming”. This is efficient because “trans” and “forming” are common subword units appearing in many other words (“translate”, “transport”, “information”, “performer”).
” the” and ” world” are common words, again starting with spaces.
“.” is a separate token.

Counting the tokens: In this hypothetical example, the sentence is broken into 8 tokens.

If this sentence was part of a longer document or conversation, these 8 tokens would contribute to the overall token count that the AI model processes within its context window. If the context window is, say, 4096 tokens, these 8 tokens take up a tiny fraction of that capacity. However, if you were processing a book chapter that tokenized to 10,000 tokens, and your model has a 4096 token limit, it wouldn’t be able to see the whole chapter at once.

This breakdown illustrates how the AI doesn’t see a continuous string of characters or even neat dictionary words. It sees a sequence of these specific token units and learns patterns between these units. When asked to generate text, it predicts the next most probable token based on the sequence it has seen so far.

Research and Statistics: Tokens in the Real AI World

The concept of tokens and tokenization is central to the vast majority of research and development in Large Language Models. While finding hard, universally agreed-upon statistics solely about tokens (like “the average number of tokens per word globally”) can be difficult due to varying languages and tokenizers, we can discuss research trends and model capabilities directly related to tokens:

Model Scale and Token Limits: A key area of progress is increasing the context window size, directly measured in tokens.
- Early transformer models might have had limits around 512 tokens.
- Models like GPT-3 had limits of 2048 or 4096 tokens.
- More recent models have pushed these limits significantly. For example, some versions of GPT-4 offer 8k and 32k token contexts.. Anthropic’s Claude models have been notable for their large context windows, with some versions offering 100k tokens. Google’s Gemini models also feature large context windows, with variations offering up to 1 million tokens in research or specific versions. These figures highlight the rapid progress in enabling AIs to process much longer inputs.
Efficiency of Tokenization: Researchers constantly analyze the efficiency of different tokenization schemes. A common metric is the average number of tokens per word for a given language and tokenizer. For English, using common subword tokenizers, this is often cited as roughly 1.3 tokens per word. This isn’t a strict rule, as it varies greatly depending on the complexity of the text and the specific tokenizer used, but it gives a sense that words are often broken into slightly more than one token on average. Research looks at how to minimize this ratio while retaining the ability to represent all words, improving processing speed, and reducing context window usage for the same amount of text.
Handling Out-of-Vocabulary (OOV) Tokens: Research focuses on how well tokenizers and models handle words not explicitly in their vocabulary. Subword tokenization is a solution, but models still need to learn to compose meaning from these subword sequences. Papers explore alternative methods or ways to improve OOV handling, crucial for domains with evolving language or many proper nouns.
Multimodal Tokens: In cutting-edge AI, tokens are no longer limited to text. Researchers are developing ways to represent other types of data, like images, audio, or video, as sequences of “tokens” that can be processed by similar transformer architectures. This is how multimodal models (like some versions of Gemini or GPT-4’s vision capabilities) work – they convert different inputs into a common “token” language the model understands. Research in this area involves determining the optimal way to break down non-text data into meaningful token-like units.
Tokenization Bias: An emerging area of research looks at potential biases introduced by tokenization. Some tokenization schemes might disproportionately break down words related to specific minority groups or sensitive topics into less common or less meaningful subwords compared to majority-group terms. This could potentially impact how well models understand and generate text about these topics, a subtle form of bias being actively studied in the AI ethics community.

These research areas demonstrate that tokens are not just a simple technical detail but a fundamental aspect influencing the capabilities, efficiency, and even fairness of AI language models.

Challenges and the Future of Tokens

While subword tokenization has been a major leap forward, it’s not without its challenges:

Arbitrary Splits: Sometimes, a tokenizer might split a meaningful word or phrase in a way that seems arbitrary to humans, potentially making it harder for the model to fully grasp the intended meaning.
Increased Sequence Length: Even with subwords, representing text still requires more tokens than the number of words (remember the ~1.3 tokens per word average). This contributes to the context window limitation.
Handling Specific Languages: Tokenization needs to be adapted or carefully chosen for different languages, especially those without clear word boundaries or with complex morphology (word structure).

The future of tokens and tokenization in AI research is exciting and dynamic:

More Intelligent Tokenization: Could tokenization become more semantic? Instead of purely statistical merging, could tokens represent concepts or meaningful phrases directly?
Beyond Fixed Vocabularies: Research into methods that don’t rely on a fixed, predefined vocabulary of tokens could lead to models that are even better at handling novel language.
Infinite Context? While true “infinite” context might be impossible, researchers are exploring architectural changes and new techniques to allow models to process and recall information from extremely long inputs, potentially reducing the hard constraints imposed by current token limits. Techniques like “retrieval-augmented generation” (where the AI can look up relevant information from a large database based on the prompt) are one way to work around strict token limits.

Conclusion

While invisible to the end-user most of the time, tokens are the invisible engine driving the AI’s ability to communicate with us.

As AI continues to evolve, our understanding of tokens and how they are processed will deepen, leading to even more powerful, efficient, and capable language models.