What are Tokens?

A token is how an AI system breaks down text into small, manageable pieces. It is one of the fundamental building blocks of how language AI works.

Why break text into pieces?

AI systems do not read words the way you do. They cannot process an entire sentence at once. Instead, they break language into smaller units called tokens.

This is a bit like how a cashier might handle money. Rather than looking at a £50 note as one unit, they break it into smaller denominations to count and process it.

What is a token?

A token is usually a word, or part of a word. Sometimes it is a single character or piece of punctuation.

The sentence "I love AI" might be broken into three tokens: "I", "love", and "AI".

A longer or more complex word might be broken into multiple tokens. For example, "unhappily" might become "un", "happy", and "ly" as separate tokens.

Why not just use whole words?

You might wonder why AI systems do not just process one word at a time. There are several reasons.

First, languages contain an enormous number of unique words and word forms. By breaking text into smaller tokens, the AI system has a more manageable set of patterns to work with.

Second, by using smaller tokens, the system can handle words it has never seen before. If it understands the tokens "un", "happy", and "ly", it can make sense of "unhappy" even if that exact word combination was not in its training data.

Tokens and costs

If you use a paid AI service, you are often charged by the number of tokens you use, not by the number of words.

This is important to understand because one sentence might use more or fewer tokens than you expect. A question like "What's the best way to cook pasta?" might use 9 tokens, not 8 words, because contractions and punctuation affect the count.

Context windows and tokens

AI systems have limits on how much text they can process at once. These limits are often described in terms of tokens, not words.

A system with a "4,000 token context window" can process up to 4,000 tokens in a single conversation. Since tokens are smaller than words, this can still be a substantial amount of text.

Why this matters

Understanding tokens helps explain why AI systems work the way they do.

It also explains why very long documents might need to be split into smaller pieces, and why some conversations reach a limit where the AI cannot remember earlier parts of the discussion.

Once text is tokenized, those tokens are converted into embeddings—numerical representations that capture the meaning and relationships between words—which is how AI systems actually understand language.

The simple explanation

A token is a small piece of text that an AI system uses to process language. It is usually a word or part of a word. By breaking language into tokens, AI systems can process text more efficiently and handle words they have never seen before. Tokens are also used to measure the cost of using AI services and to define the limits of how much text an AI can process at once.

To see how tokens fit into the complete picture of how AI works, read How Does AI Actually Work On The Inside?