LLM Tokens Explained: How AI Models Read Text, Count Costs and Fill Context Windows

An AI provider charges per million tokens. A model has a “200K context window”. A chatbot cannot count the letters in a word. All three facts make sense once you understand LLM tokens, the units a large language model actually reads and writes.

To show real numbers, we used tiktoken 0.14, the open-source tokenizer library from OpenAI, with its o200k_base and older cl100k_base encodings. Other providers, including Anthropic and Google, use their own tokenizers, so their counts will differ. The principles are the same.

What is a token?

A token is a piece of text from a fixed vocabulary that the model knows, typically containing tens or hundreds of thousands of entries. Common words are often a single token. Rarer words are split into several pieces. Punctuation, spaces and digits are tokens or parts of tokens too.

The model never sees letters. It sees a list of integer IDs. “Hello world” became the IDs [13225, 2375] in our test. Each ID is then turned into a vector of numbers, similar to the embeddings used for search, and the model works with those.

How tokenization works

Most modern models use a method based on byte pair encoding (BPE) or a close relative. During training of the tokenizer, the algorithm starts from single bytes and repeatedly merges the most frequent neighbouring pairs into new tokens. After many merges, common words and word pieces have their own tokens, while unusual strings fall back to smaller pieces. Because it works on bytes, any text, in any language or even binary-looking data, can be encoded.

Tokenizing is a separate step from the model. It happens before the text reaches the model and after the model produces output IDs, which are decoded back into text.

Real examples

'Hello world'      2  ['Hello', ' world']
' Hello'           1  [' Hello']
'tokenization'     2  ['token', 'ization']
'unbelievably'     3  ['un', 'bel', 'ievably']
'Stack Quora'      3  ['Stack', ' Qu', 'ora']
'2026-11-08'       6  ['202', '6', '-', '11', '-', '08']
'1234567'          3  ['123', '456', '7']
'def add(a, b):\n    return a + b'
                   11 ['def', ' add', '(a', ',', ' b', '):\n', '   ', ' return', ' a', ' +', ' b']

A few patterns stand out. The leading space is usually part of the token, so ” world” and “world” are different tokens. Long or rare words, and brand names like ours, are split into pieces. Numbers are broken into groups of up to three digits, which is one reason models can stumble on arithmetic with long numbers. Code is tokenized like any other text, including its indentation.

Rules of thumb

For ordinary English prose, a common estimate is about four characters, or three-quarters of a word, per token. We measured a 67-word paragraph about software testing:

words 67, tokens 74, chars 387
chars/token 5.23, tokens/word 1.1

Our text averaged slightly more characters per token than the usual estimate, since it used many common words. Treat the rule of thumb as a rough guide for planning, and count with the real tokenizer when accuracy matters, for example when checking that a prompt fits a limit. Most providers offer a token-counting tool or API.

Languages and formats cost differently

We tokenized the same refund message in several forms:

With the older tokenizer, the Hindi sentence needed 66 tokens, more than four times the English version, even though it has fewer characters. The newer tokenizer, with a larger vocabulary, brought it down to 16. This matters in practice: if a model’s tokenizer handles your language poorly, every request costs more, runs slower and fits less text into the context window. Test with your real content, in your users’ languages.

Structured formats have overhead too. JSON’s quotes, braces and keys all use tokens. In our sample, the JSON object needed 19 tokens to hold three short values.

Different models, different tokenizers

Each model family has its own tokenizer, and even one provider’s tokenizer can change between model generations, as the table above shows. So a prompt that is 1,000 tokens for one model may be 1,200 for another. When comparing prices between providers, compare the cost of your actual workload, not just the price per token.

Context windows

The context window is the maximum number of tokens a model can handle in one request. It includes everything: system instructions, the conversation history, any documents you include, tool definitions and the model’s answer. If the total exceeds the limit, the request fails or older content must be dropped.

Context windows have grown dramatically, and many current models accept hundreds of thousands of tokens or more. Check your model’s documentation, because limits change often. A bigger window is not the same as perfect recall, though. Research has found that models can retrieve information from the middle of very long inputs less reliably than from the beginning or end. For large document collections, retrieval-augmented generation (RAG) sends only the relevant passages, which is usually cheaper and more accurate.

Tokens and cost

AI APIs typically charge separately for input tokens and output tokens, with output usually priced higher. Long conversations grow expensive because the whole history is usually sent again with each new message. Many providers also offer prompt caching, which charges less for a repeated prefix such as long fixed instructions. Our model comparison shows how much the choice of model affects cost for the same task.

Quirks that tokens explain

• Counting letters: “strawberry” was split into ['st', 'raw', 'berry']. The model sees three chunks, not ten letters, so letter-level tasks are harder than they look. Newer models handle many of these better, but the cause is the same.
• Whitespace: “a b” was 2 tokens, while the same letters separated by six spaces, or by four line breaks, were 3. Messy formatting quietly adds cost.
• Dates and IDs: “2026-11-08” became 6 tokens. Long identifiers, hashes and URLs can be surprisingly expensive.
• Output limits: a “maximum tokens” setting cuts the answer off mid-sentence when reached, which is easy to mistake for a model failure.

Practical ways to use fewer tokens

1. Trim the prompt. Remove repeated instructions and boilerplate. Keep fixed instructions at the start so caching can help.
2. Send only relevant context. Retrieve the right passages instead of whole documents.
3. Summarise long histories instead of resending every past message.
4. Ask for concise output and set a sensible maximum. Output tokens usually cost the most.
5. Choose compact formats. Short field names and minimal whitespace help when sending structured data.
6. Use a smaller model for simple tasks such as classification.

Clear, well-structured prompts also tend to need fewer retries, as our prompt engineering guide explains.

Counting tokens in code

A token budget is easy to enforce once you can count. With tiktoken, counting and trimming take a few lines:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

def count_tokens(text):
    return len(enc.encode(text))

def truncate_to(text, max_tokens):
    return enc.decode(enc.encode(text)[:max_tokens])

doc = ("Refunds are paid within 7 days to the original payment method. "
       "Orders over 1000 rupees ship free.")
print(count_tokens(doc))      # 24
print(truncate_to(doc, 8))    # Refunds are paid within 7 days

Cutting by tokens rather than characters guarantees the result fits, but it can stop mid-sentence, as it did here. In real systems, trim at paragraph or sentence boundaries, and keep the most important content, such as instructions and the user’s question, outside the part you trim. Remember that this counts tokens for one tokenizer only; for other providers’ models, use their counting tools or the usage numbers their APIs return.

Test this logic like any other code. A unit test that feeds an over-long document and checks the result stays under budget will catch the day someone adds a large new instruction block to the prompt.

Final thoughts

Tokens are the currency of language models. They set what fits in a request, what it costs and how fast the answer arrives, and they explain several odd model behaviours. Measure token counts with your model’s own tokenizer, test with your real languages and formats, and treat prompt length as something worth optimising, just like the speed of any other part of your system.

Frequently asked questions

Keep learning