What is a token in AI?

A token is the basic unit of text an AI model reads and writes — usually a word-piece, not a whole word. Models do not see letters or words directly; they convert text into tokens (numbers) first. A rough rule of thumb in English is that one token is about 4 characters, or roughly 0.75 words.

How many tokens are in a word?

On average about 1.3 tokens per English word, though it varies: common words are one token, while rarer or longer words split into several. As a quick estimate, 1,000 tokens is around 750 words. Other languages, code and numbers tokenise differently.

What is a context window?

The context window is the maximum number of tokens a model can consider at once — its working memory for a single request. If a 2026 model has a 200,000-token window, everything you send plus everything it replies must fit inside that budget, or the earliest tokens fall out of view.

Why do tokens matter for cost?

AI APIs bill per token, both for what you send (input) and what the model generates (output). Longer prompts and longer answers cost more and run slower, because the model does work proportional to the number of tokens. Understanding tokens is the first step to controlling AI cost.

What is BPE tokenization?

Byte-Pair Encoding (BPE) is the most common tokenisation method. It starts from characters and repeatedly merges the most frequent pairs into larger pieces, so common sequences become single tokens and rare words are still representable as smaller parts. It balances vocabulary size against sequence length.

AI Tokens & Tokenization Explained (2026)

AI tokens & tokenization: the unit behind every LLM

Before a language model can reason about text, it has to chop it into tokens. Get tokens right and you understand context windows, latency and cost — the three things that bite everyone.

Every large language model has a hidden first step that shapes everything after it: it does not read letters or words, it reads tokens. Understanding tokens is the fastest way to demystify three things that confuse most people — why models have a memory limit, why some prompts cost more, and why the model sometimes miscounts characters. This is a companion to our LLM fundamentals overview, going one level deeper.

From text to tokens

What a token actually is

A token is a chunk of text, usually a word-piece rather than a whole word. The word “training” might be one token; “tokenization” might split into “token” + “ization”. Each unique token maps to an integer ID, and it is those integers the model actually processes. A useful rule of thumb for English:

1 token ≈ 4 characters ≈ 0.75 words → 1,000 tokens ≈ 750 words

How tokenization works (BPE)

The dominant method, Byte-Pair Encoding, builds its vocabulary by starting from individual characters and repeatedly merging the most frequent adjacent pair. After enough merges, common sequences like “ing” or “the” become single tokens, while a rare word is still representable by stitching smaller pieces together. This is the trade-off at the heart of tokenisation: a bigger vocabulary means shorter sequences but more parameters to learn, and vice-versa.

The context window is a token budget

A model can only “see” a fixed number of tokens at once — its context window. Everything counts against it: your system prompt, the conversation history, any documents you paste, and the model's own reply. When you hear a 2026 model has a “200K context,” that is 200,000 tokens of working memory. Exceed it and the oldest tokens drop out of view, which is why long chats sometimes “forget” the start.

Why tokens = cost and speed

AI providers bill per token, for both input and output, and the model does computation roughly proportional to the number of tokens it handles. So a verbose prompt is a slower, pricier prompt. This is not a minor detail — for a business running thousands of AI calls a day, token discipline (tighter prompts, shorter outputs, caching) is the difference between an affordable system and a runaway bill.

The famous “how many r's in strawberry” problem

Because models see tokens, not letters, they are genuinely bad at character-level tasks like counting letters or reversing strings — the information is blurred inside a token. It is a neat illustration that a model's view of language is not our view. Once text is tokenised, it flows into the model's core machinery, the transformer.

Why Bahasa Malaysia and Chinese cost more tokens

Tokenizers are trained mostly on English, so English is the most “token-efficient” language at roughly 1.3 tokens per word. Bahasa Malaysia, and especially non-Latin scripts like Chinese and Tamil, fragment into many more pieces — the same sentence can use 1.5–2× the tokens. For Malaysian businesses this has two practical effects: multilingual prompts cost more per call, and they eat your context window faster, so less fits in the model's working memory.

A quick sense of scale

Rules of thumb worth memorising: a single-page email is a few hundred tokens; a 2,000-word document is about 2,700 tokens; a 100-page contract can exceed 60,000 tokens. Because you pay for both the tokens you send and the tokens the model writes back, long documents and long answers are where AI bills quietly grow — and why techniques like retrieval (sending only the relevant chunks) beat pasting everything into the prompt.