Signum
Learn

What are tokens in AI? How language models read text and charge for it

By Elliot Marsh, Editor · · 4 min read

Read any AI price list or model announcement and you will see the word token: "$2 per million input tokens", "a 200,000-token context window", "a shared token budget". Tokens are the basic unit that language models read, write and bill by. Understanding them explains a surprising number of AI quirks.

The short version

A token is a chunk of text, often a word or part of a word, that a language model treats as a single unit. Before a model sees your text, a tokeniser splits it into tokens and turns each one into a number. The model works only with those numbers, and its output is a sequence of token numbers that are turned back into text.

Why not just use words or letters?

  • Letters would make sequences very long, which makes models slower and more expensive, and each letter carries little meaning.
  • Whole words would need an enormous vocabulary, and any word not in it (a new name, a typo, a word in another language) could not be represented.

Subword tokens are the compromise. Common words get their own token; rare words are built from smaller pieces. That keeps the vocabulary manageable, typically tens of thousands to a few hundred thousand tokens, while any text can still be represented.

How tokenisers are built

The most common approach is byte-pair encoding (BPE), adapted for language models in a 2015 paper by Sennrich, Haddow and Birch. The idea is simple:

  1. Start with individual characters (or bytes).
  2. Count which pair of adjacent units appears most often in a large body of text.
  3. Merge that pair into a new unit and add it to the vocabulary.
  4. Repeat until the vocabulary reaches the target size.

Frequent sequences such as "the", "ing" or " model" end up as single tokens. Google's SentencePiece library and OpenAI's tiktoken are widely used open-source implementations of these methods. Each model family has its own tokeniser, so the same sentence can be a different number of tokens for different models.

A worked example

Most English tokenisers treat the leading space as part of the token, so " cat" (with a space) and "cat" can be different tokens. A sentence like "Tokenisation is surprisingly fiddly." might split roughly as:

Token isation is surprisingly f iddly .

The exact split depends on the tokeniser. You can try your own text in OpenAI's free tokeniser tool.

Rules of thumb

For English prose with modern tokenisers:

  • 1 token is about 4 characters, or about three-quarters of a word;
  • 100 tokens is about 75 words;
  • a typical 1,500-word article is about 2,000 tokens.

These ratios get worse for other languages (particularly those not written in the Latin alphabet), for code with lots of symbols and for long numbers. The same content can cost noticeably more in some languages than in English.

Why tokens matter to you

Price

API pricing is per token, usually quoted per million, with separate prices for input (what you send) and output (what the model writes). Output is normally several times more expensive than input. To estimate a cost:

cost = input tokens x input price + output tokens x output price

For example, at $2 per million input tokens and $10 per million output tokens, a request with 10,000 tokens in and 1,000 out costs $0.02 + $0.01 = $0.03. Many providers discount cached input, meaning text repeated at the start of many requests.

Models that "think" before answering usually bill their reasoning as output tokens, even if you never see them. That is why reasoning modes cost more.

Limits

Context windows and output limits are measured in tokens, not words. Subscription plans increasingly use token budgets too, which is why heavier models or longer tasks use up your allowance faster.

Quirks

Tokenisation explains some famous failures:

  • Counting letters. A model asked how many "r"s are in "strawberry" sees a few tokens, not ten letters, so it has to infer spelling it never directly observes.
  • Arithmetic. Long numbers can be split into irregular chunks, which makes digit-by-digit maths harder.
  • Odd behaviour with rare strings. Unusual sequences may be split in unexpected ways, which can confuse the model.

Newer models handle these better, often by reasoning step by step or using a code tool, but the root cause is the same.

Tokens beyond text

Models that handle images, audio and video convert those into tokens too. An image might be cut into patches, each represented as one or more tokens, so a high-resolution image can use hundreds or thousands of tokens of your context window and budget.

The bottom line

Tokens are how language models see the world: small chunks of text turned into numbers. They decide what a request costs, how much fits in a context window and why models sometimes stumble over spelling or sums. When an AI price or limit is quoted in tokens, a quick rule of three-quarters of a word per token gets you close enough to plan.

Sources