Web design · Development · SEO
Token (AI)
Understanding text units, context and usage
Editorially reviewed ·
Clear definition
A token is a small unit that an AI language model uses to process text. It may be a whole word, part of a word or punctuation. Spaces matter, too. A tokenizer splits the text according to defined rules and assigns numbers to those units: token IDs.
Imagine giving an AI a long document and asking for a summary. You see paragraphs and sentences. The model receives a sequence of tokens. Understanding how these units are counted makes limits on text length, context and usage much easier to interpret. You do not need to be a developer to follow the idea.
From text to four tokens
Take Hello, world!. The cl100k_base tokenizer produces four pieces: Hello, ,, world and !. Their IDs are 9906, 11, 1917 and 0. The space before world belongs to the third piece. Two words give us four tokens in this example.
This split was verified locally. It applies to this exact text and tokenizer, not to every AI system. Different spellings, languages and tokenizers can produce different results. Long words, code and special characters are good reasons not to rely on a fixed conversion from words. The code example below lets you reproduce the count.
At a glance
How the context window fits in
The context window describes how many tokens a model can handle in one request. Think of a limited work surface: your question sits alongside instructions, supplied documents and the conversation history being used. There must also be room for the generated response. Some models additionally use internal reasoning tokens.
Maximum output length is a separate limit. A large context window does not mean the answer can be equally long. Nor is it permanent memory. What an application stores, supplies again or summarizes from earlier conversations is a separate feature. If a chat loses track of details, it often helps to restate the essential requirements clearly.
Separating input, output and costs
Input tokens belong to the material the model receives. Output tokens are generated during the response. An API may charge different rates for these categories. Counting only the visible words tells you little about the bill.
Some providers also distinguish cached inputs and reasoning tokens. Prompt caching can reuse previously computed states for an unchanged prompt prefix; a matching cache entry may reduce input costs. OpenAI bills internal reasoning tokens as output even though you do not see their full text. Compare the reported usage categories with the current price list for the specific model.
A chat subscription is a different product from API billing. A token count does not automatically tell you how many messages your subscription allows.
How to check actual usage
A suitable tokenizer helps with a single piece of text. Counting that text alone often does not cover a complete request: message roles, tool descriptions and other structures can add tokens. Images and audio also follow their own model-dependent processing rules. Dividing the number of characters by four is not suitable for those inputs.
Where available, use the provider’s counting function for the full request, then inspect the returned usage data. OpenAI provides an input-token counting endpoint for this purpose. The local example in this entry counts only the supplied text; it does not represent a complete chat request or a future response.
Using this knowledge in everyday work
Comparing ten long documents? Start by stating which differences matter. Supply the relevant information and leave room for the response. For searches across your own content, a RAG system can select relevant passages instead of copying every document into every request. That selection must still match the question.
Shortening text is not an end in itself. Removing important conditions can make the task less clear. A useful token budget combines sufficient context with a clear request. Continue checking the substance and sources: a long, fluent LLM response can still be wrong.
Practical code example
Count the four tokens yourself
Original example using Python and the tiktoken package (install with python -m pip install tiktoken). No API key or model call is needed. Loading the encoding for the first time may download its public data file. The IDs and round trip were verified locally with cl100k_base.
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = "Hello, world!"
ids = encoding.encode(text)
print(ids) # [9906, 11, 1917, 0]
print(len(ids)) # 4
print(encoding.decode(ids)) # Hello, world!Sources and further reading
- Documentation Key concepts: Tokens OpenAI · Checked
- Documentation How to count tokens with tiktoken OpenAI · Checked
- Documentation Conversation state: Managing the context window OpenAI · Checked
- Documentation Counting tokens OpenAI · Checked
- Documentation Reasoning models OpenAI · Checked
- Documentation Prompt caching OpenAI · Checked
