Tokens are the small pieces of text that an AI such as ChatGPT or Claude splits every text into before it works with it: whole words, parts of words, punctuation and spaces. They have nothing to do with crypto tokens, NFTs or the security token for your online banking. In AI, a token is simply the unit in which a language model reads, writes and counts. It decides how much text fits into a conversation and how long an answer can get.
I work with Claude every day. My fourteen AI employees run on Claude Team, some also on open models or through the API. Tokens come up for me all the time, even though you never see them in the chat window. In this article I explain them so you can follow along without any background. Everything on AI basics is collected on my AI Basics page.
What a token is in AI
A language model can't calculate with letters directly. So a program called a tokenizer first splits every text into pieces the model knows. Each piece gets a number, and the model works with those numbers.
OpenAI describes tokens as commonly occurring sequences of characters. A short, common word like "the" is a single token, while a longer word is split into several parts. Anthropic, the company behind Claude, writes in its glossary that a token can correspond to a word, part of a word, a single character or even a byte, the smallest unit of computer memory.
Here is what it looks like when you send the German word "Kundenzufriedenheitsumfrage" (customer satisfaction survey) through OpenAI's open tokenizer (tiktoken, encoding o200k_base):
K · und · enz · uf · rieden · heits · um · frage
One word, eight tokens. The English "customer satisfaction survey", by contrast, is exactly three tokens, one per word.
How tokens are counted
The vendors give rules of thumb, and all of them are approximate:
- OpenAI: For English text, one token is roughly 4 characters or three quarters of a word.
- Anthropic: On older Claude models, one token stands for about 3.5 English characters. Since Claude Opus 4.7, Claude uses a new tokenizer that counts roughly 30 percent more tokens for the same text. According to Anthropic, one million tokens are about 555,000 words on it.
Every vendor has its own tokenizers, and they often change with new models. The same sentence can therefore give a different number in ChatGPT than in Claude, and on a new Claude model a different one than on its predecessor. Spaces and punctuation count too. Images become tokens as well: according to Anthropic, Claude splits an image into patches of 28 by 28 pixels, and each patch counts as one token.
Why German needs more tokens than English
Common English words are often a single token in these tokenizers, as OpenAI shows with "the". Words in other languages, such as long German compounds like the one above, often split into several pieces. Anthropic only says in general that the characters per token depend on the model and the language.
How big the difference can be shows in an example, counted with the same OpenAI tokenizer as above:
Almost the same number of characters, but the German sentence needs nearly half as many tokens again. "Teambesprechung" (team meeting) alone is five. This is one example with one tokenizer, not an average. Claude can give different numbers. The vendors' rules of thumb are for English. For German text, this example suggests you should expect more tokens; for other languages, count instead of estimating.
Why tokens matter for length and context
In the chat you see words, the AI counts in tokens. That shows up in three places:
- How much fits into a conversation. Everything the AI has in front of it at one moment is called context, and the space for it is called the context window. Its size is given in tokens. According to Anthropic, everything counts toward it: your messages, the answers, attached files and images, the instructions in the background, and the new answer itself.
- How long an answer can get. A single answer also has a maximum length, and that is measured in tokens too.
- How well long conversations hold up. More context isn't automatically better. Anthropic writes that accuracy and recall degrade as the token count grows.
If you use Claude through its programming interface rather than the chat, usage is also billed in tokens. How that works for Claude is in my article on the Claude API. What context is exactly and what happens when the window is full, I explain in detail in Context and Context Windows in AI.
An example from the office
An example: you upload a long lease agreement as a PDF into the chat, ask ten questions about it and finally ask for a summary. Every page of the contract, every question and every answer takes up tokens in the context window. With a large window, that fits easily. But if you keep the same chat going for hours and attach three more documents, it gets tight. According to Anthropic, chat interfaces such as claude.ai can then manage the context themselves, for example by letting older parts drop out first. For you, that means: what was at the very start may no longer be as present at the end as you think.
My advice: start a new chat for a new task instead of collecting everything in one long conversation.
What to watch out for
- Rules of thumb are estimates. For exact numbers, the vendors offer counting tools: OpenAI has the Tokenizer on its developer site, Anthropic a counting function in its API.
- German counts differently from English. For German text, expect more tokens than the English rules of thumb suggest; for other languages, count instead of estimating.
- New models count differently. Switching models can make the same request larger, as with Claude since Opus 4.7.
- Attachments count. A PDF or a screenshot often takes up more space than your actual question.
- Tokens are not words. If you ask for a certain length, give it in words or paragraphs. You don't see the token count in a normal chat anyway.
Frequently asked questions
What is the difference between AI tokens and crypto tokens?
They only share the name. A crypto token is a digital asset on a blockchain. An AI token is a piece of text, for example part of a word, that a language model calculates with.
How many words are 1,000 tokens?
By OpenAI's rule of thumb, 1,000 tokens of English text are about 750 words. For German text it is often fewer words, because German words more often split into several tokens. There is no fixed conversion; it depends on the model and the text.
What is a tokenizer?
A tokenizer is the program that splits text into tokens and gives each piece a number. Every vendor has its own, and it often changes with new models. OpenAI publishes its tokenizer openly under the name tiktoken.
Do images and PDFs count as tokens too?
Yes. According to Anthropic, Claude splits an image into patches of 28 by 28 pixels, and each patch counts as one token. A large image therefore takes up more space than a small one, up to a maximum size above which Claude scales the image down first. PDFs and other files count toward the context as well.
Why don't I see tokens in the chat?
Because the chat interface shows you text and the splitting happens in the background. Anthropic writes that tokens are typically hidden when you work with text and only become relevant when you look at the exact inputs and outputs of a model. In Claude Code, for example, a Claude Code Statusline can show you how full the context window is.
Where to go from here
If you don't know yet what happens to the tokens inside the AI, first read how an LLM works. The next step is the context window: how many tokens fit in and what happens when it is full. Which Claude models exist and how large their context windows are is in Claude Models: Opus, Sonnet, Haiku, Fable. If you want to run AI models on your own hardware, you count in tokens there too; more on that in Local AI: What Actually Works in 2026. You'll find all the other technical terms in my glossary of AI terms.
Your first AI employee up and running within an hour
Join the community$5/month founding price, first 50 members
