AI Tokenizer
See how language models read your text: every token, highlighted, with a live count.
How it works
- 1
Paste or type any text: a prompt, a note, code, any language.
- 2
Pick the tokenizer: o200k for current GPT models, cl100k for GPT-4 and GPT-3.5.
- 3
Read the count and the highlighted tokens; switch to token IDs to see what the model receives.
Runs in your browser. Nothing is uploaded.
Keep writing in HermesMarkdown
A local-first Markdown editor that runs in your browser: notes stay on your device, with tables, diagrams and AI when you want it. No account needed.
FAQ
What is a token?
A token is the unit a language model reads and writes: a whole word, part of a word, a space plus a word, a punctuation mark or a piece of an emoji. Models see your text as a list of token IDs, not letters.
How many tokens is a word?
In English, a word is about 1.3 tokens on average, or roughly 4 characters per token. Rare words, code, numbers and many non-English languages take more tokens per word, which is why this tool counts the real thing instead of estimating.
Which models does this tokenizer match?
o200k_base is the tokenizer of GPT-4o, GPT-4.1, GPT-5 and OpenAI's o-series models. cl100k_base is used by GPT-4 and GPT-3.5 Turbo. Other models, such as Claude, Gemini or Llama, use their own tokenizers, so their counts for the same text will differ.
Why do tokens matter?
Models are priced per token and have a context window measured in tokens. Knowing the count tells you what a prompt will cost, whether a document fits, and where wording changes save tokens.
Is my text uploaded?
No. The tokenizer runs in your browser, in a background worker. Your text is never sent to a server, and it's kept only in this browser tab.
Can I keep working on the text?
Yes. Open in HermesMarkdown carries the text into the editor as a draft, where you can keep writing and save it as a Markdown note.