BPE Token Counter & Context Visualizer
Exact token counts for the cl100k_base encoding (GPT-3.5/4-class), sub-word fragmentation view, and context-window usage — computed by the real BPE rank table in your browser.
Packages & Libraries: real Byte-Pair Encoding via js-tiktoken with the cl100k_base rank table (the encoding used by GPT-3.5/GPT-4-class models). Counts are exact for this encoding — other model families (Llama-3, Mistral) use different BPE vocabularies and will tokenize differently.
⚖️ Legal Notice: Provided "as is", without warranty. Token counts are exact for cl100k_base only; KV-cache VRAM figures are rough uniform-model approximations. All tokenization runs locally in your browser.
What this tool does
It counts tokens the way a real tokenizer does — not with a characters-divided-by-four estimate, but by running the actual cl100k_base byte-pair-encoding table (the vocabulary used by GPT-3.5- and GPT-4-class models) over your text with js-tiktoken in your browser. You get the exact token count, the character count, a rough KV-cache memory estimate, and a percentage of an 8k context window — plus the interesting part: every sub-word fragment your text breaks into, drawn as colored chips, so you can see why an unusual word or a line of code costs more tokens than plain English.
How to use it
- Paste your prompt, code, or document into the input.
- Read the stats — they update on every keystroke. The first visit loads the rank table (about 1.6 MB) once; after that your browser caches it and counts are instant.
- Use the fragment view to find expensive passages — long identifiers, unbroken strings, non-English text — and rewrite them where it matters.
Where your data goes
Tokenization runs in your browser. The rank table is fetched once from this site and cached; your text never leaves the tab, is never uploaded, and is never stored. Close the tab and it's gone.
Who it's for
Anyone budgeting a prompt before it hits a context limit; anyone comparing models and wondering why the "same" prompt consumes different token counts across them; and anyone building against a cl100k-class API who wants a local count before spending a request. It's also a decent teaching tool — the fragment chips make byte-pair encoding's behavior visible in a way a single number doesn't.
Honest limits
- Exact for cl100k_base only. Other model families — Llama-3, Mistral, Qwen, and OpenAI's newer o200k_base — use different vocabularies and tokenize the same text differently. Treat those cases as approximations, not predictions.
- The KV-cache figure is deliberately rough — a uniform bytes-per-token sketch, not your model's real cache. For actual VRAM planning use the VRAM calculator, which models layers and attention heads.
- 8k is a fixed reference window, not your model's context — rescale the percentage yourself for 4k, 16k, or 128k.
Questions people ask
Why does code cost more tokens than prose?
Byte-pair encoding merges frequent text patterns into single tokens; rare strings shatter into more pieces. Code mixes punctuation, camelCase identifiers, and symbols, so it fragments more than everyday language. The chip view makes each split visible, which is the fastest way to develop an intuition for it.
Will this match what OpenAI's API counts?
For models that use the cl100k_base encoding, yes — this page runs the real rank table, so the counts should agree byte-for-byte with any other correct implementation. OpenAI's newer models moved to a different encoding, and Llama- or Mistral-family models have vocabularies of their own; for those, treat the number here as a ballpark.
Back to the tools directory.