Train a byte pair encoding tokenizer in the browser, then watch it cut text up.
Tokens, from a vocabulary trained on this page
The merges that made them, in the order it learned them
Byte pair encoding, built in front of you. Give it text, watch it choose which pair of symbols to glue together next, and then use the tokenizer you just trained.
The tokenizer knows nothing except what is in this box. Everything it learns, it learns from here.
A chunk is what the pretokenizer produces before any merging happens: a regex splits the text on word, number, punctuation and whitespace boundaries, and merges are only ever allowed inside one chunk. That is why a token never straddles two words, and why the leading space is glued to the front of a word instead of standing alone.
One merge per step. Count every adjacent pair of symbols in the corpus, take the most common one, and replace it everywhere with a single new symbol. That is the whole algorithm. Repeat until the vocabulary is as big as you want it.
This round's candidates
The bar in colour is the pair that wins this round. Ties are broken by whichever pair the corpus scan met first, so a run is reproducible.
Merge table
| # | pair | token | count |
|---|
Tokens needed for a held out paragraph, as the vocabulary grows
The held out paragraph is not in the corpus, so this is compression on text the tokenizer has never seen. It drops fast and then flattens. That elbow is the real argument about vocabulary size: past it you are buying embedding rows and getting almost no tokens back.
Now use it. Type anything. Every chip is one token, and the colour is just a hash of the token so the same token always looks the same. Watch a capitalised word at the start of a sentence: the tokenizer learned ·the with its leading space and never met The, so it pays for that capital letter one byte at a time.
Click any chip above, or type a word here, to watch it climb from raw bytes to its final tokens. Each rung is one merge firing, and the label is that merge's rank.
The encoder never picks the longest token it can see. It picks the merge with the lowest rank that is present anywhere in the chunk, applies it everywhere in that chunk, and looks again. Rank order is learned order, so an early cheap merge can permanently block a later, longer one. Panel 5 is that fact with receipts.
Same vocabulary, two ways to cut a word up. Rank ordered merging is what this tokenizer does. Longest match is what almost everyone assumes it does.
Train first, then press the button.
Longest match walks left to right taking the biggest vocabulary entry that fits. Byte pair encoding replays its merges in the order it learned them. Both are using the identical vocabulary and they still cut some words in different places, which is the clearest possible proof that a tokenizer is its merge order, not just its word list.