Tokenization & Byte-Pair Encoding (BPE)

"newest" split into individual characters — no merges applied yet

Start from raw characters and merge the single most frequent adjacent pair, over and over. Drag the slider forward and watch "newest" collapse from 7 individual characters down to just 3 pieces — each merge fusing together whatever pair showed up most often across the whole training vocabulary.

1. Byte-pair encoding, defined

   This algorithm is byte-pair encoding (BPE): starting from individual characters, repeatedly find the most frequent adjacent pair of symbols across the whole corpus and merge it into one new symbol, forming a growing vocabulary of subword pieces.

2. Trained on just four words

   Trained on a corpus of just four words — low, lower, newest, widest — it discovers pieces like est</w> (a common suffix) and low (a common stem), simply because those pairs of symbols kept showing up together.

Type any word. These merge rules were learned from only four training words, yet they generalize: type "lowest" — a word the algorithm never saw — and it still gets cut cleanly into low + est</w>, reusing pieces learned from completely different words ("low" from low/lower, "est</w>" from newest/widest). That's the entire point of subword tokenization: no word is ever truly unknown, because it can always fall back to pieces — down to individual characters if it has to.

1. Start from characters

   Tokenizing "lowest" from scratch: $l\ o\ w\ e\ s\ t\ \langle/w\rangle$.

2. Apply the learned merges in order

   1. $e{+}s\to es$
   2. $es{+}t\to est$
   3. $est{+}\langle/w\rangle \to est\langle/w\rangle$
   4. $l{+}o\to lo$
   5. $lo{+}w\to low$
3. Result

   Exactly two tokens: $low$ and $est\langle/w\rangle$. Every merge rule came from low, lower, newest, and widest alone; "lowest" itself was never part of training.