minbpe vs turboBPE: Two ways to think about tokenizer training

If you have spent time understanding how LLMs process text, you have probably come across Byte Pair Encoding. It is the algorithm sitting quietly under the hood of GPT, Llama, Mistral, and most other major models, turning raw text into a sequence of tokens before anything else happens. The algorithm itself is elegant. Start with a vocabulary of all 256 individual bytes. Find the most frequently occurring adjacent pair of tokens in your training text, merge it into a new single token, and repeat until you hit your target vocabulary size. That is it. The vocabulary that comes out is entirely data-driven, efficient for the corpus it was trained on, and remarkably simple to implement. For anyone who wants to understand this from first principles, Andrej Karpathy's minbpe is the obvious starting point. The entire codebase is short enough to read in an afternoon, thoroughly commented, and covers everything you need: a basic BPE tokenizer, a regex-based variant that mirrors how GPT-2 and GPT-4 handle text preprocessing, and even a wrapper that reproduces GPT-4 tokenization exactly. You can train a tokenizer, encode text, decode tokens, save and load. It does what it says. The problem is not what minbpe does. It is what happens when you try to use it on anything real. Every round of BPE training requires a full pass over the corpus to count adjacent token pairs. You pick the top pair, merge it everywhere, and then sweep the entire corpus again. One merge per sweep. For a vocabulary of 10,000 tokens, that is roughly 9,744 sweeps. On minbpe's own test file, the Wikipedia article on Taylor Swift (about 182 KB), training a 10K vocab tokenizer takes around 782 seconds. Scale up to a 4 MB corpus and you are looking at about six hours. This is not a flaw in minbpe specifically — it is what faithful, pure-Python implementation of the naive algorithm looks like. For a learning exercise, this is completely fine. Nobody needs fast training to understand the algorithm. But if you actually want to train a tokenizer on your own data, this becomes a problem. turboBPE takes minbpe as its foundation and asks a simple question: why perform one merge per stat sweep when you could do ten? The idea, called batch merging, picks the top N candidate pairs per round (controlled by a batch_size parameter, default 10) and merges all of them in a single pass. One corpus sweep, ten merges. The number of expensive stat sweeps drops by roughly an order of magnitude. The catch is that you cannot blindly merge multiple pairs in the same round. If you try to merge both (A, B) and (B, C) simultaneously, you have a chain conflict: the first merge consumes the token that the second one depends on. turboBPE detects these overlaps automatically and drops the unsafe pair from the batch, deferring it to the next round. The first pair (which has higher frequency anyway, since the list is sorted) gets merged; the second waits. The safety check is built into the batch logic, not an afterthought. On top of this, the inner training and encoding loops run in a compiled C extension instead of pure Python. The hot path, iterating over potentially millions of token pairs thousands of times, benefits significantly from this. The benchmark numbers tell the story: the same 182 KB file that takes minbpe 782 seconds finishes in about 1.3 seconds with turboBPE. On 4 MB, six hours becomes roughly ten seconds. Encoding gets a similar treatment, with a linked-list structure for in-place merges replacing the repeated list copies in the naive approach. One deliberate choice in turboBPE is that the Python-facing API is nearly identical to minbpe. If you have read or written code against minbpe, switching is a one-line import change. from turbobpe import RegexTokenizer tokenizer = RegexTokenizer() tokenizer.train(very_long_training_string, vocab_size=32768) tokenizer.encode("hello world") tokenizer.decode([1000, 2000, 3000]) tokenizer.save("tok32k") tokenizer.load("tok32k.model") Special tokens work the same way. Register them after training with register_special_tokens, and use the allowed_special parameter to control whether they get parsed during encoding. The default behavior raises an error if a special token appears in input that has not explicitly permitted it, which is the right call. Silently tokenizing attacker-controlled input with special tokens is exactly the kind of subtle bug that causes real problems. If for some reason you need strict classical BPE merge ordering, you can set batch_size=1 and get the same per-merge behavior as minbpe, just with the C backend still helping on the stat counting. The parameter is a genuine dial between speed and ordering fidelity, not just a knob with one useful position. The speed difference is not just a benchmark curiosity. It changes what is actually feasible. Domain-specific tokenizers are a real need. Medical literature, legal documents, code in a particular language, multilingual corpora — the GPT-4 vocabulary was trained on general web text and it performs accordingly on specialized data. It may spend multiple tokens on subwords that appear constantly in your domain while single-tokening things that are rare. A tokenizer trained on your data is always going to be better matched to your task. With minbpe, that training is a one-time, multi-hour event on even a modest corpus. You pick your vocab size, wait, and commit to the result. With turboBPE, you can iterate. Train on a sample, look at the vocabulary, adjust your preprocessing, retrain. The feedback loop becomes tight enough that tokenizer design can actually be part of your workflow. The codebase also stays readable. The Python layer is a few hundred lines. You can audit it, extend it, modify the preprocessing, add a custom split pattern. This matters if you are working outside standard pipelines and need to understand exactly what is happening. minbpe is the right place to learn BPE. turboBPE is what you reach for when you actually want to use it.