>a tokenizer that works everywhere

etok: a deterministic universal tokenizer

I train (very) small models. That means the tokenizer ships inside the product, on hardware somebody already owns. It needs to work for whatever language the person holding the phone actually speaks, and you don't get to route users by locale after the fact.

The normal way is to train a BPE on a large multilingual corpus. I didn't want that. A trained vocabulary is a result of whatever curriculum its fed. It moves every time I rebuild it, and it quietly bakes my sampling biases into every model downstream. If my corpus is 60% English, my tokenizer is too, and every non-English user pays for that forever.

So etok is a tokenizer thats generated deterministically. You give it a size, it builds a vocabulary from the structure of Unicode itself. Same in, same out, every time, no corpora, no trainer.

why I actually made it

The immediate reason was a tiny cognition core I'm building that has to do image generation, speech, decisions and language in one trunk, and will be reused by a bigger system model later. One tokenizer, many models. It had to not fall over on any script, and it had to stay small enough that the embedding table doesnt eat the model.

The constraint that shaped everything: it had to be a plain HuggingFace tokenizer.json. No custom model class, no patched PreTrainedTokenizer. It loads with AutoTokenizer.from_pretrained or Tokenizer.from_file and behaves like any other tokenizer, because anything else is a thing I'd have to maintain forever and a thing every downstream framework or user would have to special-case.

the measurement that changed the design

I built the first version, benchmarked it on twelve languages, and it looked fine. Then I wrote a sweep that runs every assigned Unicode codepoint through the tokenizer one at a time. All 149,878 of them.

The result was embarrassing. Every 4-byte UTF-8 codepoint cost exactly 4.000 tokens. Not on average. Exactly 4, for 100% of them. Every emoji, every CJK Ext B character, Egyptian hieroglyphs, cuneiform, Tangut, Adlam, Deseret, the mathematical alphanumerics. All of it was raw bytes.

Twelve hand-picked languages had hidden the fact that a huge fraction of Unicode wasn't tokenized at all. That's the whole lesson of this project: sample languages and you measure your sample. Sweep the codepoints and you measure the thing.

coverage before compression

The redesign inverts the usual priority. Instead of spending a small vocabulary on the most frequent text and letting the tail fall apart, etok spends a fixed block up front buying a hard cost ceiling for every codepoint, then spends what's left on making common text shorter.

That block is 3,533 tokens at an 8,192 vocabulary. 43% of the budget.

That bounds the cost of every scalar value in Unicode. Nothing is above 3 tokens. Nothing in the BMP is above 2.

Getting all 4-byte characters to 2 tokens instead of 3 would cost 20,480 prefixes, which is more than the entire vocabulary. So emoji and mathematical alphanumerics get promoted, because those are the 4-byte blocks that carry real text, and the historic scripts sit at 3. That's a deliberate trade, not an oversight.

the pre-tokenizer shortcut was the real issue

This one took me embarrassingly long to notice.

The GPT-2 regex pre-tokenizer i used splits text at combining marks. Which means for Devanagari, Thai, Tamil and basically every other syllabic script, that a consonant and its vowel sign can never be one token. The regex cuts between them before any merge gets a chance to apply. Those scripts were permanently stuck at one token per character instead of one per syllable, and the consonant-plus-matra pairs I'd put in the vocabulary were dead weight that could never fire.

You can see it directly:

tok.pre_tokenize_str("मा")
# [('म', (0, 1)), ('ा', (1, 2))]

Two pieces. No merge can join them.

So the default pre-tokenizer is now ByteLevel(use_regex=False). No splitting rules at all. Every byte is equal, and a merge may span any boundary the vocabulary defines. With that one change, मा goes from 2 tokens to 1.

It also made code and terminal output cheaper, because digit runs and punctuation stop being forced apart for no reason. GPT-2 mode is still there if you want compatibility, but it's not the default.

code, terminal and maths are not an afterthought

Most multilingual tokenizers treat programming as a rounding error. If I'm training a model that writes software, that's backwards. So code gets its own tier, placed before the general content interleave so it can never be crowded out.

It's a few hundred tokens of curated vocabulary: operators, keywords, the strings a developer actually types (" sudo", " pip", "--help", "./", "~/", ".py"), SQL keywords, hex and exponent literals, all 100 digit pairs, and the maths symbols that show up in real notation rather than the ones that merely exist in a block.

Measured on samples:

All 136 characters I consider critical are single tokens: brackets, operators, set and logic symbols, integrals, superscripts, matrix brackets, currency.

it has to be a generating set, not a lookup table

The thing I care about most, and the thing I got wrong at first.

You cannot cover every n-gram. There are infinitely many. What you can guarantee is that any n-gram is constructible from the tokens at bounded cost. The vocabulary is a generating set, not a dictionary of memorized fragments.

So the test doesn't check that specific strings are present. It generates random n-grams, 1 to 24 characters long, from each script's own inventory of letters, marks and digits, so syllables and clusters appear naturally. Fifty scripts, forty samples each. Then it checks three things: does it round-trip exactly, does it ever emit an id outside the vocabulary, and does the cost stay at or below the ceiling.

50 scripts x 40 random n-grams: round-trip failures 0, out-of-vocab 0
worst cost 2.00 tok/char, mixed-script worst 1.79

A tokenizer that only worked for the n-grams it had memorized would fail that immediately. This is also why "can it form complex responses" is a property of composition and not of vocabulary width. 8k composes anything.

what it costs

Honest numbers, because a small vocabulary has a price and pretending otherwise is how you get talked into a bad decision.

• 2.02 bytes per token on average across 70 languages, against 4 to 5 for a trained 128k BPE. So roughly 1.8 to 2x the token count for the same text.
• Median language parity 1.24x English. Worst case 3.09x, which is Tibetan.
• 17 living scripts reach one token per character at 8,192. The rest sit at the guaranteed 2-token ceiling.

On utilization I'm reporting Rényi efficiency rather than compression, because compression is a bad proxy. Zouhar et al. (2023) found compressed length correlates negatively with downstream BLEU (-0.32) while Rényi entropy efficiency correlates strongly (+0.78). A tokenizer that's only good at English compresses well and tokenizes badly. etok sits at 0.474, with the top single token holding 7.2% of emitted tokens and no script starved or dominant.

8,192 vs 11,264

I went back and forth on this, so here's the actual data rather than the vibe.

Going from 8,192 to 11,264 improves 10 of 70 languages. Sixty are unchanged. English, source code, terminal output and mathematics are bit-identical: 54 tokens, 1595, 557, 524, the same at both sizes.

The largest wins are Tibetan (-49.7%), Georgian (-46.2%), Sinhala (-44.4%), Tamil (-21.7%), Telugu (-12.6%). Real gains, but concentrated in a narrow set.

Even though it cost almost 5% of a core, and buys nothing on the workloads that dominate that model, I'm using 11,264. Especially with Japanese and Indic with roughly 500M speakers in the improved set.

what it isn't

It is not a competitive tokenizer on compression. A trained 128k BPE beats it, and will keep beating it. That's the trade for having no corpus, no trainer, and reproducible output.

The per-language parity numbers above come from one hand-written passage per language. That's enough to catch a script falling off a cliff. It is not a corpus, and it is not a context budget. Before I lock the size for real I'll run a few MB of actual target-language text through both presets and compare bytes per token.

Tibetan, Georgian and Sinhala sit at the 2-token ceiling at 8k by design. They're the smallest-speaker scripts in the priority list and the budget runs out before they're covered. The ceiling is the guarantee; full coverage is the bonus.

4-byte historic scripts stay at 3 tokens. Cuneiform is not worth 20,480 prefixes.

try it

pip install -e .

etok build -o ./my-tokenizer -v 8192 -d 0.15
etok coverage ./my-tokenizer

etok coverage is the interesting one. It runs the exhaustive codepoint sweep, the edge cases, the code and maths samples, the n-gram fuzzing and the parity table, and prints all of it. --json if you want the numbers.

It's a plain tokenizer.json, so:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("./my-tokenizer")

116 tests. Code at github.com/empero-org/etok, license is free for non-commercial use with visible credit to empero.

It's not the best tokenizer. It's one that works everywhere, costs the same everywhere, and tells you the truth about where it doesn't.