Your tokenizer taxes Asian languages.Measure it in two minutes.

asia-fertility is an open-source benchmark that measures what LLM tokenizers really charge for 16 Asian languages: cost multiplier, effective context window and script-native retrieval. One pip install, one command, no API key.

Install the package GitHub

8.9×: median cost multiplier on cl100k_base across 11 non-Latin scripts
0–2 / 10: needle-in-a-haystack recall at 4k tokens for four of five frontier models
~1 billion: first-language speakers across the 16 languages benchmarked

Run it in two minutes

pip install "asia-fertility[oai]"
asia-fertility reproduce

Runs a bundled 10-sentence × 16-language reference suite offline in about three seconds, no API key, enough to verify the install matches the paper's methodology.

Measure your own corpus

asia-fertility measure --text "your text here" --tokenizer cl100k_base
asia-fertility cost --text "your text here" --model gpt-4o
asia-fertility tokenizers list
asia-fertility languages list

1. Cost ratio: the API bill

Median cost multiplier for the 11 non-Latin scripts on OpenAI's cl100k_base (GPT-3.5, GPT-4 Turbo): 8.9×. Burmese pays 11.66×, Lao 9.60×, Malayalam 8.94×, on the same FLORES-200 parallel content. The tax is largely a tokenizer choice, not a fine-tuning problem: switching Tamil to o200k_base (GPT-4o, GPT-5) cuts it from 7.61× to 1.98×, Burmese from 11.66× to 3.18×.

Cost ratio heatmap across 16 Asian languages and 10 tokenizers

2. Recall: can the model retrieve script-native content?

A 4,000-cell needle-in-a-haystack grid: 16 languages × 5 frontier models × 5 fill levels × 5 marker positions × 2 trials. Four of five models collapse to 0–2 out of 10 recall on non-Latin Asian languages already at 4k tokens, well inside every advertised context window. gemini-2.5-flash retains 84.5% pooled recall on the same grid: the collapse is a vendor-level training choice, not intrinsic to the task.

In-context capacity: recall degradation across scripts

3. Latency: does the cost penalty reach the user?

Pooled Pearson r between cost ratio and wall-clock ratio: 0.314 across 5 models × 16 languages × 10 trials. Modern serving stacks absorb most of the input-side penalty: a CFO sees 11× on Burmese, a user sees roughly 1.5× on average. Different problems, different mitigations.

What ships

MIT

Package

CLI + Python API, 10 tokenizers, 16 languages, 6 paper figures, deterministic offline reproduce. PyPI + GitHub.

github.com/Helmo21/asia-fertility

CC-BY-SA 4.0

Dataset

Three configs: leaderboard (160 rows), niah (4,000 rows), latency (1,040 rows). Every version ships a manifest.json.

huggingface.co/datasets/Helmo21/asia-fertility

CC-BY 4.0

Paper

15 pages, three benchmarks, three findings, 8 figures. Zenodo preprint with a permanent DOI.

doi.org/10.5281/zenodo.21069313

Why it matters

Any AI product priced against English-cost economics will underserve, deprioritise or quietly exclude native-language users at scale. The tokenizer decides the API bill, the effective context window and whether script-native retrieval works at all, and for years that decision has been made globally without measurement.

That makes the tokenizer a procurement decision. Different models ship very different ones. The measurement costs nothing and takes two minutes: if you build AI for Asian markets, benchmark before you deploy.

contact

Describe your situation in a few lines, no commitment. One conversation is enough to know whether there's a system worth building, and which door to take first. Or send me a message.