Multilingual Tokenization Showdown 
text
Multilingual Tokenization Showdown 
new_line
Analyzing 12 LLM Tokenizers Across 204 Languages.
text
Analyzing 12 LLM Tokenizers Across 204 Languages.
new_line
new_line
First, I've created a dataset with Wikipedia's  "Cat" article text in 272 languages: 
text
First, I've created a dataset with Wikipedia's  "Cat" article text in 272 languages: 
new_line
https://huggingface.co/datasets/Norod78/WikiCat-Multilingual
Norod78/WikiCat-Multilingual
dataset
resource
https://huggingface.co/datasets/Norod78/WikiCat-Multilingual
new_line
new_line
For each language entry with at least 100 words, I tokenized the text using 12 tokenizers and calculated the "Characters per token" ratio and "Word per token" ratio. The higher this ratio is, the more information each token represents on average for that language (and perhaps allowing the llm to potentially learn more per-parameter if trained on a dataset of that language).     
text
For each language entry with at least 100 words, I tokenized the text using 12 tokenizers and calculated the "Characters per token" ratio and "Word per token" ratio. The higher this ratio is, the more information each token represents on average for that language (and perhaps allowing the llm to potentially learn more per-parameter if trained on a dataset of that language).     
new_line
new_line
You can see a slideshow summary of the results here:
text
You can see a slideshow summary of the results here:
new_line
https://norod.github.io/wikicat-tokenizer-eval/tokenizer-slideshow.html
https://norod.github.io/wikicat-tokenizer-eval/tokenizer-slideshow.html
link
new_line
new_line
I hope I interpreted the results correctly, I've made the code available on GitHub so you can re-create the raw results jsonl with this repo:
text
I hope I interpreted the results correctly, I've made the code available on GitHub so you can re-create the raw results jsonl with this repo:
new_line
https://github.com/Norod/wikicat-tokenizer-eval
https://github.com/Norod/wikicat-tokenizer-eval
link
new_line
new_line
Post on X:
text
Post on X:
new_line
https://x.com/Norod78/status/1984366900550266999
https://x.com/Norod78/status/1984366900550266999
link
new_line
new_line
Multilingual Tokenization Showdown 
Analyzing 12 LLM Tokenizers Across 204 Languages.

First, I've created a dataset with Wikipedia's  "Cat" article text in 272 languages: 
https://huggingface.co/datasets/Norod78/WikiCat-Multilingual

For each language entry with at least 100 words, I tokenized the text using 12 tokenizers and calculated the "Characters per token" ratio and "Word per token" ratio. The higher this ratio is, the more information each token represents on average for that language (and perhaps allowing the llm to potentially learn more per-parameter if trained on a dataset of that language).     

You can see a slideshow summary of the results here:
https://norod.github.io/wikicat-tokenizer-eval/tokenizer-slideshow.html

I hope I interpreted the results correctly, I've made the code available on GitHub so you can re-create the raw results jsonl with this repo:
https://github.com/Norod/wikicat-tokenizer-eval

Post on X:
https://x.com/Norod78/status/1984366900550266999


👍
John6666 lucadiliello j14i yjoonjang tarekmasryo
🔥
umair894 lucadiliello DeathGodlike