After spending some time practicing tokenization, I have come to realize that the difficulties we face in understanding each other are analogous to the challenges faced by LLMs in processing and interpreting tokens - as in untrained tokens lead to out of distribution qualms.
text
After spending some time practicing tokenization, I have come to realize that the difficulties we face in understanding each other are analogous to the challenges faced by LLMs in processing and interpreting tokens - as in untrained tokens lead to out of distribution qualms.
new_line
new_line
One could think of how we understand as a process that involves trained tokens (known/learned facts) grappling with prompts/tweets/lessons from someone else. This process is distinct for each person - with unique encoding, decoding, merging and splitting patterns. 
text
One could think of how we understand as a process that involves trained tokens (known/learned facts) grappling with prompts/tweets/lessons from someone else. This process is distinct for each person - with unique encoding, decoding, merging and splitting patterns. 
new_line
new_line
This distinction might as well be categorized in gpt levels lol, which brings the question what level of tokenizer are you?  GPT-2, GPT-3, GPT-4 or GPT-4o tokenizer:)
text
This distinction might as well be categorized in gpt levels lol, which brings the question what level of tokenizer are you?  GPT-2, GPT-3, GPT-4 or GPT-4o tokenizer:)
new_line
new_line
Papers:
text
Papers:
new_line
- Neural Machine Translation of Rare Words with Subword Units (
text
- Neural Machine Translation of Rare Words with Subword Units (
https://arxiv.org/abs/1508.07909
https://arxiv.org/abs/1508.07909
link
)
text
)
new_line
- Learning to Compress Prompts with Gist Tokens
text
- Learning to Compress Prompts with Gist Tokens
new_line
(
text
(
https://arxiv.org/abs/2304.08467
https://arxiv.org/abs/2304.08467
link
)
text
)
new_line
- Language Models are Few-Shot Learners
text
- Language Models are Few-Shot Learners
new_line
(
text
(
https://arxiv.org/abs/2005.14165
https://arxiv.org/abs/2005.14165
link
)
text
)
new_line
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
text
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
new_line
(
text
(
https://arxiv.org/abs/2405.05417
https://arxiv.org/abs/2405.05417
link
)
text
)
new_line
- Language Models are Unsupervised Multitask Learners 
text
- Language Models are Unsupervised Multitask Learners 
new_line
new_line
Code: 
text
Code: 
new_line
https://github.com/karpathy/minbpe
https://github.com/karpathy/minbpe
link
new_line
https://github.com/openai/tiktoken
https://github.com/openai/tiktoken
link
new_line
https://github.com/openai/gpt-2
https://github.com/openai/gpt-2
link
new_line
https://github.com/google/sentencepiece
https://github.com/google/sentencepiece
link
new_line
After spending some time practicing tokenization, I have come to realize that the difficulties we face in understanding each other are analogous to the challenges faced by LLMs in processing and interpreting tokens - as in untrained tokens lead to out of distribution qualms.

One could think of how we understand as a process that involves trained tokens (known/learned facts) grappling with prompts/tweets/lessons from someone else. This process is distinct for each person - with unique encoding, decoding, merging and splitting patterns. 

This distinction might as well be categorized in gpt levels lol, which brings the question what level of tokenizer are you?  GPT-2, GPT-3, GPT-4 or GPT-4o tokenizer:)

Papers:
- Neural Machine Translation of Rare Words with Subword Units (https://arxiv.org/abs/1508.07909)
- Learning to Compress Prompts with Gist Tokens
(https://arxiv.org/abs/2304.08467)
- Language Models are Few-Shot Learners
(https://arxiv.org/abs/2005.14165)
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
(https://arxiv.org/abs/2405.05417)
- Language Models are Unsupervised Multitask Learners 

Code: 
https://github.com/karpathy/minbpe
https://github.com/openai/tiktoken
https://github.com/openai/gpt-2
https://github.com/google/sentencepiece

❤️
LeroyDyer fuzzy-g
👀
LeroyDyer
😎
LeroyDyer
65d883893a52cd9bcd8ab7cf
https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
leroy Samuel Dyer
LeroyDyer
user
6438a9027de34e8ea7e4b257
https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
Jaward Sesay
Jaward
user