💡 Thinking Tokens For Language Models!
text
💡 Thinking Tokens For Language Models!
new_line
new_line
How much is 56 times 37? Can you answer that right away?
text
How much is 56 times 37? Can you answer that right away?
new_line
new_line
In a short paper, David Herel and Tomas Mikolov propose a simple method to improve the reasoning of language models when performing complex calculations. 
text
In a short paper, David Herel and Tomas Mikolov propose a simple method to improve the reasoning of language models when performing complex calculations. 
new_line
new_line
📌 They note that, although language models are not that good with difficult calculations, humans also cannot perform these calculations immediately and require a considerable amount of time to come up with an answer.
text
📌 They note that, although language models are not that good with difficult calculations, humans also cannot perform these calculations immediately and require a considerable amount of time to come up with an answer.
new_line
new_line
Inspired by this, they introduce 💡Thinking Tokens💡
text
Inspired by this, they introduce 💡Thinking Tokens💡
new_line
new_line
So what are those "thinking tokens"?! Nothing fancy, they are just special tokens '<T>' that you insert after each word in a sentence whenever a complex problem is encountered. That's it!
text
So what are those "thinking tokens"?! Nothing fancy, they are just special tokens '<T>' that you insert after each word in a sentence whenever a complex problem is encountered. That's it!
new_line
new_line
👉 The main idea is to "buy" the model "some time" to think about the problem with these additional computations before answering. Using this method they observed an improved (a little bit) perplexity.
text
👉 The main idea is to "buy" the model "some time" to think about the problem with these additional computations before answering. Using this method they observed an improved (a little bit) perplexity.
new_line
new_line
👉 Before getting excited note that: They have added these tokens manually, and they have used an RNN language model. From the paper:
text
👉 Before getting excited note that: They have added these tokens manually, and they have used an RNN language model. From the paper:
new_line
new_line
"As a proof of concept, we have added N ’thinking tokens’ (< T >) after each observed word in a dataset. Our vision is that this basic concept can be extended to a self-adjusting model, which will be able to decide itself if and how many ’thinking tokens’ will be used for a specific problem, where N could also vary throughout the sentence. This would allow us to reduce the computational time, which would not increase N times."
text
"As a proof of concept, we have added N ’thinking tokens’ (< T >) after each observed word in a dataset. Our vision is that this basic concept can be extended to a self-adjusting model, which will be able to decide itself if and how many ’thinking tokens’ will be used for a specific problem, where N could also vary throughout the sentence. This would allow us to reduce the computational time, which would not increase N times."
💡 Thinking Tokens For Language Models!

How much is 56 times 37? Can you answer that right away?

In a short paper, David Herel and Tomas Mikolov propose a simple method to improve the reasoning of language models when performing complex calculations. 

📌 They note that, although language models are not that good with difficult calculations, humans also cannot perform these calculations immediately and require a considerable amount of time to come up with an answer.

Inspired by this, they introduce 💡Thinking Tokens💡

So what are those "thinking tokens"?! Nothing fancy, they are just special tokens '<T>' that you insert after each word in a sentence whenever a complex problem is encountered. That's it!

👉 The main idea is to "buy" the model "some time" to think about the problem with these additional computations before answering. Using this method they observed an improved (a little bit) perplexity.

👉 Before getting excited note that: They have added these tokens manually, and they have used an RNN language model. From the paper:

"As a proof of concept, we have added N ’thinking tokens’ (< T >) after each observed word in a dataset. Our vision is that this basic concept can be extended to a self-adjusting model, which will be able to decide itself if and how many ’thinking tokens’ will be used for a specific problem, where N could also vary throughout the sentence. This would allow us to reduce the computational time, which would not increase N times."
😎
LeroyDyer raincandy-u victor
❤️
LeroyDyer
🔥
pragyananda
👍
pragyananda
62645f88c39850dc093d6105
https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
Mohammed Hamdy
mmhamdy
user
65d883893a52cd9bcd8ab7cf
https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
leroy Samuel Dyer
LeroyDyer
user
62507a5f49c76eb605dafa26
https://cdn-avatars.huggingface.co/v1/production/uploads/62507a5f49c76eb605dafa26/esX97YcNiRh8pncgaQ9D-.png
Ritwik Mishra
ritwikm
user