After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: 𝗪𝗲𝗹𝗰𝗼𝗺𝗲 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧! 🤗
text
After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: 𝗪𝗲𝗹𝗰𝗼𝗺𝗲 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧! 🤗
new_line
new_line
We talk a lot about ✨Generative AI✨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
text
We talk a lot about ✨Generative AI✨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
new_line
new_line
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
text
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
new_line
new_line
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
text
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
new_line
new_line
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
text
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
new_line
new_line
➡️ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
text
➡️ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
new_line
new_line
𝗧𝗟;𝗗𝗥:
text
𝗧𝗟;𝗗𝗥:
new_line
🏛️ Architecture changes:
text
🏛️ Architecture changes:
new_line
⇒ First, standard modernizations:
text
⇒ First, standard modernizations:
new_line
- Rotary positional embeddings (RoPE)
text
- Rotary positional embeddings (RoPE)
new_line
- Replace GeLU with GeGLU,
text
- Replace GeLU with GeGLU,
new_line
- Use Flash Attention 2  
text
- Use Flash Attention 2  
new_line
✨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
text
✨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
new_line
new_line
🥇 As a result, the model tops the game of encoder models:
text
🥇 As a result, the model tops the game of encoder models:
new_line
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
text
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
new_line
new_line
Read the blog post 👉 
text
Read the blog post 👉 
https://huggingface.co/blog/modernbert
https://huggingface.co/blog/modernbert
link
After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: 𝗪𝗲𝗹𝗰𝗼𝗺𝗲 𝗠𝗼𝗱𝗲𝗿𝗻𝗕𝗘𝗥𝗧! 🤗

We talk a lot about ✨Generative AI✨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.

The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).

It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.

Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.

➡️ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.

𝗧𝗟;𝗗𝗥:
🏛️ Architecture changes:
⇒ First, standard modernizations:
- Rotary positional embeddings (RoPE)
- Replace GeLU with GeGLU,
- Use Flash Attention 2  
✨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.

🥇 As a result, the model tops the game of encoder models:
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!

Read the blog post 👉 https://huggingface.co/blog/modernbert
🔥
tudorizer victor John6666 wilsoneaton prithivMLmods celinah GoDjMike comarproject dillfrescott
👍
bouy1915 xargs01 comarproject dillfrescott linekin
6691250d76c0fa097fd048be
/avatars/5a398c9961a1f76e18203b3521f3b5da.svg
Corwin Black
Mescalamba
user