Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?
text
Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?
new_line
new_line
Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄
text
Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄
new_line
new_line
🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈
text
🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈
new_line
new_line
📄 Paper: 
text
📄 Paper: 
RegMix: Data Mixture as Regression for Language Model Pre-training (2407.01492)
https://huggingface.co/papers/2407.01492
2407.01492
paper
resource
https://huggingface.co/papers/2407.01492
new_line
💻 Code: 
text
💻 Code: 
https://github.com/sail-sg/regmix
https://github.com/sail-sg/regmix
link
new_line
📊 Collection: 
text
📊 Collection: 
https://huggingface.co/collections/sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
collection
resource
https://huggingface.co/collections/sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
new_line
 🎮 Demo: 
text
 🎮 Demo: 
https://huggingface.co/spaces/sail/RegMix
https://huggingface.co/spaces/sail/RegMix
link
new_line
Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?

Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄

🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈

📄 Paper: https://huggingface.co/papers/2407.01492
💻 Code: https://github.com/sail-sg/regmix
📊 Collection: https://huggingface.co/collections/sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
 🎮 Demo: https://huggingface.co/spaces/sail/RegMix

👍
xszheng2020 osanseviero Taylor658 Ksgk-fy
🔥
xszheng2020
🚀
xszheng2020