Impressed by the work of 
text
Impressed by the work of 
@guipenedo
mention
guipenedo
text
@hynky
mention
hynky
text
@loubnabnl
mention
loubnabnl
text
@anton-l
mention
anton-l
text
@craffel
mention
craffel
text
@lvwerra
mention
lvwerra
text
@thomwolf
mention
thomwolf
 on FineWeb.
text
 on FineWeb.
new_line
new_line
LLMs are only as good as the data they have been trained on, but the crucial aspect of pretraining data remains obscure. Our approach lifts the veil on building high-quality pretraining datasets by sharing every detail about this process to enable a wider community to build on top of it.
text
LLMs are only as good as the data they have been trained on, but the crucial aspect of pretraining data remains obscure. Our approach lifts the veil on building high-quality pretraining datasets by sharing every detail about this process to enable a wider community to build on top of it.
new_line
new_line
- The FineWeb-Edu dataset, which outperforms all openly accessible web datasets in a number of educational benchmarks. We built it by developing a quality classifier using annotations generated by an LLM.
text
- The FineWeb-Edu dataset, which outperforms all openly accessible web datasets in a number of educational benchmarks. We built it by developing a quality classifier using annotations generated by an LLM.
new_line
new_line
- A new technical report explaining in detail how to create a large and high-quality web-scale dataset for LLM pretraining such as FineWeb
text
- A new technical report explaining in detail how to create a large and high-quality web-scale dataset for LLM pretraining such as FineWeb
new_line
new_line
👉 
text
👉 
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
HuggingFaceFW/blogpost-fineweb-v1
space
resource
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
new_line
Impressed by the work of @guipenedo @hynky @loubnabnl @anton-l @craffel @lvwerra @thomwolf on FineWeb.

LLMs are only as good as the data they have been trained on, but the crucial aspect of pretraining data remains obscure. Our approach lifts the veil on building high-quality pretraining datasets by sharing every detail about this process to enable a wider community to build on top of it.

- The FineWeb-Edu dataset, which outperforms all openly accessible web datasets in a number of educational benchmarks. We built it by developing a quality classifier using annotations generated by an LLM.

- A new technical report explaining in detail how to create a large and high-quality web-scale dataset for LLM pretraining such as FineWeb

👉 https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1

🔥
osanseviero adamm-hf YannisTevissen ZennyKenny
➕
osanseviero adamm-hf