Introducing 📐𝐅𝐢𝐧𝐞𝐌𝐚𝐭𝐡: the best public math pre-training dataset with 50B+ tokens!
text
Introducing 📐𝐅𝐢𝐧𝐞𝐌𝐚𝐭𝐡: the best public math pre-training dataset with 50B+ tokens!
new_line
https://huggingface.co/datasets/HuggingFaceTB/finemath
HuggingFaceTB/finemath
dataset
resource
https://huggingface.co/datasets/HuggingFaceTB/finemath
new_line
new_line
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
text
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
new_line
new_line
We build the dataset by:
text
We build the dataset by:
new_line
🛠️ carefully extracting math data from Common Crawl;
text
🛠️ carefully extracting math data from Common Crawl;
new_line
🔎 iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction. 
text
🔎 iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction. 
new_line
new_line
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
text
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
new_line
new_line
We hope this helps advance the performance of LLMs on math and reasoning! 🚀 
text
We hope this helps advance the performance of LLMs on math and reasoning! 🚀 
new_line
We’re also releasing all the ablation models as well as the evaluation code. 
text
We’re also releasing all the ablation models as well as the evaluation code. 
new_line
new_line
https://huggingface.co/collections/HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
collection
resource
https://huggingface.co/collections/HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
Introducing 📐𝐅𝐢𝐧𝐞𝐌𝐚𝐭𝐡: the best public math pre-training dataset with 50B+ tokens!
https://huggingface.co/datasets/HuggingFaceTB/finemath

Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.

We build the dataset by:
🛠️ carefully extracting math data from Common Crawl;
🔎 iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction. 

We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.

We hope this helps advance the performance of LLMs on math and reasoning! 🚀 
We’re also releasing all the ablation models as well as the evaluation code. 

https://huggingface.co/collections/HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
🚀
prithivMLmods loubnabnl capybarist John6666 wilsoneaton lamaraguilar AtAndDev gabrielmbmb eastbrick
🔥
eliebak loubnabnl davanstrien anakin87 victor AtAndDev SultanR