While doing various projects I kept running into situations where I wanted to be able to have representative samples of some of the current large SOTA datasets that were smaller so I didn't need to worry about slicing or anything else at runtime.  So, I created sub datasets making sure to keep the same ratios of data sources.  Each dataset card provides info for what's in it.
text
While doing various projects I kept running into situations where I wanted to be able to have representative samples of some of the current large SOTA datasets that were smaller so I didn't need to worry about slicing or anything else at runtime.  So, I created sub datasets making sure to keep the same ratios of data sources.  Each dataset card provides info for what's in it.
new_line
100M token datasets:
text
100M token datasets:
new_line
RedPajama v2 100M
text
RedPajama v2 100M
new_line
Falcon RefinedWeb 100M
text
Falcon RefinedWeb 100M
new_line
Cosmopedia 100M
text
Cosmopedia 100M
new_line
new_line
1B token datasets:
text
1B token datasets:
new_line
Fineweb-edu 1B
text
Fineweb-edu 1B
new_line
RedPajama v1 1B
text
RedPajama v1 1B
new_line
RedPajama v2 1B (use this one)
text
RedPajama v2 1B (use this one)
new_line
Cosmopedia 1B
text
Cosmopedia 1B
new_line
new_line
10B token datasets:
text
10B token datasets:
new_line
RedPajama v1 10B
text
RedPajama v1 10B
new_line
Cosmopedia 10B
text
Cosmopedia 10B
new_line
new_line
Collection here:
text
Collection here:
new_line
https://huggingface.co/collections/krisbailey/bite-size-data
https://huggingface.co/collections/krisbailey/bite-size-data
link
While doing various projects I kept running into situations where I wanted to be able to have representative samples of some of the current large SOTA datasets that were smaller so I didn't need to worry about slicing or anything else at runtime.  So, I created sub datasets making sure to keep the same ratios of data sources.  Each dataset card provides info for what's in it.
100M token datasets:
RedPajama v2 100M
Falcon RefinedWeb 100M
Cosmopedia 100M

1B token datasets:
Fineweb-edu 1B
RedPajama v1 1B
RedPajama v2 1B (use this one)
Cosmopedia 1B

10B token datasets:
RedPajama v1 10B
Cosmopedia 10B

Collection here:
https://huggingface.co/collections/krisbailey/bite-size-data
👀
John6666 victor