🗄️ Massive data release on the HF Hub for 75 languages!
text
🗄️ Massive data release on the HF Hub for 75 languages!
new_line
new_line
https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2
https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2
link
new_line
new_line
In December of last year, HPLT (
text
In December of last year, HPLT (
https://hplt-project.org/
https://hplt-project.org/
link
) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:
text
) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:
new_line
new_line
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
python
```python
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
```
code_fence
new_line
Let's use this big blob of data to build something awesome in our languages! 🥳
text
Let's use this big blob of data to build something awesome in our languages! 🥳
🗄️ Massive data release on the HF Hub for 75 languages!

https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2

In December of last year, HPLT (https://hplt-project.org/) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:

```python
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
```
Let's use this big blob of data to build something awesome in our languages! 🥳
❤️
stefan-it samusenps ajibawa-2023 clem osanseviero ymoslem damerajee rafaelpierrehf
6487239cca30096ea9f52115
https://cdn-avatars.huggingface.co/v1/production/uploads/6487239cca30096ea9f52115/HMte9wjKJgfcxsO-5vb_Q.jpeg
dame rajee
damerajee
user