Your Language Model needs better (open) environments to learn 🌀
text
Your Language Model needs better (open) environments to learn 🌀
new_line
new_line
📝 
text
📝 
https://huggingface.co/blog/anakin87/environments-hub
https://huggingface.co/blog/anakin87/environments-hub
link
new_line
new_line
RL environments help LLMs practice, reason, and improve.
text
RL environments help LLMs practice, reason, and improve.
new_line
I explored the Environments Hub and wrote a walkthrough showing how to train and evaluate models using these open environments.
text
I explored the Environments Hub and wrote a walkthrough showing how to train and evaluate models using these open environments.
new_line
new_line
1️⃣ 𝗪𝗵𝘆 𝗥𝗟 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗳𝗼𝗿 𝗟𝗟𝗠𝘀
text
1️⃣ 𝗪𝗵𝘆 𝗥𝗟 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗳𝗼𝗿 𝗟𝗟𝗠𝘀
new_line
new_line
DeepSeek-R1 made clear that Reinforcement Learning can be used to incentivize reasoning in LLMs.
text
DeepSeek-R1 made clear that Reinforcement Learning can be used to incentivize reasoning in LLMs.
new_line
In GRPO, the model generates multiple answers and learns to prefer the better ones from rewards.
text
In GRPO, the model generates multiple answers and learns to prefer the better ones from rewards.
new_line
new_line
new_line
2️⃣ 𝗪𝗵𝗮𝘁 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 𝗮𝗿𝗲
text
2️⃣ 𝗪𝗵𝗮𝘁 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 𝗮𝗿𝗲
new_line
In classic RL, the environment is the world where the Agent lives, interacts, and get rewards to learn.
text
In classic RL, the environment is the world where the Agent lives, interacts, and get rewards to learn.
new_line
new_line
We can also think of them as software packages, containing data, harness and scoring rules - for the model
text
We can also think of them as software packages, containing data, harness and scoring rules - for the model
new_line
to learn and be evaluated.
text
to learn and be evaluated.
new_line
new_line
Nowadays, the Agent is not just the LLM. It can use tools, from a weather API to a terminal.
text
Nowadays, the Agent is not just the LLM. It can use tools, from a weather API to a terminal.
new_line
new_line
This makes environments for training and evaluation more complex and critical.
text
This makes environments for training and evaluation more complex and critical.
new_line
new_line
new_line
3️⃣ 𝐓𝐡𝐞 𝐨𝐩𝐞𝐧 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞
text
3️⃣ 𝐓𝐡𝐞 𝐨𝐩𝐞𝐧 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞
new_line
new_line
Big labs are advancing, but open models and the community still face a fragmented ecosystem.
text
Big labs are advancing, but open models and the community still face a fragmented ecosystem.
new_line
We risk becoming users of systems built with tools we can't access or fully understand.
text
We risk becoming users of systems built with tools we can't access or fully understand.
new_line
new_line
new_line
4️⃣ 𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭𝐬 𝐇𝐮𝐛
text
4️⃣ 𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭𝐬 𝐇𝐮𝐛
new_line
That's why, I was excited when Prime Intellect released the Environments Hub.
text
That's why, I was excited when Prime Intellect released the Environments Hub.
new_line
new_line
It's a place where people share RL environments: tasks you can use to train LLMs with RL (GRPO-style) or evaluate Agents.
text
It's a place where people share RL environments: tasks you can use to train LLMs with RL (GRPO-style) or evaluate Agents.
new_line
Plus, the Verifiers library (
text
Plus, the Verifiers library (
@willcb
mention
willcb
) standardizes the creation of RL environments and evaluations.
text
) standardizes the creation of RL environments and evaluations.
new_line
They can help to keep science and experimentation open. 🔬
text
They can help to keep science and experimentation open. 🔬
new_line
new_line
new_line
I explored the Hub and wrote a hands-on walkthrough 📝
text
I explored the Hub and wrote a hands-on walkthrough 📝
new_line
- RL + LLMs basics
text
- RL + LLMs basics
new_line
- Environments Hub navigation
text
- Environments Hub navigation
new_line
- Evaluating 
text
- Evaluating 
models/Agents
text
models/Agents
new_line
- GRPO Training a tiny model on an alphabetical sort task
text
- GRPO Training a tiny model on an alphabetical sort task
new_line
new_line
Take a look!
text
Take a look!
new_line
new_line
📝 
text
📝 
https://huggingface.co/blog/anakin87/environments-hub
https://huggingface.co/blog/anakin87/environments-hub
link
Your Language Model needs better (open) environments to learn 🌀

📝 https://huggingface.co/blog/anakin87/environments-hub

RL environments help LLMs practice, reason, and improve.
I explored the Environments Hub and wrote a walkthrough showing how to train and evaluate models using these open environments.

1️⃣ 𝗪𝗵𝘆 𝗥𝗟 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗳𝗼𝗿 𝗟𝗟𝗠𝘀

DeepSeek-R1 made clear that Reinforcement Learning can be used to incentivize reasoning in LLMs.
In GRPO, the model generates multiple answers and learns to prefer the better ones from rewards.


2️⃣ 𝗪𝗵𝗮𝘁 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 𝗮𝗿𝗲
In classic RL, the environment is the world where the Agent lives, interacts, and get rewards to learn.

We can also think of them as software packages, containing data, harness and scoring rules - for the model
to learn and be evaluated.

Nowadays, the Agent is not just the LLM. It can use tools, from a weather API to a terminal.

This makes environments for training and evaluation more complex and critical.


3️⃣ 𝐓𝐡𝐞 𝐨𝐩𝐞𝐧 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞

Big labs are advancing, but open models and the community still face a fragmented ecosystem.
We risk becoming users of systems built with tools we can't access or fully understand.


4️⃣ 𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭𝐬 𝐇𝐮𝐛
That's why, I was excited when Prime Intellect released the Environments Hub.

It's a place where people share RL environments: tasks you can use to train LLMs with RL (GRPO-style) or evaluate Agents.
Plus, the Verifiers library (@willcb) standardizes the creation of RL environments and evaluations.
They can help to keep science and experimentation open. 🔬


I explored the Hub and wrote a hands-on walkthrough 📝
- RL + LLMs basics
- Environments Hub navigation
- Evaluating models/Agents
- GRPO Training a tiny model on an alphabetical sort task

Take a look!

📝 https://huggingface.co/blog/anakin87/environments-hub
🚀
John6666