What happens when you make an LLM drive a car where physics are real and actions can't be undone?
text
What happens when you make an LLM drive a car where physics are real and actions can't be undone?
new_line
new_line
I ported CARLA, the autonomous driving simulator, to OpenEnv and added training support via TRL + Hugging Face Spaces.
text
I ported CARLA, the autonomous driving simulator, to OpenEnv and added training support via TRL + Hugging Face Spaces.
new_line
new_line
The model interacts with the simulator through tool calls (observe, brake, change lane) and learns from a reward signal.
text
The model interacts with the simulator through tool calls (observe, brake, change lane) and learns from a reward signal.
new_line
new_line
In 50 training steps, Qwen 0.6B learns to swerve and brake to avoid pedestrians in emergency situations.
text
In 50 training steps, Qwen 0.6B learns to swerve and brake to avoid pedestrians in emergency situations.
new_line
new_line
The project supports text and vision (VLMs can see through a camera sensor), open-world driving with traffic, and multiple driving scenarios.
text
The project supports text and vision (VLMs can see through a camera sensor), open-world driving with traffic, and multiple driving scenarios.
new_line
new_line
This builds on the carla-env project by sinatras, which originally placed LLMs inside CARLA for evaluation. We extended it with vision, new scenarios, rubric-based rewards, and made it trainable end-to-end.
text
This builds on the carla-env project by sinatras, which originally placed LLMs inside CARLA for evaluation. We extended it with vision, new scenarios, rubric-based rewards, and made it trainable end-to-end.
new_line
new_line
Blog: 
text
Blog: 
https://huggingface.co/blog/sergiopaniego/bringing-carla-to-openenv-trl/
https://huggingface.co/blog/sergiopaniego/bringing-carla-to-openenv-trl/
link
new_line
CARLA env in OpenEnv: 
text
CARLA env in OpenEnv: 
https://github.com/meta-pytorch/OpenEnv/tree/main/envs/carla_env
https://github.com/meta-pytorch/OpenEnv/tree/main/envs/carla_env
link
new_line
Training script: 
text
Training script: 
https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/carla.py
https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/carla.py
link
What happens when you make an LLM drive a car where physics are real and actions can't be undone?

I ported CARLA, the autonomous driving simulator, to OpenEnv and added training support via TRL + Hugging Face Spaces.

The model interacts with the simulator through tool calls (observe, brake, change lane) and learns from a reward signal.

In 50 training steps, Qwen 0.6B learns to swerve and brake to avoid pedestrians in emergency situations.

The project supports text and vision (VLMs can see through a camera sensor), open-world driving with traffic, and multiple driving scenarios.

This builds on the carla-env project by sinatras, which originally placed LLMs inside CARLA for evaluation. We extended it with vision, new scenarios, rubric-based rewards, and made it trainable end-to-end.

Blog: https://huggingface.co/blog/sergiopaniego/bringing-carla-to-openenv-trl/
CARLA env in OpenEnv: https://github.com/meta-pytorch/OpenEnv/tree/main/envs/carla_env
Training script: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/carla.py
🚀
qgallouedec pcuenq alvarobartt John6666 unmodeled-tyler ExodyZz YatharthS