Add pipeline tag, library name, and GitHub README content

README.md

@@ -1,10 +1,104 @@
 ---
-license: apache-2.0
+base_model:
+- Qwen/Qwen2.5-VL-7B-Instruct
 datasets:
 - WaltonFuture/Multimodal-Cold-Start
 - WaltonFuture/Multimodal-RL-Data
-base_model:
-- Qwen/Qwen2.5-VL-7B-Instruct
+license: apache-2.0
+pipeline_tag: image-text-to-text
+library_name: transformers
 ---
+
 * 🐙 **GitHub Repo:** [waltonfuture/RL-with-Cold-Start](https://github.com/waltonfuture/RL-with-Cold-Start)
-* 📜 **Paper (arXiv):** [Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start (arXiv:2505.22334)](https://arxiv.org/abs/2505.22334)
+* 📜 **Paper (arXiv):** [Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start (arXiv:2505.22334)](https://arxiv.org/abs/2505.22334)
+
+<div align=center>
+<img src="assets/model_comparison.png"  width = "80%" alt="model_comparison" align=center/>
+</div>
+
+## Cold Start Stage
+
+We conduct supervised fine-tuning on Qwen2.5-VL-3B and Qwen2.5-VL-7B using [ms-swift](https://github.com/modelscope/ms-swift). In this stage, please refer to this curated [dataset](https://huggingface.co/datasets/WaltonFuture/Multimodal-Cold-Start) distilled from Qwen2.5-VL-32B using rejection sampling.
+
+### Setup
+
+```bash
+git clone https://github.com/waltonfuture/RL-with-Cold-Start.git
+cd RL-with-Cold-Start/SFT
+pip install -e .
+```
+
+### Prepare Data
+
+```bash
+python convert_data.py
+```
+
+### SFT
+
+```bash
+bash qwen2.5vl_sft.sh
+```
+
+The checkpoint can be found in SFT/output.
+
+## RL Stage
+
+We further conduct GRPO using [EasyR1](https://github.com/hiyouga/EasyR1). Please refer to this [dataset](https://huggingface.co/datasets/WaltonFuture/Multimodal-RL-Data) for the GRPO training.
+
+### Setup
+
+```bash
+git clone https://github.com/waltonfuture/RL-with-Cold-Start.git
+cd RL-with-Cold-Start/GRPO
+pip install -e .
+```
+
+### GRPO Training (replace the checkpoint with the model after SFT)
+
+```bash
+bash examples/qwen2_5_vl_7b_grpo.sh
+```
+
+### Merge Checkpoint in Hugging Face Format
+
+```bash
+python3 scripts/model_merger.py --local_dir checkpoints/easyr1/qwen2_5_vl_7b_grpo/global_step_80/actor
+```
+
+## Data Access
+
+Our two stage datasets are now available on Huggingface.
+
+|  Stage          | Data  |
+| ------------------ | ------------- |
+| Cold Start  | [Multimodal-Cold-Start](https://huggingface.co/datasets/WaltonFuture/Multimodal-Cold-Start) |
+| RL  | [Multimodal-RL-Data](https://huggingface.co/datasets/WaltonFuture/Multimodal-RL-Data) |
+
+## Model Access
+
+Our models are now available on Huggingface.
+
+|  Backbone           | Our model   |
+| ------------------ | ------------- |
+| Qwen2.5-VL-7b  | [Qwen2.5VL-7b-RL-with-Cold-Start](https://huggingface.co/WaltonFuture/Qwen2.5VL-7b-RLCS) |
+| Qwen2.5-VL-3b  | [Qwen2.5VL-3b-RL-with-Cold-Start](https://huggingface.co/WaltonFuture/Qwen2.5VL-3b-RLCS) |
+
+## Acknowledgment
+
+Our models are built upon the amazing [Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl-6795ffac22b334a837c0f9a5) family.
+We thank [EasyR1](https://github.com/hiyouga/EasyR1) and [ms-swift](https://github.com/modelscope/ms-swift) for their training codes.
+
+## Contact
+
+Please contact Lai Wei ([email protected]) if needed.
+
+## Citation
+```
+@article{wei2025advancing,
+  title={Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start},
+  author={Wei, Lai and Li, Yuting and Zheng, Kaipeng and Wang, Chen and Wang, Yue and Kong, Linghe and Sun, Lichao and Huang, Weiran},
+  journal={arXiv preprint arXiv:2505.22334},
+  year={2025}
+}
+```