---
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
license: apache-2.0
pipeline_tag: video-text-to-text
tags:
- robotic-manipulation
- reinforcement-learning
- chain-of-thought
datasets:
- LeonOverload/primo-bench-json
- LeonOverload/primo-sft-json
- LeonOverload/primo-video-media
---

# PRIMO R1: Process Reasoning Induced Monitoring

PRIMO R1 (Process Reasoning Induced Monitoring) is a 7B video multimodal large language model (MLLM) framework designed for accurate process supervision in long-horizon robotic manipulation. It was introduced in the paper [From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation](https://huggingface.co/papers/2603.15600).

This is the **final checkpoint**: a Qwen2.5-VL-7B-Instruct base taken through an SFT cold start and then GRPO reinforcement learning. Use this model unless you specifically want the ablation.

## Model Description

Current video MLLMs often function as passive "Observers" that recognize ongoing events rather than evaluating the current state relative to the final task goal. PRIMO R1 transforms these models into active "Critics" by:

- **Reinforcement Learning**: Leveraging outcome-based RL to incentivize explicit Chain-of-Thought (CoT) generation for progress estimation.
- **Temporal Anchoring**: Constructing a structured temporal input that explicitly anchors the video sequence between initial and current state images.
- **Process Reasoning**: Focusing on evaluating the current state against the intended task goal to detect failures and track progress.

## Resources

| | |
| --- | --- |
| Code | [10-OASIS-01/PRIMO-R1](https://github.com/10-OASIS-01/PRIMO-R1) |
| Collection | [PRIMO R1](https://huggingface.co/collections/LeonOverload/primo-r1) |
| Paper | [arXiv 2603.15600](https://arxiv.org/abs/2603.15600) · [project page](https://10-oasis-01.github.io/primo-r1-website/) |
| Stage-1 ablation | [PRIMO-COT-SFT-7B](https://huggingface.co/LeonOverload/PRIMO-COT-SFT-7B) |
| Benchmark | [primo-bench-json](https://huggingface.co/datasets/LeonOverload/primo-bench-json) |
| Training data | [primo-sft-json](https://huggingface.co/datasets/LeonOverload/primo-sft-json) · [primo-rl-json](https://huggingface.co/datasets/LeonOverload/primo-rl-json) |
| Videos | [primo-video-media](https://huggingface.co/datasets/LeonOverload/primo-video-media) |

## Download

The `main` branch holds inference files only (**16.6 GB**). DeepSpeed resume state from RL step 2500 lives on a separate branch, so you do not pay ~100 GB for it by accident.

```python
from huggingface_hub import snapshot_download

# Inference: 16.6 GB
snapshot_download("LeonOverload/PRIMO-R1-7B", local_dir="models/PRIMO-R1-7B")

# Resume RL training from step 2500: ~116 GB
snapshot_download("LeonOverload/PRIMO-R1-7B", revision="training-state")
```

Or from the CLI:

```bash
hf download LeonOverload/PRIMO-R1-7B --local-dir models/PRIMO-R1-7B
```

If you are on a revision where the optimizer state is still on `main`, exclude it explicitly. Note that `--exclude` is ignored if you also pass filenames positionally:

```bash
hf download LeonOverload/PRIMO-R1-7B \
    --exclude "global_step*" --exclude "rng_state*" \
    --exclude "scheduler.pt" --exclude "latest" \
    --local-dir models/PRIMO-R1-7B
```

## Setup

The environment matters more than usual here. Qwen2.5-VL support shifted between transformers releases, and installing a PyPI `transformers` over the pinned tree is the most common cause of shape and processor errors:

```bash
git clone https://github.com/10-OASIS-01/PRIMO-R1 && cd PRIMO-R1
conda create -n primo-r1 python=3.11 && conda activate primo-r1
bash setup.sh
```

`setup.sh` pins `vllm==0.7.2`, `trl==0.16.0`, and installs the vendored `transformers-main/` tree last so nothing replaces it.

## Input format — read this before running anything

PRIMO R1 is trained on an **interleaved** input that anchors the clip between two still frames. The content list must be in exactly this order:

```
image (initial frame)  →  video (the clip)  →  image (current frame)  →  text (question)
```

Feeding a bare video clip will still produce output, but it degrades quality silently — the paper's Table 4 ablation measures this directly, and a current-state image alone raises average MAE from 15.52 to 59.50. Keep the order and keep both anchor frames.

The prompt template is equally load-bearing: the answer extractor is a regex over `<answer>...</answer>`, and the `<think>` block is required to contain `<planning>`, `<observation>`, and `<reasoning>` subsections in that order. Changing the template without changing the extractor collapses scores rather than raising an error.

## Usage

This example is transcribed from `src/eval/eval_interleave.py`, the harness that produced the published numbers.

The prompts and the frame extraction are **imported, not pasted**. Both live in the repo cloned during setup: `src/primo_prompts.py` holds every prompt constant and `src/primo_video_utils.py` holds the frame helpers. They are the same objects the eval harness uses, so an example that imports them cannot drift out of sync with the checkpoint. `setup.sh` puts `src/` on `PYTHONPATH`; from elsewhere, add it yourself:

```python
import sys
sys.path.insert(0, "/path/to/PRIMO-R1/src")
```

```python
import torch
from transformers import AutoProcessor, AutoTokenizer
from vllm import LLM, SamplingParams
from qwen_vl_utils import process_vision_info

# The single source of truth for the prompt format and the anchor frames.
from primo_prompts import SYSTEM_PROMPT, build_question
from primo_video_utils import extract_frames_on_demand

MODEL_PATH = "models/PRIMO-R1-7B"   # or "LeonOverload/PRIMO-R1-7B"
video_path = "path/to/your/episode.mp4"
question = "What is the completion percentage of the task in the video?"
problem_type = "regression"

# (initial state, current state) as PIL images. LRU-cached, so calling this
# again for the same video is free.
init_img, current_img = extract_frames_on_demand(video_path)

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
    {
        "role": "user",
        "content": [
            {"type": "image", "image": init_img},                      # 1. initial state
            {"type": "video", "video": video_path, "nframes": 22},     # 2. the clip
            {"type": "image", "image": current_img},                   # 3. current state
            # QUESTION_TEMPLATE.format(...) + TYPE_TEMPLATE[problem_type]
            {"type": "text", "text": build_question(question, problem_type)},
        ],
    },
]

llm = LLM(
    model=MODEL_PATH,
    tensor_parallel_size=torch.cuda.device_count(),
    max_model_len=16384,
    gpu_memory_utilization=0.8,
    limit_mm_per_prompt={"image": 3, "video": 1},   # 2 anchor frames + 1 video
)

# top_p must stay this low. Larger values produce garbled output on this model.
sampling_params = SamplingParams(temperature=0.1, top_p=0.001, max_tokens=4096)

processor = AutoProcessor.from_pretrained(MODEL_PATH)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
tokenizer.padding_side = "left"
processor.tokenizer = tokenizer

prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)

mm_data = {"video": video_inputs[0]}
if image_inputs:
    mm_data["image"] = image_inputs

outputs = llm.generate(
    [{
        "prompt": prompt,
        "multi_modal_data": mm_data,
        "mm_processor_kwargs": {k: v[0] for k, v in video_kwargs.items()},
    }],
    sampling_params=sampling_params,
)
print(outputs[0].outputs[0].text)
```

### Parsing the output

```python
import re

def extract_answer(text):
    m = re.search(r"<answer>\s*(.*?)\s*</answer>", text, re.DOTALL)
    return m.group(1).strip() if m else ""

def extract_think(text):
    m = re.search(r"<think>\s*(.*?)\s*</think>", text, re.DOTALL)
    return m.group(1).strip() if m else ""
```

For `regression` and `numerical` questions the answer is a **progress percentage on a 0–100 scale** (`42.5`, not `0.425`). A trailing `%` is tolerated by the reference parser, which divides by 100 when it sees one.

### Question types

`regression` and `numerical` (progress estimation), `multiple choice`, `boolean` (failure detection), `free-form`, `OCR`. `build_question` puts the type in both places — inside `QUESTION_TEMPLATE` and as the `TYPE_TEMPLATE` hint appended after it — because the model was trained with both present.

### The prompt constants

If you need to inspect or extend them rather than just call `build_question`:

```python
from primo_prompts import SYSTEM_PROMPT, QUESTION_TEMPLATE, TYPE_TEMPLATE

print(QUESTION_TEMPLATE.format(Question="...", question_type="regression"))
print(sorted(TYPE_TEMPLATE))
# ['OCR', 'boolean', 'free-form', 'multiple choice', 'numerical', 'regression']
```

`primo_prompts.py` also carries the baseline and ablation prompts under `*_BASELINE` / `*_PARSER_ONLY` / `*_QUESTION_ONLY` names. Those are for reproducing the paper's comparison rows, **not** for this checkpoint — it was trained on `QUESTION_TEMPLATE` and expects it.

### A note on frame counts

The published results were produced with the per-video frame cap at 22, so the launcher's `--nframes 32` effectively sampled 22. `nframes=22` above reproduces that; `primo_video_utils.choose_nframes` is what applies the cap in the harness (`MAX_NFRAMES`, overridable via `INTERLEAVE_MAX_NFRAMES`). Training capped videos at 16 frames; eval samples more at higher resolution.

## Performance

Progress estimation averaged over four environments (paper Table 1):

| Model | Avg MRA ↑ | Avg MAE ↓ |
| --- | --- | --- |
| GPT-4o | 79.33 | 20.67 |
| GPT-5 mini | 75.38 | 23.96 |
| Qwen2.5-VL-72B | 73.80 | 23.80 |
| Qwen2.5-VL-7B (base) | 67.79 | 29.99 |
| InternVL 3.5 8B | 71.74 | 28.09 |
| ProgressLM | 78.32 | 20.87 |
| VLAC | 74.90 | 25.10 |
| **PRIMO R1 (7B)** | **82.90** | **15.52** |

Both training stages are needed (paper Table 2, MRA↑):

| Model | ID avg | OOD avg | Overall |
| --- | --- | --- | --- |
| Qwen2.5-VL-7B (base) | 70.38 | 65.26 | 67.46 |
| SFT only | 81.46 | 77.77 | 79.35 |
| RL only | 81.71 | 72.97 | 76.72 |
| **PRIMO R1 (SFT+RL)** | **88.47** | **82.90** | **85.28** |

RL without SFT underperforms because the model struggles to discover the output format from scratch; SFT alone generalizes poorly out of domain.

Zero-shot failure detection on RoboFail (paper Table 3): **67.0%**, matching Gemini 2.0 Flash and above GPT-4o at 63.0.

To reproduce these numbers, see [primo-bench-json](https://huggingface.co/datasets/LeonOverload/primo-bench-json), which documents the full evaluation path. One caveat when comparing against other work: the repo's harnesses score `regression` with three different formulas, and each output file records which one it used in its `regression_metric` field. The published numbers above come from `linear_relative_accuracy`.

## Citations

If you find our work helpful for your research, please consider citing our work.

```bibtex
@misc{liu2026passiveobserveractivecritic,
      title={From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation},
      author={Yibin Liu and Yaxing Lyu and Daqi Gao and Zhixuan Liang and Weiliang Tang and Shilong Mu and Xiaokang Yang and Yao Mu},
      year={2026},
      eprint={2603.15600},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.15600},
}
```