RPC-Bench Leaderboard

How to Submit

1. Fork this repository.
2. Create a new branch for your submission.
3. Add your submission folder under submissions/<organization>__<model>__<input_config>/.
4. Open a Pull Request with the new submission folder.

Submission Directory Requirements

Each submission directory must contain the metadata and predictions for one model/input configuration pair:

<organization>__<model>__<input_config>/
  metadata.yaml
  predictions.jsonl
  generation_config.json      # optional, recommended
  artifacts/                  # optional logs or prompt notes

Use URL-safe directory names. Replace spaces, slashes, and special characters with hyphens; keep input_config as TEXT or VISUAL.

metadata.yaml

model_name: "My Model"
organization: "My Org"
model_url: https://...        # optional work link: paper, GitHub, model card, etc.
date: "2026-06-17"           # model release date, not submission date
split: test
input_config: TEXT            # TEXT or VISUAL

predictions.jsonl

Each line must be one JSON object:

{
  "id": "paper-id",
  "part_idx": 1,
  "question": "question text",
  "category": "category",
  "gen_answer": "model answer"
}

part_idx is the question index in the current paper's qa_pairs list (1 for the first item). category must match the corresponding item in test.json.

Validation Rules

Your submission will be validated before evaluation. To pass:

• metadata.yaml must include model_name, organization, date, split, and input_config.
• model_url is optional.
• date is the model release date, not the submission date.
• split must be test.
• input_config must be TEXT or VISUAL.
• predictions.jsonl must contain exactly one line for every QA item in test.json.
• part_idx is the question index in the current paper's qa_pairs list (1 for the first item).
• id, part_idx, question, and category must exactly match the benchmark item.
• gen_answer must be a string.
• For Claim_Verification, gen_answer must be exactly True or False.

Submission Process

1. Open PR: add your folder under submissions/<organization>__<model>__<input_config>/.
2. Fix issues: if validation fails, update the PR with corrected files.
3. Review: once validation passes, a maintainer reviews the submission.
4. Evaluate: maintainers run the official evaluator in a controlled local environment.
5. Import: accepted aggregate results are imported to the leaderboard.