📊 MMEB LEADERBOARD (VLM2Vec)
Introduction
We introduce a novel benchmark, MMEB-V1 (Massive Multimodal Embedding Benchmark), which includes 36 datasets spanning four meta-task categories: classification, visual question answering, retrieval, and visual grounding. MMEB provides a comprehensive framework for training and evaluating embedding models across various combinations of text and image modalities. All tasks are reformulated as ranking tasks, where the model follows instructions, processes a query, and selects the correct target from a set of candidates. The query and target can be an image, text, or a combination of both. MMEB-V1 is divided into 20 in-distribution datasets, which can be used for training, and 16 out-of-distribution datasets, reserved for evaluation.
Building upon on MMEB-V1, MMEB-V2 expands the evaluation scope to include five new tasks: four video-based tasks — Video Retrieval, Moment Retrieval, Video Classification, and Video Question Answering — and one task focused on visual documents, Visual Document Retrieval. This comprehensive suite enables robust evaluation of multimodal embedding models across static, temporal, and structured visual data settings.
⚠️ Your attention please:==================================================================================
We have fixed the errors found in ViDoSeek-page and MMLongBench-page datasets. Detailed information about the issues can be viewed HERE.
‼️ CALL TO ACTION: Please verify that your model's scores are accurate by following the instructions below. ‼️
The scores submitted before the fix have been renamed to ViDoSeek-page-before-fix and MMLongBench-page-before-fix, and the current Overall and Visdoc-Overall are using the fixed versions of these two datasets (eg. ViDoSeek-page-fixed and MMLongBench-page-fixed), which means the models submitted before the fix might now have a lower ranking.
Here is the list of models affected by this fix: File. If your model is on this list, we kindly ask you to visit the VisDoc leaderboard and verify if your model is missing the scores for ViDoSeek-page-fixed and MMLongBench-page-fixed, and then re-evaluate your model on these two datasets and submit the updated scores to ensure accurate ranking on the leaderboard.
In the generated report sheet, please double check that ViDoSeek-page and MMLongBench-page have been renamed to ViDoSeek-page-fixed and MMLongBench-page-fixed respectively.
Please let us know if you have any questions or concerns, or there is an issue with your model's scores. We appreciate your understanding and cooperation in maintaining the integrity of the benchmark. ===========================================================================================================
🔥 What's NEW:- [2026-01] ⚠️ The issues found in ViDoSeek-page and MMLongBench-page datasets have been fixed.
- [2025-11] The leaderboards' rankings can be directly downloaded in csv/json format. Scroll down to the bottom of this page and click the button to download.
- [2025-06] MMEB-V2 released!
| 📈Overview | Github | 📖MMEB-V2/VLM2Vec-V2 Paper | 📖MMEB-V1/VLM2Vec-V1 Paper | 🤗Hugging Face | Discord |
MMEB: Massive MultiModal Embedding Benchmark
Models are ranked based on Overall
Rank | Models | Model Size(B) | Date | Overall | Image-Overall | Video-Overall | Visdoc-Overall |
|---|---|---|---|---|---|---|---|
10 | unknown | 2026-01-06 | 77.82 | 80.12 | 67.15 | 82.36 |
I-CLS: Image Classification, I-QA: (Image) Visual Question Answering, I-RET: Image Retrieval, I-VG: (Image) Visual Grounding
Models are ranked based on Image-Overall
Models from the old V1 leaderboard are missing detailed scores of each dataset. We hope the authors of the models on V1 leaderboard could rerun your models using our updated V2 pipeline, and provide us the scores sheet with the new format, so that we can make them consistent with the other models' formats.
Rank | Models | Model Size(B) | Date | Image-Overall | I-CLS | I-QA | I-RET | I-VG | VOC2007 | N24News | SUN397 | ObjectNet | Country211 | Place365 | ImageNet-1K | HatefulMemes | ImageNet-A | ImageNet-R | OK-VQA | A-OKVQA | DocVQA | InfographicsVQA | ChartQA | Visual7W | ScienceQA | GQA | TextVQA | VizWiz | VisDial | CIRR | VisualNews_t2i | VisualNews_i2t | MSCOCO_t2i | MSCOCO_i2t | NIGHTS | WebQA | FashionIQ | Wiki-SS-NQ | OVEN | EDIS | MSCOCO | RefCOCO | RefCOCO-Matching | Visual7W-Pointing |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
10 | unknown | 2026-01-06 | 72.175 | 74.19 | 81.14 | 71.175 | 87.075 | 93.4 | 80.6 | 82.7 | 79.9 | 27.3 | 47.6 | 81.9 | 77.5 | 77.1 | 93.9 | 77.8 | 72.9 | 96.3 | 88.4 | 74.3 | 70.9 | 78.9 | 92.6 | 92.8 | 64.4 | 87.6 | 74.9 | 81.1 | 85.7 | 81.1 | 79.1 | 72.7 | 91.8 | 44.5 | 87.9 | 79.2 | 96.3 | 86.3 | 95.7 | 91.1 | 96.1 |
V-CLS: Video Classification, V-QA: (Video) Visual Question Answering, V-RET: Video Retrieval, V-MRET: Video Moment Retrieval
Models are ranked based on Video-Overall
Rank | Models | Model Size(B) | Date | Video-Overall | V-CLS | V-QA | V-RET | V-MRET | K700 | UCF101 | HMDB51 | SmthSmthV2 | Breakfast | Video-MME | MVBench | NExTQA | EgoSchema | ActivityNetQA | MSR-VTT | MSVD | DiDeMo | VATEX | YouCook2 | QVHighlight | Charades-STA | MomentSeeker |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
10 | unknown | 2025-12-18 | 67.74 | 85.19 | 66.71 | 59.09 | 54.79 | 67.6 | 98.7 | 93.5 | 94.7 | 63.05 | 54.67 | 56.35 | 78.61 | 66.8 | 65.05 | 53.37 | 74.48 | 66.43 | 54.87 | 39.48 | 75.25 | 33.43 | 55.68 |
⚠️ Please re-evaluate your models if you see a 0 on ViDoSeek-page-fixed or MMLongBench-page-fixed datasets.
VisDoc: Visual Document Understanding
Models are ranked based on **Visdoc-Overall**
Rank | Models | Model Size(B) | Date | Visdoc-Overall | Visdoc-Overall-before-fix | VisDoc-OOD-before-fix | ViDoRe-V1 | ViDoRe-V2 | VisRAG | VisDoc-OOD | ViDoRe_arxivqa | ViDoRe_docvqa | ViDoRe_infovqa | ViDoRe_tabfquad | ViDoRe_tatdqa | ViDoRe_shiftproject | ViDoRe_syntheticDocQA_artificial_intelligence | ViDoRe_syntheticDocQA_energy | ViDoRe_syntheticDocQA_government_reports | ViDoRe_syntheticDocQA_healthcare_industry | ViDoRe_esg_reports_human_labeled_v2 | ViDoRe_biomedical_lectures_v2_multilingual | ViDoRe_economics_reports_v2_multilingual | ViDoRe_esg_reports_v2_multilingual | VisRAG_ArxivQA | VisRAG_ChartQA | VisRAG_MP-DocVQA | VisRAG_SlideVQA | VisRAG_InfoVQA | VisRAG_PlotQA | ViDoSeek-page-fixed | ViDoSeek-doc | MMLongBench-page-fixed | MMLongBench-doc | ViDoSeek-page-before-fix | MMLongBench-page-before-fix |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
10 | unknown | 2026-01-27 | 84.12 | 77.88 | 44.77 | 91.31 | 68.66 | 89.66 | 71.87 | 91.46 | 65.34 | 94.26 | 93.45 | 83.24 | 91.42 | 99.63 | 96.63 | 98.02 | 99.63 | 78.22 | 65.62 | 64.88 | 66.02 | 90.37 | 87.65 | 95.19 | 97.42 | 97.03 | 76.28 | 88.54 | 82.48 | 60.68 | 55.79 | 22.29 | 17.15 |
Dataset Overview
This is the dictionary of all datasets used in our code. Please make sure all datasets' scores are included in your submission.
{'image': {'I-CLS': ['VOC2007',
'N24News',
'SUN397',
'ObjectNet',
'Country211',
'Place365',
'ImageNet-1K',
'HatefulMemes',
'ImageNet-A',
'ImageNet-R'],
'I-QA': ['OK-VQA',
'A-OKVQA',
'DocVQA',
'InfographicsVQA',
'ChartQA',
'Visual7W',
'ScienceQA',
'GQA',
'TextVQA',
'VizWiz'],
'I-RET': ['VisDial',
'CIRR',
'VisualNews_t2i',
'VisualNews_i2t',
'MSCOCO_t2i',
'MSCOCO_i2t',
'NIGHTS',
'WebQA',
'FashionIQ',
'Wiki-SS-NQ',
'OVEN',
'EDIS'],
'I-VG': ['MSCOCO',
'RefCOCO',
'RefCOCO-Matching',
'Visual7W-Pointing']},
'video': {'V-CLS': ['K700', 'UCF101', 'HMDB51', 'SmthSmthV2', 'Breakfast'],
'V-MRET': ['QVHighlight', 'Charades-STA', 'MomentSeeker'],
'V-QA': ['Video-MME',
'MVBench',
'NExTQA',
'EgoSchema',
'ActivityNetQA'],
'V-RET': ['MSR-VTT', 'MSVD', 'DiDeMo', 'VATEX', 'YouCook2']},
'visdoc': {'ViDoRe-V1': ['ViDoRe_arxivqa',
'ViDoRe_docvqa',
'ViDoRe_infovqa',
'ViDoRe_tabfquad',
'ViDoRe_tatdqa',
'ViDoRe_shiftproject',
'ViDoRe_syntheticDocQA_artificial_intelligence',
'ViDoRe_syntheticDocQA_energy',
'ViDoRe_syntheticDocQA_government_reports',
'ViDoRe_syntheticDocQA_healthcare_industry'],
'ViDoRe-V2': ['ViDoRe_esg_reports_human_labeled_v2',
'ViDoRe_biomedical_lectures_v2_multilingual',
'ViDoRe_economics_reports_v2_multilingual',
'ViDoRe_esg_reports_v2_multilingual'],
'VisDoc-OOD': ['ViDoSeek-page-fixed',
'ViDoSeek-doc',
'MMLongBench-page-fixed',
'MMLongBench-doc'],
'VisRAG': ['VisRAG_ArxivQA',
'VisRAG_ChartQA',
'VisRAG_MP-DocVQA',
'VisRAG_SlideVQA',
'VisRAG_InfoVQA',
'VisRAG_PlotQA']}}
Submit on MMEB Leaderboard Introduction
We have provided detailed step-by-step guides on how to submit your model. Please read carefully in order to submit successfully.
Step 1️⃣: Please refer to the GitHub page for detailed instructions about evaluating your model.
Step 2️⃣: After running the evaluation pipelines, please use the script we provided (e.g., report_score_v2.py) to generate the unified score sheet.
- Reminder: Adjust your model's configurations in the script, including model name, URL, model size, and data source.
- Note the "model size" field is digits-only and is by default in Billions (B), so please convert it accordingly if your model size is in different units/formats (e.x., "8" for 8 billion, "0.5" for 500 million, don't include non-digits in it).
- If possible, please also add a contact method in case we want to reach you in the future
- Reminder: Adjust your model's configurations in the script, including model name, URL, model size, and data source.
Step 3️⃣: Finally, create a pull request and upload the generated JSON file to the scores folder.
- You can directly use the Hugging Face Space's web UI to upload your files:
- Go to the scores folder
- Select "Upload file" and upload your JSON files.
- Please inform us on our discord server, or send us an email at m7su@uwaterloo.ca, or leave a simple message (@ us) in the PR to indicate that you are ready to merge your PR.
- We will then review your submission and update the leaderboard accordingly.
- You can directly use the Hugging Face Space's web UI to upload your files:
Please double check your score sheet have the following JSON format ⬇️:
{
"metadata": {
"model_name": "<Model Name>",
"url": "<Model URL>" or null,
"model_size": <Model Size> or null,
"data_source": "Self-Reported",
"contact": xxx@gmail.com
... ...
},
"metrics": {
"image": {
"ImageNet-1K": {
"hit@1": 0.5,
"ndcg@1": 0.5,
... ...
},
"N24News": {
... ...
},
... ...
},
"video": {
... ...
},
... ...
}
}
⚠️ Special Instructions for submitting to MMEB Image (Previously MMEB-V1) Leaderboard
We understand that some researchers want to exclusively submit to the Image leaderboard, but unfortunately our current leaderboard system cannot exclude your model from other modalities' leaderboards. To do so, please run the 36 image datasets only and simply ignore the other datasets. The leaderboard will automatically assign a 0 to the missing datasets. We might be able to hide your model from the other leaderboards in the next leaderboard updates, but for now your model will be shown on all leaderboards and might have a lower rank.
We highly recommend joining our discord server, which provides a convenient way to stay informed with latest updates, or share any feedback you have for improving the leaderboard experience. We appreciate your contributions to the MMEB community!