New foundation model on image and video captioning just dropped by NVIDIA AI 🔥
text
New foundation model on image and video captioning just dropped by NVIDIA AI 🔥
new_line
new_line
Describe Anything Model (DAM) is a 3B vision language model to generate detailed captions with localized references 😮
text
Describe Anything Model (DAM) is a 3B vision language model to generate detailed captions with localized references 😮
new_line
new_line
The team released the models, the dataset, a new benchmark and a demo 🤩 
text
The team released the models, the dataset, a new benchmark and a demo 🤩 
https://huggingface.co/collections/nvidia/describe-anything-680825bb8f5e41ff0785834c
nvidia/describe-anything-680825bb8f5e41ff0785834c
collection
resource
https://huggingface.co/collections/nvidia/describe-anything-680825bb8f5e41ff0785834c
new_line
new_line
Most of the vision LMs focus on image as a whole, lacking localized references in captions, and not taking in visual prompts (points, boxes, drawings around objects)
text
Most of the vision LMs focus on image as a whole, lacking localized references in captions, and not taking in visual prompts (points, boxes, drawings around objects)
new_line
new_line
DAM addresses this on two levels: new vision backbone that takes in focal crops and the image itself, and a large scale dataset 👀 
text
DAM addresses this on two levels: new vision backbone that takes in focal crops and the image itself, and a large scale dataset 👀 
new_line
new_line
They generate a dataset by extending existing segmentation and referring expression generation datasets like REFCOCO, by passing in the images and classes to VLMs and generating captions.
text
They generate a dataset by extending existing segmentation and referring expression generation datasets like REFCOCO, by passing in the images and classes to VLMs and generating captions.
new_line
new_line
Lastly, they also release a new benchmark again with self-supervision, they use an LLM to evaluate the detailed captions focusing on localization 👏
text
Lastly, they also release a new benchmark again with self-supervision, they use an LLM to evaluate the detailed captions focusing on localization 👏
New foundation model on image and video captioning just dropped by NVIDIA AI 🔥

Describe Anything Model (DAM) is a 3B vision language model to generate detailed captions with localized references 😮

The team released the models, the dataset, a new benchmark and a demo 🤩 https://huggingface.co/collections/nvidia/describe-anything-680825bb8f5e41ff0785834c

Most of the vision LMs focus on image as a whole, lacking localized references in captions, and not taking in visual prompts (points, boxes, drawings around objects)

DAM addresses this on two levels: new vision backbone that takes in focal crops and the image itself, and a large scale dataset 👀 

They generate a dataset by extending existing segmentation and referring expression generation datasets like REFCOCO, by passing in the images and classes to VLMs and generating captions.

Lastly, they also release a new benchmark again with self-supervision, they use an LLM to evaluate the detailed captions focusing on localization 👏
🔥
John6666 foonom BrigitteTousi tebal91901 tjdoomer m8x7b CCP6 Viewegger Rsln Vaporeonn