Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍
text
Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍
new_line
A vision language model that comes in 7B and 34B sizes 🤩
text
A vision language model that comes in 7B and 34B sizes 🤩
new_line
But what makes this model so special?  
text
But what makes this model so special?  
new_line
new_line
Demo: 
text
Demo: 
https://huggingface.co/spaces/merve/chameleon-7b
merve/chameleon-7b
space
resource
https://huggingface.co/spaces/merve/chameleon-7b
new_line
Models: 
text
Models: 
https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58
facebook/chameleon-668da9663f80d483b4c61f58
collection
resource
https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58
new_line
new_line
keep reading ⥥
text
keep reading ⥥
new_line
new_line
Chameleon is a unique model: it attempts to scale early fusion 🤨
text
Chameleon is a unique model: it attempts to scale early fusion 🤨
new_line
But what is early fusion?
text
But what is early fusion?
new_line
Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM)
text
Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM)
new_line
new_line
Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 
text
Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 
new_line
new_line
Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO)
text
Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO)
new_line
new_line
This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. 
text
This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. 
new_line
new_line
One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks 
text
One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks 
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2405.09818)
https://huggingface.co/papers/2405.09818
2405.09818
paper
resource
https://huggingface.co/papers/2405.09818
)
text
)
new_line
Thanks for reading!
text
Thanks for reading!
Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍
A vision language model that comes in 7B and 34B sizes 🤩
But what makes this model so special?  

Demo: https://huggingface.co/spaces/merve/chameleon-7b
Models: https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58

keep reading ⥥

Chameleon is a unique model: it attempts to scale early fusion 🤨
But what is early fusion?
Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM)

Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 

Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO)

This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. 

One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks https://huggingface.co/papers/2405.09818)
Thanks for reading!
🔥
sumandas atahanuz Nymbo ucsahin GPT007 fdaudens osanseviero alkzar90 dblasko Svngoku gokaygokay mikestaub abdulbasit-nubytes
🤗
alanonbing
6141a88b3a0ec78603c9e784
https://cdn-avatars.huggingface.co/v1/production/uploads/6141a88b3a0ec78603c9e784/DJsxSmWV39M33JFheLobC.jpeg
merve
merve
user
66218234d00d7b6d99badeb0
/avatars/cfc399a521b05df08b799595b0390d13.svg
Prasanna Iyer
prasiyer
user
65d883893a52cd9bcd8ab7cf
https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
leroy Samuel Dyer
LeroyDyer
user
667e763c6f5dee59721f016d
https://cdn-avatars.huggingface.co/v1/production/uploads/667e763c6f5dee59721f016d/V2e6Z0BjzXE3t_obD3JXS.jpeg
Anastasia
Ana111op
user