Google's SigLIP is another alternative to openai's CLIP, and it  just got merged to 🤗transformers and it's super easy to use!
text
Google's SigLIP is another alternative to openai's CLIP, and it  just got merged to 🤗transformers and it's super easy to use!
new_line
To celebrate this, I have created a repository including notebooks and bunch of Spaces on various SigLIP based projects 🥳  
text
To celebrate this, I have created a repository including notebooks and bunch of Spaces on various SigLIP based projects 🥳  
new_line
Search for art 👉 
text
Search for art 👉 
https://huggingface.co/spaces/merve/draw_to_search_art
merve/draw_to_search_art
space
resource
https://huggingface.co/spaces/merve/draw_to_search_art
new_line
Compare SigLIP with CLIP 👉 
text
Compare SigLIP with CLIP 👉 
https://huggingface.co/spaces/merve/compare_clip_siglip
merve/compare_clip_siglip
space
resource
https://huggingface.co/spaces/merve/compare_clip_siglip
new_line
new_line
How does SigLIP work? 
text
How does SigLIP work? 
new_line
SigLIP an vision-text pre-training technique based on contrastive learning. It jointly trains an image encoder and text encoder such that the dot product of embeddings are most similar for the appropriate text-image pairs
text
SigLIP an vision-text pre-training technique based on contrastive learning. It jointly trains an image encoder and text encoder such that the dot product of embeddings are most similar for the appropriate text-image pairs
new_line
The image below is taken from CLIP, where this contrastive pre-training takes place with softmax, but SigLIP replaces softmax with sigmoid. 📎  
text
The image below is taken from CLIP, where this contrastive pre-training takes place with softmax, but SigLIP replaces softmax with sigmoid. 📎  
new_line
new_line
Highlights from the paper on why you should use it ✨
text
Highlights from the paper on why you should use it ✨
new_line
🖼️📝 Authors used medium sized B/16 ViT for image encoder and B-sized transformer for text encoder
text
🖼️📝 Authors used medium sized B/16 ViT for image encoder and B-sized transformer for text encoder
new_line
😍 More performant than CLIP on zero-shot
text
😍 More performant than CLIP on zero-shot
new_line
🗣️ Authors trained a multilingual model too!
text
🗣️ Authors trained a multilingual model too!
new_line
⚡️ Super efficient, sigmoid is enabling up to 1M items per batch, but the authors chose 32k because the performance saturates after that
text
⚡️ Super efficient, sigmoid is enabling up to 1M items per batch, but the authors chose 32k because the performance saturates after that
new_line
new_line
It's super easy to use thanks to transformers 👇 
text
It's super easy to use thanks to transformers 👇 
new_line
from transformers import pipeline
from PIL import Image
import requests

# load pipe
image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-256-i18n")

# load image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# inference
outputs = image_classifier(image, candidate_labels=["2 cats", "a plane", "a remote"])
outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
print(outputs)
python
```python
from transformers import pipeline
from PIL import Image
import requests

# load pipe
image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-256-i18n")

# load image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# inference
outputs = image_classifier(image, candidate_labels=["2 cats", "a plane", "a remote"])
outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
print(outputs)
```
code_fence
new_line
For all the SigLIP notebooks on similarity search and indexing, you can check this [repository](
text
For all the SigLIP notebooks on similarity search and indexing, you can check this [repository](
https://github.com/merveenoyan/siglip
https://github.com/merveenoyan/siglip
link
) out. 🤗 
text
) out. 🤗 
Google's SigLIP is another alternative to openai's CLIP, and it  just got merged to 🤗transformers and it's super easy to use!
To celebrate this, I have created a repository including notebooks and bunch of Spaces on various SigLIP based projects 🥳  
Search for art 👉 https://huggingface.co/spaces/merve/draw_to_search_art
Compare SigLIP with CLIP 👉 https://huggingface.co/spaces/merve/compare_clip_siglip

How does SigLIP work? 
SigLIP an vision-text pre-training technique based on contrastive learning. It jointly trains an image encoder and text encoder such that the dot product of embeddings are most similar for the appropriate text-image pairs
The image below is taken from CLIP, where this contrastive pre-training takes place with softmax, but SigLIP replaces softmax with sigmoid. 📎  

Highlights from the paper on why you should use it ✨
🖼️📝 Authors used medium sized B/16 ViT for image encoder and B-sized transformer for text encoder
😍 More performant than CLIP on zero-shot
🗣️ Authors trained a multilingual model too!
⚡️ Super efficient, sigmoid is enabling up to 1M items per batch, but the authors chose 32k because the performance saturates after that

It's super easy to use thanks to transformers 👇 
```python
from transformers import pipeline
from PIL import Image
import requests

# load pipe
image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-256-i18n")

# load image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# inference
outputs = image_classifier(image, candidate_labels=["2 cats", "a plane", "a remote"])
outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
print(outputs)
```
For all the SigLIP notebooks on similarity search and indexing, you can check this [repository](https://github.com/merveenoyan/siglip) out. 🤗 
👍
dreamworks2050 victor dvilasuero julien-c osanseviero samusenps Leyo abidlabs TuringsSolutions chansung merve Kernel cdr6934 clem cataluna84 Amin24 AndresJara
❤️
samusenps TuringsSolutions pryanshusharma rwightman chansung linoyts SRDdev akashicmarga yihaopeng clem afrideva cataluna84 AndresJara
🤝
nezubn VictorSanh dvilasuero julien-c osanseviero TuringsSolutions chansung clem cataluna84 AndresJara
5dd96eb166059660ed1ee413
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg
Julien Chaumond
julien-c
user
60d3b57ad7b174177faabd6e
https://cdn-avatars.huggingface.co/v1/production/uploads/1659971187637-60d3b57ad7b174177faabd6e.jpeg
chansung park
chansung
user
67c99dddafe823d480ecc008
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/aJGAWmr9udQk8nOCxleTF.png
Andres Emanuel Jara
AndresJara
user