🚀 I built a Multimodal Vision-Language Model from using Gemma-270M + CLIP!
text
🚀 I built a Multimodal Vision-Language Model from using Gemma-270M + CLIP!
new_line
new_line
Just finished training my multimodal model on the full LLaVA-Instruct-150K dataset (157K samples) and wanted to share the results!
text
Just finished training my multimodal model on the full LLaVA-Instruct-150K dataset (157K samples) and wanted to share the results!
new_line
new_line
🔧 What I Built:
text
🔧 What I Built:
new_line
A vision-language model that can understand images and answer questions about them, combining:
text
A vision-language model that can understand images and answer questions about them, combining:
new_line
- Google Gemma-3-270M (language)
text
- Google Gemma-3-270M (language)
new_line
- OpenAI CLIP ViT-Large/14 (vision)
text
- OpenAI CLIP ViT-Large/14 (vision)
new_line
- LoRA fine-tuning for efficiency
text
- LoRA fine-tuning for efficiency
new_line
new_line
📊  Training Stats:
text
📊  Training Stats:
new_line
- 157,712 training samples (full LLaVA dataset)
text
- 157,712 training samples (full LLaVA dataset)
new_line
- 3 epochs on A100 40GB
text
- 3 epochs on A100 40GB
new_line
- ~9 hours training time
text
- ~9 hours training time
new_line
- Final loss: 1.333 training / 1.430 validation
text
- Final loss: 1.333 training / 1.430 validation
new_line
- Only 18.6M trainable params (3.4% of 539M total)
text
- Only 18.6M trainable params (3.4% of 539M total)
new_line
new_line
📈  
text
📈  
https://huggingface.co/sagar007/multigemma
sagar007/multigemma
model
resource
https://huggingface.co/sagar007/multigemma
new_line
Benchmark Results:
text
Benchmark Results:
new_line
- VQA Accuracy: 53.8%
text
- VQA Accuracy: 53.8%
new_line
- Works great for: animal detection, room identification, scene understanding
text
- Works great for: animal detection, room identification, scene understanding
new_line
new_line
new_line
new_line
🔗 **Try it yourself:**
text
🔗 **Try it yourself:**
new_line
- 🤗 Model: 
text
- 🤗 Model: 
https://huggingface.co/sagar007/multigemma
sagar007/multigemma
model
resource
https://huggingface.co/sagar007/multigemma
new_line
- 🎮 Demo: 
text
- 🎮 Demo: 
https://huggingface.co/spaces/sagar007/Multimodal-Gemma
https://huggingface.co/spaces/sagar007/Multimodal-Gemma
link
new_line
- 💻 GitHub: 
text
- 💻 GitHub: 
https://github.com/sagar431/multimodal-gemma-270m
https://github.com/sagar431/multimodal-gemma-270m
link
new_line
new_line
Built with PyTorch Lightning + MLflow for experiment tracking. Full MLOps pipeline with CI/CD!
text
Built with PyTorch Lightning + MLflow for experiment tracking. Full MLOps pipeline with CI/CD!
new_line
new_line
Would love to hear your feedback! 🙏
text
Would love to hear your feedback! 🙏
new_line
new_line
#multimodal #gemma #clip #llava #vision-language #pytorch
text
#multimodal #gemma #clip #llava #vision-language #pytorch
🚀 I built a Multimodal Vision-Language Model from using Gemma-270M + CLIP!

Just finished training my multimodal model on the full LLaVA-Instruct-150K dataset (157K samples) and wanted to share the results!

🔧 What I Built:
A vision-language model that can understand images and answer questions about them, combining:
- Google Gemma-3-270M (language)
- OpenAI CLIP ViT-Large/14 (vision)
- LoRA fine-tuning for efficiency

📊  Training Stats:
- 157,712 training samples (full LLaVA dataset)
- 3 epochs on A100 40GB
- ~9 hours training time
- Final loss: 1.333 training / 1.430 validation
- Only 18.6M trainable params (3.4% of 539M total)

📈  https://huggingface.co/sagar007/multigemma
Benchmark Results:
- VQA Accuracy: 53.8%
- Works great for: animal detection, room identification, scene understanding



🔗 **Try it yourself:**
- 🤗 Model: https://huggingface.co/sagar007/multigemma
- 🎮 Demo: https://huggingface.co/spaces/sagar007/Multimodal-Gemma
- 💻 GitHub: https://github.com/sagar431/multimodal-gemma-270m

Built with PyTorch Lightning + MLflow for experiment tracking. Full MLOps pipeline with CI/CD!

Would love to hear your feedback! 🙏

#multimodal #gemma #clip #llava #vision-language #pytorch
🔥
Sri-Vigneshwar-DJ Dreanhunter30 sirev cmacha2 Cameron1334 cyberunit enzoescipy edwixx williamliao John6666 Norod78 ares2324 akahana
🤝
hackeronetestt Kito12 Dreanhunter30 edwixx
👍
Mindweller
6758a9850e3fff481964ca6d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/EolfJfjW25hC4Bt_hCPq8.png
Jean Louis
JLouisBiz
user
62a464cfe0de0c5c6d8b04a1
https://cdn-avatars.huggingface.co/v1/production/uploads/62a464cfe0de0c5c6d8b04a1/1gCs46R_bW9apQzLQUrn5.png
Sagar pallai
sagar007
user
66b6ab42e5492aaa883315b9
/avatars/9250cc22b53a5c93b7c6e46a71b30d74.svg
Warshawsky
EladofWar
user
68f4abf8f64bb4002a21a428
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qoGuyvqteEz6gnrIxROr-.png
Boning Cui
Bc-AI
user