My latest project is the outcome of the last 2+ years working with TPUs from the amazing TPU Research Cloud (TRC) program and training Encoder-only LMs with the TensorFlow Model Garden library.
text
My latest project is the outcome of the last 2+ years working with TPUs from the amazing TPU Research Cloud (TRC) program and training Encoder-only LMs with the TensorFlow Model Garden library.
new_line
new_line
👉 Link: 
text
👉 Link: 
https://github.com/stefan-it/model-garden-lms
https://github.com/stefan-it/model-garden-lms
link
new_line
new_line
An overview of some features:
text
An overview of some features:
new_line
new_line
- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
text
- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
new_line
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
text
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
new_line
- Supported architectures include BERT, BERT with Token Dropping and TEAMS
text
- Supported architectures include BERT, BERT with Token Dropping and TEAMS
new_line
new_line
I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!
text
I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!
new_line
new_line
👉 Model Hub Link: 
text
👉 Model Hub Link: 
https://cdn-avatars.huggingface.co/v1/production/uploads/5e6a3d4ea9afd5125d9ec064/LD0Zc-cz-VZpqYi3qvhQ8.png
https://huggingface.co/model-garden-lms
model-garden-lms
org
resource
https://huggingface.co/model-garden-lms
new_line
new_line
If you find these resources useful, please give them a like!
text
If you find these resources useful, please give them a like!
new_line
new_line
Made from Bavarian Oberland with ❤️ and 🥨.
text
Made from Bavarian Oberland with ❤️ and 🥨.
My latest project is the outcome of the last 2+ years working with TPUs from the amazing TPU Research Cloud (TRC) program and training Encoder-only LMs with the TensorFlow Model Garden library.

👉 Link: https://github.com/stefan-it/model-garden-lms

An overview of some features:

- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
- Supported architectures include BERT, BERT with Token Dropping and TEAMS

I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!

👉 Model Hub Link: https://huggingface.co/model-garden-lms

If you find these resources useful, please give them a like!

Made from Bavarian Oberland with ❤️ and 🥨.
❤️
davanstrien mochammadasholeh s3nh hannayukhymenko jqop
👀
John6666 jqop