Very interesting model just released by MyShell: 
text
Very interesting model just released by MyShell: 
https://huggingface.co/jetmoe/jetmoe-8b
jetmoe/jetmoe-8b
model
resource
https://huggingface.co/jetmoe/jetmoe-8b
 . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.
text
 . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.
new_line
new_line
Main characteristics:
text
Main characteristics:
new_line
- impressive performances for its size (beating 
text
- impressive performances for its size (beating 
https://huggingface.co/meta-llama/Llama-2-7b
meta-llama/Llama-2-7b
model
resource
https://huggingface.co/meta-llama/Llama-2-7b
 and 
text
 and 
https://huggingface.co/huggyllama/llama-13b
huggyllama/llama-13b
model
resource
https://huggingface.co/huggyllama/llama-13b
)
text
)
new_line
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
text
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
new_line
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
text
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
new_line
- $100k to train
text
- $100k to train
new_line
- open weights - open sharing of recipes - open dataset - open code => ♡
text
- open weights - open sharing of recipes - open dataset - open code => ♡
new_line
- still interesting room to improve performances (be it only by training longer)
text
- still interesting room to improve performances (be it only by training longer)
new_line
new_line
Links:
text
Links:
new_line
- report: 
text
- report: 
https://research.myshell.ai/jetmoe
https://research.myshell.ai/jetmoe
link
new_line
- model: 
text
- model: 
https://huggingface.co/jetmoe/jetmoe-8b
jetmoe/jetmoe-8b
model
resource
https://huggingface.co/jetmoe/jetmoe-8b
new_line
- code: 
text
- code: 
https://github.com/myshell-ai/JetMoE
https://github.com/myshell-ai/JetMoE
link
new_line
new_line
Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: 
text
Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: 
https://www.youtube.com/watch?v=2-SPH9hIKT8
https://www.youtube.com/watch?v=2-SPH9hIKT8
link
Very interesting model just released by MyShell: https://huggingface.co/jetmoe/jetmoe-8b . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.

Main characteristics:
- impressive performances for its size (beating https://huggingface.co/meta-llama/Llama-2-7b and https://huggingface.co/huggyllama/llama-13b)
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
- $100k to train
- open weights - open sharing of recipes - open dataset - open code => ♡
- still interesting room to improve performances (be it only by training longer)

Links:
- report: https://research.myshell.ai/jetmoe
- model: https://huggingface.co/jetmoe/jetmoe-8b
- code: https://github.com/myshell-ai/JetMoE

Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: https://www.youtube.com/watch?v=2-SPH9hIKT8
🚀
samusenps a9i giux78 dillfrescott clem osanseviero
❤️
samusenps ajibawa-2023 dillfrescott clem osanseviero
64aea8ff67511bd3d965697b
https://cdn-avatars.huggingface.co/v1/production/uploads/64aea8ff67511bd3d965697b/Jxn52EmDF5RApJh8antxn.jpeg
Feynman Innovations
ajibawa-2023
user