A Little guide to building Large Language Models in 2024
text
A Little guide to building Large Language Models in 2024
new_line
new_line
This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.
text
This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.
new_line
new_line
In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
text
In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
new_line
* finding, preparing and evaluating web scale data
text
* finding, preparing and evaluating web scale data
new_line
* understanding model parallelism and efficient training
text
* understanding model parallelism and efficient training
new_line
* fine-tuning/aligning models
text
* fine-tuning/aligning models
new_line
* fast inference
text
* fast inference
new_line
new_line
There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.
text
There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.
new_line
new_line
Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
text
Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
new_line
*
text
*
datatrove
`datatrove`
inline_code
 for all things web-scale data preparation: 
text
 for all things web-scale data preparation: 
https://github.com/huggingface/datatrove
https://github.com/huggingface/datatrove
link
new_line
*
text
*
nanotron
`nanotron`
inline_code
 for lightweight 4D parallelism LLM training: 
text
 for lightweight 4D parallelism LLM training: 
https://github.com/huggingface/nanotron
https://github.com/huggingface/nanotron
link
new_line
*
text
*
lighteval
`lighteval`
inline_code
 for in-training fast parallel LLM evaluations: 
text
 for in-training fast parallel LLM evaluations: 
https://github.com/huggingface/lighteval
https://github.com/huggingface/lighteval
link
new_line
new_line
Here is the link to watch the lecture on Youtube: 
text
Here is the link to watch the lecture on Youtube: 
https://www.youtube.com/watch?v=2-SPH9hIKT8
https://www.youtube.com/watch?v=2-SPH9hIKT8
link
new_line
And here is the link to the Google slides: 
text
And here is the link to the Google slides: 
https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p
https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p
link
new_line
new_line
Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
text
Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
A Little guide to building Large Language Models in 2024

This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.

In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
* finding, preparing and evaluating web scale data
* understanding model parallelism and efficient training
* fine-tuning/aligning models
* fast inference

There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.

Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
*`datatrove` for all things web-scale data preparation: https://github.com/huggingface/datatrove
*`nanotron` for lightweight 4D parallelism LLM training: https://github.com/huggingface/nanotron
*`lighteval` for in-training fast parallel LLM evaluations: https://github.com/huggingface/lighteval

Here is the link to watch the lecture on Youtube: https://www.youtube.com/watch?v=2-SPH9hIKT8
And here is the link to the Google slides: https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p

Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
❤️
clem osanseviero giux78 samusenps dillfrescott mohammedbriman reach-vb rwightman Venkman42 gusthema 9bow amenur lisuizhe The-Great-Genius danja drFarid MicheleStingo chkla Skier8402 Artples alemiaschi alirezamsh catastropiyush dlicari iproskurina vikas mkhalifa mertege ianlee1996 LiuPengNGP salkhon jdzw2014 serdarcaglar ma7583 derek-thomas fpadovani zackli4ai Davidqian123 JoyboyBrian alanzhuly eugrug-60
🔥
clem osanseviero giux78 nikgr taufiqdp tvergho dillfrescott noh-au-00 reach-vb ajibawa-2023 wip87 thisisanshgupta RichardForests Andron00e 9bow lisuizhe santyzenith yusukewm Ashu212 sid-cypher davzoku gigant lorinma serdarcaglar dnabanita7 derek-thomas zackli4ai Davidqian123 JoyboyBrian
👍
peilin23 dillfrescott reach-vb 9bow jpoag lisuizhe shiv2050 ssml2050 dsindex ad1555 derek-thomas zackli4ai Davidqian123 alexchen4ai JoyboyBrian
🚀
JoyboyBrian
64bec91d76a6e2efccd4ad21
/avatars/cc5c07cc99c647da19cff5219f43869d.svg
noh au
noh-au-00
user
651baa9837fecec1fe84034d
https://cdn-avatars.huggingface.co/v1/production/uploads/651baa9837fecec1fe84034d/bnbi-vyKcbU5OOnqvJpgI.jpeg
Piyush Maharana
catastropiyush
user