Intermediate Checkpoints
I understand it may be cumbersome, but has the team considered releasing more intermediate checkpoints? IF you have them on hand or are training another one, consider releasing checkpoints in closer intervals similar to Pythia.
Yes, it is entirely possible to release more intermediate checkpoints, e.g. 500B, 1T, 1.5T, 2T, 2.5T, 3T. Would that meet your needs?
That would be great! My proposal is that there is a new repository called "Skywork-13B-Base-Intermediate" where every checkpoint saved as a different branch in one repo. https://huggingface.co/EleutherAI/pythia-70m That would be a great effort to make a good alternative to Pythia. If it's saved and not critical to anything Huggingface storage is free anyways so might as well put all you have. Just my opinion. Thank you for responding!
That would be great! My proposal is that there is a new repository called "Skywork-13B-Base-Intermediate" where every checkpoint saved as a different branch in one repo. https://huggingface.co/EleutherAI/pythia-70m That would be a great effort to make a good alternative to Pythia. If it's saved and not critical to anything Huggingface storage is free anyways so might as well put all you have. Just my opinion. Thank you for responding!
Hi there, we have uploaded intermediate checkpoints in the following repo: https://huggingface.co/Skywork/Skywork-13B-Base-Intermediate.
Hope that helps!
I deeply appreciate the release of these intermediate checkpoints and I am excited to use them for research. Thank you for working with me and generally the community to make AI transparent and open as it always should be.