Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
text
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
new_line
new_line
UCLA-AGI has proven that large language models, even weaker large language models, can improve themselves with data only produced by original model. The question they answer in their paper is:
text
UCLA-AGI has proven that large language models, even weaker large language models, can improve themselves with data only produced by original model. The question they answer in their paper is:
new_line
new_line
"Can we empower a weak LLM to improve itself without acquiring additional human annotated data?"
text
"Can we empower a weak LLM to improve itself without acquiring additional human annotated data?"
new_line
new_line
They answer this question by the proposal and testing of a novel fine-tuning method they call Self-Play fIne-tuNing (SPIN). The process starts by applying a supervised fine-tune (SFT) to zephyr-7b using all 200k samples of HuggingfaceH4/ultrachat_200k to eliminate the need for a human annotator.
text
They answer this question by the proposal and testing of a novel fine-tuning method they call Self-Play fIne-tuNing (SPIN). The process starts by applying a supervised fine-tune (SFT) to zephyr-7b using all 200k samples of HuggingfaceH4/ultrachat_200k to eliminate the need for a human annotator.
new_line
new_line
Once the model has completed SFT, the SPIN method suggests generating 50k samples of synthetic data pairs of 'chosen' and 'rejected' samples. The model will be fine tuned on those generations, and this process will repeat for another 3 iterations for a total 200k samples.
text
Once the model has completed SFT, the SPIN method suggests generating 50k samples of synthetic data pairs of 'chosen' and 'rejected' samples. The model will be fine tuned on those generations, and this process will repeat for another 3 iterations for a total 200k samples.
new_line
new_line
This experiment is unique because they propose that their method can yield upwards of 10% performance gains without using any additional human annotated data. The strategy was designed to improve the less strong language models, but with further experimentation could be a formidable strategy for improving language models.
text
This experiment is unique because they propose that their method can yield upwards of 10% performance gains without using any additional human annotated data. The strategy was designed to improve the less strong language models, but with further experimentation could be a formidable strategy for improving language models.
new_line
new_line
If you would like to explore this strategy for yourself, here are some resources:
text
If you would like to explore this strategy for yourself, here are some resources:
new_line
Colab: 
text
Colab: 
https://colab.research.google.com/drive/1IjDeNVBsRru2-hM_9aauD6gVI-VvJnpk?usp=sharing
https://colab.research.google.com/drive/1IjDeNVBsRru2-hM_9aauD6gVI-VvJnpk?usp=sharing
link
new_line
Github: 
text
Github: 
https://github.com/uclaml/SPIN
https://github.com/uclaml/SPIN
link
new_line
The product of the experiment: 
text
The product of the experiment: 
https://huggingface.co/UCLA-AGI/zephyr-7b-sft-full-SPIN-iter3
UCLA-AGI/zephyr-7b-sft-full-SPIN-iter3
model
resource
https://huggingface.co/UCLA-AGI/zephyr-7b-sft-full-SPIN-iter3
new_line
Paper: 
text
Paper: 
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models (2401.01335)
https://huggingface.co/papers/2401.01335
2401.01335
paper
resource
https://huggingface.co/papers/2401.01335
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

UCLA-AGI has proven that large language models, even weaker large language models, can improve themselves with data only produced by original model. The question they answer in their paper is:

"Can we empower a weak LLM to improve itself without acquiring additional human annotated data?"

They answer this question by the proposal and testing of a novel fine-tuning method they call Self-Play fIne-tuNing (SPIN). The process starts by applying a supervised fine-tune (SFT) to zephyr-7b using all 200k samples of HuggingfaceH4/ultrachat_200k to eliminate the need for a human annotator.

Once the model has completed SFT, the SPIN method suggests generating 50k samples of synthetic data pairs of 'chosen' and 'rejected' samples. The model will be fine tuned on those generations, and this process will repeat for another 3 iterations for a total 200k samples.

This experiment is unique because they propose that their method can yield upwards of 10% performance gains without using any additional human annotated data. The strategy was designed to improve the less strong language models, but with further experimentation could be a formidable strategy for improving language models.

If you would like to explore this strategy for yourself, here are some resources:
Colab: https://colab.research.google.com/drive/1IjDeNVBsRru2-hM_9aauD6gVI-VvJnpk?usp=sharing
Github: https://github.com/uclaml/SPIN
The product of the experiment: https://huggingface.co/UCLA-AGI/zephyr-7b-sft-full-SPIN-iter3
Paper: https://huggingface.co/papers/2401.01335
❤️
osanseviero vladbogo Ji-Ha clem samusenps danielus Dlbk julien-c abidlabs RedSparkie
6455cc8d679315e4ef16fbec
https://cdn-avatars.huggingface.co/v1/production/uploads/6455cc8d679315e4ef16fbec/M6Cfifn05BUzkCFd2QDIT.png
Tim Dolan
macadeliccc
user
6448b6de4988ee01f2a9c5ad
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/m8OOUN80ktQ5sb9NXZUfw.png
Brandon "Dimentox" Husbands
dimentox
user