Could more DPO-style preference data be crucial for enhancing open LLMs across different languages? 
text
Could more DPO-style preference data be crucial for enhancing open LLMs across different languages? 
new_line
new_line
Leveraging a 7k preference dataset Argilla (
text
Leveraging a 7k preference dataset Argilla (
@alvarobartt
mention
alvarobartt
), Hugging Face (
text
), Hugging Face (
@lewtun
mention
lewtun
) and Kaist AI (
text
) and Kaist AI (
@JW17
mention
JW17
 & 
text
 & 
@nlee-208
mention
nlee-208
)
text
)
new_line
utilized Kaist AI's recently introduced ORPO technique 
text
utilized Kaist AI's recently introduced ORPO technique 
ORPO: Monolithic Preference Optimization without Reference Model (2403.07691)
https://huggingface.co/papers/2403.07691
2403.07691
paper
resource
https://huggingface.co/papers/2403.07691
 with the latest MistralAI MOE model to create a very high-performing open LLM: 
text
 with the latest MistralAI MOE model to create a very high-performing open LLM: 
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
model
resource
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
new_line
new_line
Since ORPO doesn't require a separate SFT stage, all that is needed is a strong base model + high-quality DPO-style datasets.
text
Since ORPO doesn't require a separate SFT stage, all that is needed is a strong base model + high-quality DPO-style datasets.
new_line
new_line
Currently, there is a significant lack of non-English DPO datasets. Filling this gap could significantly improve open LLMs in various languages.
text
Currently, there is a significant lack of non-English DPO datasets. Filling this gap could significantly improve open LLMs in various languages.
new_line
new_line
You can get an overview of the current state of DPO datasets across different languages here: 
text
You can get an overview of the current state of DPO datasets across different languages here: 
https://huggingface.co/spaces/DIBT/preference_data_by_language
https://huggingface.co/spaces/DIBT/preference_data_by_language
link
text
Could more DPO-style preference data be crucial for enhancing open LLMs across different languages? 

Leveraging a 7k preference dataset Argilla (@alvarobartt), Hugging Face (@lewtun) and Kaist AI (@JW17 & @nlee-208)
utilized Kaist AI's recently introduced ORPO technique https://huggingface.co/papers/2403.07691 with the latest MistralAI MOE model to create a very high-performing open LLM: https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1

Since ORPO doesn't require a separate SFT stage, all that is needed is a strong base model + high-quality DPO-style datasets.

Currently, there is a significant lack of non-English DPO datasets. Filling this gap could significantly improve open LLMs in various languages.

You can get an overview of the current state of DPO datasets across different languages here: https://huggingface.co/spaces/DIBT/preference_data_by_language 
🚀
JW17 alvarobartt osanseviero nlee-208 NMontanaBrown Violette Heng666 raincandy-u DataSoul