🇷🇸 New Benchmark for Serbian Language 🇷🇸
text
🇷🇸 New Benchmark for Serbian Language 🇷🇸
new_line
new_line
@DjMel
mention
DjMel
 and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.
text
 and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.
new_line
new_line
🥇 OAI models still hold the podium places with a significant gap compared to open-source models
text
🥇 OAI models still hold the podium places with a significant gap compared to open-source models
new_line
🤔 
text
🤔 
https://huggingface.co/Qwen/Qwen2-7B-Instruct
Qwen/Qwen2-7B-Instruct
model
resource
https://huggingface.co/Qwen/Qwen2-7B-Instruct
 and 
text
 and 
https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
model
resource
https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
 models show promising results considering they weren't trained on Serbian language
text
 models show promising results considering they weren't trained on Serbian language
new_line
📈 Best open-source model seems to be 
text
📈 Best open-source model seems to be 
https://huggingface.co/Stopwolf/Mustra-7B-Instruct-v0.2
Stopwolf/Mustra-7B-Instruct-v0.2
model
resource
https://huggingface.co/Stopwolf/Mustra-7B-Instruct-v0.2
, a merge between 
text
, a merge between 
https://huggingface.co/gordicaleksa/YugoGPT
gordicaleksa/YugoGPT
model
resource
https://huggingface.co/gordicaleksa/YugoGPT
 and 
text
 and 
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
mistralai/Mistral-7B-Instruct-v0.2
model
resource
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
new_line
📉 Some models like 
text
📉 Some models like 
https://huggingface.co/google/gemma-2-9b-it
google/gemma-2-9b-it
model
resource
https://huggingface.co/google/gemma-2-9b-it
 turned out to be a disappointment with random guessing-like accuracy
text
 turned out to be a disappointment with random guessing-like accuracy
new_line
new_line
Take a look at the whole results at the dataset page:
text
Take a look at the whole results at the dataset page:
new_line
https://huggingface.co/datasets/DjMel/oz-eval
DjMel/oz-eval
dataset
resource
https://huggingface.co/datasets/DjMel/oz-eval
new_line
new_line
P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!
text
P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!
🇷🇸 New Benchmark for Serbian Language 🇷🇸

@DjMel and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.

🥇 OAI models still hold the podium places with a significant gap compared to open-source models
🤔 https://huggingface.co/Qwen/Qwen2-7B-Instruct and https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct models show promising results considering they weren't trained on Serbian language
📈 Best open-source model seems to be https://huggingface.co/Stopwolf/Mustra-7B-Instruct-v0.2, a merge between https://huggingface.co/gordicaleksa/YugoGPT and https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
📉 Some models like https://huggingface.co/google/gemma-2-9b-it turned out to be a disappointment with random guessing-like accuracy

Take a look at the whole results at the dataset page:
https://huggingface.co/datasets/DjMel/oz-eval

P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!
🚀
DjMel Ramikan-BR intellya22
👍
monsoon-nlp Ramikan-BR