 🎈 LLM Benchmarks Update!
text
 🎈 LLM Benchmarks Update!
new_line
new_line
**tl;dr: do not depend on benchmark leaderboards to choose your "chatbot" model! (Especially for non-English languages.)**
text
**tl;dr: do not depend on benchmark leaderboards to choose your "chatbot" model! (Especially for non-English languages.)**
new_line
new_line
First of all, I'm discontinuing the Open #Dutch #LLM Leaderboard (
text
First of all, I'm discontinuing the Open #Dutch #LLM Leaderboard (
https://lnkd.in/eFnsaFR6
https://lnkd.in/eFnsaFR6
link
). It will stay online for now, but I urge the use of the ScandEval leaderboard instead (
text
). It will stay online for now, but I urge the use of the ScandEval leaderboard instead (
https://scandeval.com/dutch-nlg/
https://scandeval.com/dutch-nlg/
link
) by 
text
) by 
@saattrupdan
mention
saattrupdan
. It contains more tasks, has better reproducibility and statistics (CI) and a flexible back-end library (
text
. It contains more tasks, has better reproducibility and statistics (CI) and a flexible back-end library (
scandeval
`scandeval`
inline_code
) to run your own benchmarks with. As part of project "Leesplank" (with Michiel Buisman and Maarten Lens-FitzGerald) we recently added GPT-4-1106-preview scores to add a good "target" to the leaderboard.
text
) to run your own benchmarks with. As part of project "Leesplank" (with Michiel Buisman and Maarten Lens-FitzGerald) we recently added GPT-4-1106-preview scores to add a good "target" to the leaderboard.
new_line
new_line
An important note here is that benchmark leaderboards are not a golden truth. Especially evaluating generative models is hard. You run into issues like prompt engineering (and sensitivity of models to one or other prompt), structured output generation, and - quite simply - "how to automatically evaluate open-ended generation".
text
An important note here is that benchmark leaderboards are not a golden truth. Especially evaluating generative models is hard. You run into issues like prompt engineering (and sensitivity of models to one or other prompt), structured output generation, and - quite simply - "how to automatically evaluate open-ended generation".
new_line
new_line
💡 Another important but under-discussed facet is the discrepancy between models' capability of understanding vs. generating *in different languages* (so the NLU part of NLG benchmarking). In other words: some of the listed models score really well on, e.g., MCQ benchmarks but are not suitable to use as DUTCH chat bots. Interestingly, some of these models seem to understand questions in Dutch and are able to pick the right answer (because they have good knowledge or reasoning skills), but generating fluent and grammatical Dutch is something else entirely! This is perhaps also true for humans: it's easier to sort-of grasp the meaning of a new language and answer with "Yes" or "No", but answering fluently in the language is much harder! Yet, your language production fluency does not necessarily say anything about your knowledge and reasoning skills.
text
💡 Another important but under-discussed facet is the discrepancy between models' capability of understanding vs. generating *in different languages* (so the NLU part of NLG benchmarking). In other words: some of the listed models score really well on, e.g., MCQ benchmarks but are not suitable to use as DUTCH chat bots. Interestingly, some of these models seem to understand questions in Dutch and are able to pick the right answer (because they have good knowledge or reasoning skills), but generating fluent and grammatical Dutch is something else entirely! This is perhaps also true for humans: it's easier to sort-of grasp the meaning of a new language and answer with "Yes" or "No", but answering fluently in the language is much harder! Yet, your language production fluency does not necessarily say anything about your knowledge and reasoning skills.
new_line
new_line
Hopefully we can get a chat arena for Dutch some day - user feedback is the most powerful metric!
text
Hopefully we can get a chat arena for Dutch some day - user feedback is the most powerful metric!
new_line
 🎈 LLM Benchmarks Update!

**tl;dr: do not depend on benchmark leaderboards to choose your "chatbot" model! (Especially for non-English languages.)**

First of all, I'm discontinuing the Open #Dutch #LLM Leaderboard (https://lnkd.in/eFnsaFR6). It will stay online for now, but I urge the use of the ScandEval leaderboard instead (https://scandeval.com/dutch-nlg/) by @saattrupdan. It contains more tasks, has better reproducibility and statistics (CI) and a flexible back-end library (`scandeval`) to run your own benchmarks with. As part of project "Leesplank" (with Michiel Buisman and Maarten Lens-FitzGerald) we recently added GPT-4-1106-preview scores to add a good "target" to the leaderboard.

An important note here is that benchmark leaderboards are not a golden truth. Especially evaluating generative models is hard. You run into issues like prompt engineering (and sensitivity of models to one or other prompt), structured output generation, and - quite simply - "how to automatically evaluate open-ended generation".

💡 Another important but under-discussed facet is the discrepancy between models' capability of understanding vs. generating *in different languages* (so the NLU part of NLG benchmarking). In other words: some of the listed models score really well on, e.g., MCQ benchmarks but are not suitable to use as DUTCH chat bots. Interestingly, some of these models seem to understand questions in Dutch and are able to pick the right answer (because they have good knowledge or reasoning skills), but generating fluent and grammatical Dutch is something else entirely! This is perhaps also true for humans: it's easier to sort-of grasp the meaning of a new language and answer with "Yes" or "No", but answering fluently in the language is much harder! Yet, your language production fluency does not necessarily say anything about your knowledge and reasoning skills.

Hopefully we can get a chat arena for Dutch some day - user feedback is the most powerful metric!

👍
davanstrien adamm-hf nikgr saattrupdan reach-vb shiv2050 ssml2050
🤝
Rijgersberg osanseviero adamm-hf robinsmits reach-vb
643033401572f43a4818cea3
https://cdn-avatars.huggingface.co/v1/production/uploads/643033401572f43a4818cea3/fWoLdj94pPavA9n-Yolej.png
Carlo Moro
cnmoro
user
6362ef7a19cf373a5fc52d3f
/avatars/4dc63d2be80ce2d1610d2742f42973a1.svg
Robin Smits
robinsmits
user
5e1e17b6fcf41d740b6996a8
https://cdn-avatars.huggingface.co/v1/production/uploads/1594192845975-5e1e17b6fcf41d740b6996a8.jpeg
Bram Vanroy
BramVanroy
user