Benchmarks!
text
Benchmarks!
new_line
new_line
I have lately been diving deep into the main benchmarks we all use to evaluate and compare models. 
text
I have lately been diving deep into the main benchmarks we all use to evaluate and compare models. 
new_line
new_line
If you've never actually looked under the hood for how benchmarks work, check out the LM eval harness from EleutherAI: 
text
If you've never actually looked under the hood for how benchmarks work, check out the LM eval harness from EleutherAI: 
https://github.com/EleutherAI/lm-evaluation-harness
https://github.com/EleutherAI/lm-evaluation-harness
link
new_line
new_line
+ check out the benchmark datasets, you can find the ones for the LLM leaderboard on the about tab here: 
text
+ check out the benchmark datasets, you can find the ones for the LLM leaderboard on the about tab here: 
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
link
, then click the dataset and actually peak at the data that comprises these benchmarks.
text
, then click the dataset and actually peak at the data that comprises these benchmarks.
new_line
new_line
It feels to me like benchmarks only represent a tiny portion of what we actually use and want LLMs for, and I doubt I'm alone in that sentiment. 
text
It feels to me like benchmarks only represent a tiny portion of what we actually use and want LLMs for, and I doubt I'm alone in that sentiment. 
new_line
new_line
Beyond this, the actual evaluations of responses from models are extremely strict and often use even rudimentary NLP techniques when, at this point, we have LLMs themselves that are more than capable at evaluating and scoring responses. 
text
Beyond this, the actual evaluations of responses from models are extremely strict and often use even rudimentary NLP techniques when, at this point, we have LLMs themselves that are more than capable at evaluating and scoring responses. 
new_line
new_line
It feels like we've made great strides in the quality of LLMs themselves, but almost no change in the quality of how we benchmark.
text
It feels like we've made great strides in the quality of LLMs themselves, but almost no change in the quality of how we benchmark.
new_line
new_line
If you have any ideas for how benchmarks could be a better assessment of an LLM, or know of good research papers that tackle this challenge, please share!  
text
If you have any ideas for how benchmarks could be a better assessment of an LLM, or know of good research papers that tackle this challenge, please share!  
Benchmarks!

I have lately been diving deep into the main benchmarks we all use to evaluate and compare models. 

If you've never actually looked under the hood for how benchmarks work, check out the LM eval harness from EleutherAI: https://github.com/EleutherAI/lm-evaluation-harness

+ check out the benchmark datasets, you can find the ones for the LLM leaderboard on the about tab here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, then click the dataset and actually peak at the data that comprises these benchmarks.

It feels to me like benchmarks only represent a tiny portion of what we actually use and want LLMs for, and I doubt I'm alone in that sentiment. 

Beyond this, the actual evaluations of responses from models are extremely strict and often use even rudimentary NLP techniques when, at this point, we have LLMs themselves that are more than capable at evaluating and scoring responses. 

It feels like we've made great strides in the quality of LLMs themselves, but almost no change in the quality of how we benchmark.

If you have any ideas for how benchmarks could be a better assessment of an LLM, or know of good research papers that tackle this challenge, please share!  
👍
julien-c Dlbk lewtun dillfrescott mmhamdy Noomam welkson choukrani
🚀
mosharafh jdagh
62645f88c39850dc093d6105
https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
Mohammed Hamdy
mmhamdy
user