🔥 Prometheus 2 was recently released by Kaist AI as an alternative and closely mirroring both human and GPT-4 evaluation, and surpassing the former Prometheus!
text
🔥 Prometheus 2 was recently released by Kaist AI as an alternative and closely mirroring both human and GPT-4 evaluation, and surpassing the former Prometheus!
new_line
new_line
https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
prometheus-eval/prometheus-7b-v2.0
model
resource
https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
new_line
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
prometheus-eval/prometheus-8x7b-v2.0
model
resource
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0
new_line
new_line
🌬️Fine-tuned on top of 
text
🌬️Fine-tuned on top of 
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
mistralai/Mistral-7B-Instruct-v0.2
model
resource
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
 and 
text
 and 
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
mistralai/Mixtral-8x7B-Instruct-v0.1
model
resource
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
new_line
🗂️The datasets used for fine-tuning have been publicly released i.e. 
text
🗂️The datasets used for fine-tuning have been publicly released i.e. 
https://huggingface.co/datasets/prometheus-eval/Feedback-Collection
prometheus-eval/Feedback-Collection
dataset
resource
https://huggingface.co/datasets/prometheus-eval/Feedback-Collection
 and 
text
 and 
https://huggingface.co/datasets/prometheus-eval/Preference-Collection
prometheus-eval/Preference-Collection
dataset
resource
https://huggingface.co/datasets/prometheus-eval/Preference-Collection
new_line
🤝🏻Unified LM evaluator for absolute (a single prompt-completion pair) and relative (two completions for a given prompt) due to model merging
text
🤝🏻Unified LM evaluator for absolute (a single prompt-completion pair) and relative (two completions for a given prompt) due to model merging
new_line
❌No longer needs a mandatory reference / golden answer, but can still be provided optionally
text
❌No longer needs a mandatory reference / golden answer, but can still be provided optionally
new_line
🔝Surpasses the former version of Prometheus, and has a high correlation with human, GPT-4, and Claude 3 Opus scores when evaluating LMs
text
🔝Surpasses the former version of Prometheus, and has a high correlation with human, GPT-4, and Claude 3 Opus scores when evaluating LMs
new_line
📝Apache 2.0 license
text
📝Apache 2.0 license
new_line
new_line
Long-story short, an amazing job from Kaist AI bridging the gap with LLM evaluators other than proprietary and bigger models!
text
Long-story short, an amazing job from Kaist AI bridging the gap with LLM evaluators other than proprietary and bigger models!
new_line
new_line
This week at Argilla, we decided to add a new task to use Prometheus 2 as an LLM evaluator using 
text
This week at Argilla, we decided to add a new task to use Prometheus 2 as an LLM evaluator using 
distilabel
`distilabel`
inline_code
, so we implemented 
text
, so we implemented 
PrometheusEval
`PrometheusEval`
inline_code
.
text
.
new_line
new_line
😱 Using 
text
😱 Using 
PrometheusEval
`PrometheusEval`
inline_code
 running their 7B variant with vLLM in a single L40 on top of 
text
 running their 7B variant with vLLM in a single L40 on top of 
https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset
HuggingFaceH4/instruction-dataset
dataset
resource
https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset
, we got the 327 existing prompt-completion pairs evaluated and pushed to the Hub in less than 2 minutes!
text
, we got the 327 existing prompt-completion pairs evaluated and pushed to the Hub in less than 2 minutes!
new_line
new_line
Find the generated dataset and the code at 
text
Find the generated dataset and the code at 
https://huggingface.co/datasets/distilabel-internal-testing/instruction-dataset-prometheus
distilabel-internal-testing/instruction-dataset-prometheus
dataset
resource
https://huggingface.co/datasets/distilabel-internal-testing/instruction-dataset-prometheus
🔥 Prometheus 2 was recently released by Kaist AI as an alternative and closely mirroring both human and GPT-4 evaluation, and surpassing the former Prometheus!

https://huggingface.co/prometheus-eval/prometheus-7b-v2.0
https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0

🌬️Fine-tuned on top of https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 and https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
🗂️The datasets used for fine-tuning have been publicly released i.e. https://huggingface.co/datasets/prometheus-eval/Feedback-Collection and https://huggingface.co/datasets/prometheus-eval/Preference-Collection
🤝🏻Unified LM evaluator for absolute (a single prompt-completion pair) and relative (two completions for a given prompt) due to model merging
❌No longer needs a mandatory reference / golden answer, but can still be provided optionally
🔝Surpasses the former version of Prometheus, and has a high correlation with human, GPT-4, and Claude 3 Opus scores when evaluating LMs
📝Apache 2.0 license

Long-story short, an amazing job from Kaist AI bridging the gap with LLM evaluators other than proprietary and bigger models!

This week at Argilla, we decided to add a new task to use Prometheus 2 as an LLM evaluator using `distilabel`, so we implemented `PrometheusEval`.

😱 Using `PrometheusEval` running their 7B variant with vLLM in a single L40 on top of https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset, we got the 327 existing prompt-completion pairs evaluated and pushed to the Hub in less than 2 minutes!

Find the generated dataset and the code at https://huggingface.co/datasets/distilabel-internal-testing/instruction-dataset-prometheus
🔥
matinla adamm-hf davanstrien andrewrreed victor NickyNicky QuocKhanh osanseviero dhruva-sarma shamy777
❤️
matinla adamm-hf andrewrreed QuocKhanh osanseviero anakin87
663b5958f526db6044784904
/avatars/5ffbcc32035ad5a5c83e45a4910c0541.svg
matinlaghaei
matinla
user