A new paper titled "Long-Form Factuality in Large Language Models" proposes a new approach to evaluate the long-form factuality of large language models using an AI agent! They introduce SAFE (Search-Augmented Factuality Evaluator) which leverages an LLM to break down responses into individual facts, query Google to verify each fact, and perform multi-step reasoning.
text
A new paper titled "Long-Form Factuality in Large Language Models" proposes a new approach to evaluate the long-form factuality of large language models using an AI agent! They introduce SAFE (Search-Augmented Factuality Evaluator) which leverages an LLM to break down responses into individual facts, query Google to verify each fact, and perform multi-step reasoning.
new_line
new_line
Keypoints:
text
Keypoints:
new_line
* SAFE (Search-Augmented Factuality Evaluator) is an automated method using an LLM agent to evaluate factuality
text
* SAFE (Search-Augmented Factuality Evaluator) is an automated method using an LLM agent to evaluate factuality
new_line
* It also introduces LongFact, a 2,280 prompt set spanning 38 topics to test open-domain factual knowledge
text
* It also introduces LongFact, a 2,280 prompt set spanning 38 topics to test open-domain factual knowledge
new_line
* SAFE achieves a 72% humans agreement while being 20x cheaper. It also wins 76% of the disagreements measured on a small scale experiment where a more thorough human procedure (researchers + full internet search) was used.
text
* SAFE achieves a 72% humans agreement while being 20x cheaper. It also wins 76% of the disagreements measured on a small scale experiment where a more thorough human procedure (researchers + full internet search) was used.
new_line
* Larger models like GPT-4, Claude Opus and Gemini Ultra tend to exhibit better long-form factuality.
text
* Larger models like GPT-4, Claude Opus and Gemini Ultra tend to exhibit better long-form factuality.
new_line
new_line
Paper: 
text
Paper: 
Long-form factuality in large language models (2403.18802)
https://huggingface.co/papers/2403.18802
2403.18802
paper
resource
https://huggingface.co/papers/2403.18802
new_line
Code and data: 
text
Code and data: 
https://github.com/google-deepmind/long-form-factuality
https://github.com/google-deepmind/long-form-factuality
link
new_line
new_line
Congrats to the authors for their work!
text
Congrats to the authors for their work!
A new paper titled "Long-Form Factuality in Large Language Models" proposes a new approach to evaluate the long-form factuality of large language models using an AI agent! They introduce SAFE (Search-Augmented Factuality Evaluator) which leverages an LLM to break down responses into individual facts, query Google to verify each fact, and perform multi-step reasoning.

Keypoints:
* SAFE (Search-Augmented Factuality Evaluator) is an automated method using an LLM agent to evaluate factuality
* It also introduces LongFact, a 2,280 prompt set spanning 38 topics to test open-domain factual knowledge
* SAFE achieves a 72% humans agreement while being 20x cheaper. It also wins 76% of the disagreements measured on a small scale experiment where a more thorough human procedure (researchers + full internet search) was used.
* Larger models like GPT-4, Claude Opus and Gemini Ultra tend to exhibit better long-form factuality.

Paper: https://huggingface.co/papers/2403.18802
Code and data: https://github.com/google-deepmind/long-form-factuality

Congrats to the authors for their work!
🔥
nikgr zelus82 dillfrescott
👍
shiv2050 ssml2050 dillfrescott
❤️
samusenps dillfrescott