A new paper from Google DeepMind explores the effect of premise ordering on large language models (LLMs) in reasoning tasks. Despite the logical principle that the sequence of premises should not influence the conclusion's validity, the study finds LLMs' performance varies with different premise arrangements. Here's a summary:
text
A new paper from Google DeepMind explores the effect of premise ordering on large language models (LLMs) in reasoning tasks. Despite the logical principle that the sequence of premises should not influence the conclusion's validity, the study finds LLMs' performance varies with different premise arrangements. Here's a summary:
new_line
new_line
The research investigates how the order of premises affects LLMs in logical and mathematical reasoning tasks, challenging the assumption that premise sequence is irrelevant to the outcome.
text
The research investigates how the order of premises affects LLMs in logical and mathematical reasoning tasks, challenging the assumption that premise sequence is irrelevant to the outcome.
new_line
new_line
Key Findings:
text
Key Findings:
new_line
* Logical Reasoning: LLMs perform best when premises are in a forward order that aligns with the proof's progression. Deviations from this order result in significant performance drops.
text
* Logical Reasoning: LLMs perform best when premises are in a forward order that aligns with the proof's progression. Deviations from this order result in significant performance drops.
new_line
* Mathematical Reasoning: The introduction of the R-GSM benchmark shows a similar sensitivity in LLMs.
text
* Mathematical Reasoning: The introduction of the R-GSM benchmark shows a similar sensitivity in LLMs.
new_line
new_line
Congrats to the authors for their work!
text
Congrats to the authors for their work!
new_line
new_line
Paper: 
text
Paper: 
Premise Order Matters in Reasoning with Large Language Models (2402.08939)
https://huggingface.co/papers/2402.08939
2402.08939
paper
resource
https://huggingface.co/papers/2402.08939
.
text
.
A new paper from Google DeepMind explores the effect of premise ordering on large language models (LLMs) in reasoning tasks. Despite the logical principle that the sequence of premises should not influence the conclusion's validity, the study finds LLMs' performance varies with different premise arrangements. Here's a summary:

The research investigates how the order of premises affects LLMs in logical and mathematical reasoning tasks, challenging the assumption that premise sequence is irrelevant to the outcome.

Key Findings:
* Logical Reasoning: LLMs perform best when premises are in a forward order that aligns with the proof's progression. Deviations from this order result in significant performance drops.
* Mathematical Reasoning: The introduction of the R-GSM benchmark shows a similar sensitivity in LLMs.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.08939.
👍
osanseviero jjoker999 alielfilali01 flippers clem victor Dlbk cosmojg