Context rot is such a catchy phrase, but the problem has been identified 2+ years ago, called attention decay.
text
Context rot is such a catchy phrase, but the problem has been identified 2+ years ago, called attention decay.
new_line
Lost in the Middle: How Language Models Use Long Contexts (2307.03172)
https://huggingface.co/papers/2307.03172
2307.03172
paper
resource
https://huggingface.co/papers/2307.03172
new_line
new_line
I spotted the same problem in coding tasks, and documented in my book (
text
I spotted the same problem in coding tasks, and documented in my book (
https://www.amazon.com/dp/9999331130
https://www.amazon.com/dp/9999331130
link
).
text
).
new_line
new_line
Why did this problem become hot again? This is because many of us thought the problem has been solved by long context models, which is not true.
text
Why did this problem become hot again? This is because many of us thought the problem has been solved by long context models, which is not true.
new_line
new_line
Here we were misled by benchmarks. Most long-context benchmarks build around the QA scenario, i.e.  "finding needle in haystack". But in agentic scenarios, the model needs to find EVERYTHING in the haystack, and just can't afford enough attention for this challenge.
text
Here we were misled by benchmarks. Most long-context benchmarks build around the QA scenario, i.e.  "finding needle in haystack". But in agentic scenarios, the model needs to find EVERYTHING in the haystack, and just can't afford enough attention for this challenge.
Context rot is such a catchy phrase, but the problem has been identified 2+ years ago, called attention decay.
https://huggingface.co/papers/2307.03172

I spotted the same problem in coding tasks, and documented in my book (https://www.amazon.com/dp/9999331130).

Why did this problem become hot again? This is because many of us thought the problem has been solved by long context models, which is not true.

Here we were misled by benchmarks. Most long-context benchmarks build around the QA scenario, i.e.  "finding needle in haystack". But in agentic scenarios, the model needs to find EVERYTHING in the haystack, and just can't afford enough attention for this challenge.
👍
ajibawa-2023 djuna amirkiarafiei mike-ravkine ilintar jukofyork John6666 setianke edwixx victor
6344a1b0762379fc63017e62
https://cdn-avatars.huggingface.co/v1/production/uploads/6344a1b0762379fc63017e62/g4VIT8l2lZIj6AoQAwVy7.png
John
cmp-nct
user
669dbd709a4bf63e08f1ddc2
https://cdn-avatars.huggingface.co/v1/production/uploads/669dbd709a4bf63e08f1ddc2/aV10ZJPPzH5LbnHFZNqc7.png
Yi Cui
onekq
user