Think AGI is just around the corner? Not so fast.
text
Think AGI is just around the corner? Not so fast.
new_line
new_line
When OpenAI released its Computer-Using Agent (CUA) API, I happened to be playing Wordle 🧩 and thought, why not see how the model handles it? 
text
When OpenAI released its Computer-Using Agent (CUA) API, I happened to be playing Wordle 🧩 and thought, why not see how the model handles it? 
new_line
Spoiler: Wordle turned out to be a surprisingly effective benchmark. 
text
Spoiler: Wordle turned out to be a surprisingly effective benchmark. 
new_line
So Romain Cosentino Ph.D. and I dug in and analyzed the results of several hundred runs.
text
So Romain Cosentino Ph.D. and I dug in and analyzed the results of several hundred runs.
new_line
new_line
🔑 Takeaways
text
🔑 Takeaways
new_line
1️⃣ Even the best computer-using models struggle with simple, context-dependent tasks. 
text
1️⃣ Even the best computer-using models struggle with simple, context-dependent tasks. 
new_line
2️⃣ Visual perception and reasoning remain major hurdles for multimodal agents. 
text
2️⃣ Visual perception and reasoning remain major hurdles for multimodal agents. 
new_line
3️⃣ Real-world use cases reveal significant gaps between hype and reality. Perception accuracy drops to near zero by the last turn 📉
text
3️⃣ Real-world use cases reveal significant gaps between hype and reality. Perception accuracy drops to near zero by the last turn 📉
new_line
new_line
🔗 Read our arxiv article for more details  
text
🔗 Read our arxiv article for more details  
https://www.arxiv.org/abs/2504.15434
https://www.arxiv.org/abs/2504.15434
link
Think AGI is just around the corner? Not so fast.

When OpenAI released its Computer-Using Agent (CUA) API, I happened to be playing Wordle 🧩 and thought, why not see how the model handles it? 
Spoiler: Wordle turned out to be a surprisingly effective benchmark. 
So Romain Cosentino Ph.D. and I dug in and analyzed the results of several hundred runs.

🔑 Takeaways
1️⃣ Even the best computer-using models struggle with simple, context-dependent tasks. 
2️⃣ Visual perception and reasoning remain major hurdles for multimodal agents. 
3️⃣ Real-world use cases reveal significant gaps between hype and reality. Perception accuracy drops to near zero by the last turn 📉

🔗 Read our arxiv article for more details  https://www.arxiv.org/abs/2504.15434
👀
victor John6666 9voltfan2009 ZennyKenny
👍
gen1i 9voltfan2009 agentlans vapit
❤️
alielfilali01 9voltfan2009 ProCreations
663a18c876e6d5b98f3cc94d
/avatars/faa2a47dbdd3bce6d47e0fbed6170627.svg
Alan Tseng
agentlans
user
649c9f1ff0f66daed3bb150a
https://cdn-avatars.huggingface.co/v1/production/uploads/649c9f1ff0f66daed3bb150a/If1wQodtIADTXMvogxE-0.jpeg
Sarath Shekkizhar
shekkizh
user