We tested Qwen3-Coder, GPT-5 and other 30+ models on new SWE-Bench like tasks from July 2025!
text
We tested Qwen3-Coder, GPT-5 and other 30+ models on new SWE-Bench like tasks from July 2025!
new_line
new_line
Hi all, I’m Ibragim from Nebius.
text
Hi all, I’m Ibragim from Nebius.
new_line
new_line
We ran a benchmark on 34 fresh GitHub PR tasks from July 2025 using the SWE-rebench leaderboard 
text
We ran a benchmark on 34 fresh GitHub PR tasks from July 2025 using the SWE-rebench leaderboard 
https://swe-rebench.com/leaderboard
https://swe-rebench.com/leaderboard
link
 . These are real, recent problems — no training-set contamination — and include both proprietary and open-source models.
text
 . These are real, recent problems — no training-set contamination — and include both proprietary and open-source models.
new_line
new_line
Quick takeaways:
text
Quick takeaways:
new_line
new_line
> GPT-5-Medium leads overall (29.4% resolved rate, 38.2% pass@5).
text
> GPT-5-Medium leads overall (29.4% resolved rate, 38.2% pass@5).
new_line
> Qwen3-Coder is the best open-source performer, matching GPT-5-High in pass@5 (32.4%) despite a lower resolved rate.
text
> Qwen3-Coder is the best open-source performer, matching GPT-5-High in pass@5 (32.4%) despite a lower resolved rate.
new_line
> Claude Sonnet 4.0 lags behind in pass@5 at 23.5%.
text
> Claude Sonnet 4.0 lags behind in pass@5 at 23.5%.
new_line
new_line
All tasks come from the continuously updated, decontaminated nebius/SWE-rebench-leaderboard for real-world SWE tasks.
text
All tasks come from the continuously updated, decontaminated nebius/SWE-rebench-leaderboard for real-world SWE tasks.
new_line
new_line
We tested Qwen3-Coder, GPT-5 and other 30+ models on new SWE-Bench like tasks from July 2025!

Hi all, I’m Ibragim from Nebius.

We ran a benchmark on 34 fresh GitHub PR tasks from July 2025 using the SWE-rebench leaderboard https://swe-rebench.com/leaderboard . These are real, recent problems — no training-set contamination — and include both proprietary and open-source models.

Quick takeaways:

> GPT-5-Medium leads overall (29.4% resolved rate, 38.2% pass@5).
> Qwen3-Coder is the best open-source performer, matching GPT-5-High in pass@5 (32.4%) despite a lower resolved rate.
> Claude Sonnet 4.0 lags behind in pass@5 at 23.5%.

All tasks come from the continuously updated, decontaminated nebius/SWE-rebench-leaderboard for real-world SWE tasks.


🚀
John6666
67daa1ace464cecf6df3dc8e
https://cdn-avatars.huggingface.co/v1/production/uploads/67daa1ace464cecf6df3dc8e/Uu6ge-iDzNfpxe9vBtgIT.png
diginoron - digital transformation
diginoron
user
654e5e094319c75e3e1b6cbc
/avatars/a8889036fa38f80f2d45aea8d1471395.svg
Ibragim
ibragim-bad
user