Spooky season is coming 👻 and there's nothing scarier then poor LLM evaluation results, right?
text
Spooky season is coming 👻 and there's nothing scarier then poor LLM evaluation results, right?
new_line
new_line
The ReasonScape m12x dataset, explorer and leaderboard been updated with 12 additional models to bring up the total to *54* models and over *7.1B thinking tokens* and the groups filter has been split into two: family and size, which lets us take a peek at our first small-models-only reasoning result!
text
The ReasonScape m12x dataset, explorer and leaderboard been updated with 12 additional models to bring up the total to *54* models and over *7.1B thinking tokens* and the groups filter has been split into two: family and size, which lets us take a peek at our first small-models-only reasoning result!
new_line
new_line
The top performer, 
text
The top performer, 
https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507
Qwen/Qwen3-4B-Thinking-2507
model
resource
https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507
 has an enormous overthink problem so I would actually give this crown to 
text
 has an enormous overthink problem so I would actually give this crown to 
https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507
Qwen/Qwen3-4B-Instruct-2507
model
resource
https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507
 - simply ask it to think Step-by-Step to bring out it's latent hybrid-COT behavior and enjoy stellar performance at a fraction of the tokens!
text
 - simply ask it to think Step-by-Step to bring out it's latent hybrid-COT behavior and enjoy stellar performance at a fraction of the tokens!
new_line
new_line
Right below the Qwen3-4B set and sitting just above 
text
Right below the Qwen3-4B set and sitting just above 
https://huggingface.co/Qwen/Qwen3-1.7B
Qwen/Qwen3-1.7B
model
resource
https://huggingface.co/Qwen/Qwen3-1.7B
 is a model from a family that is somewhat underrated: 
text
 is a model from a family that is somewhat underrated: 
https://huggingface.co/tencent/Hunyuan-4B-Instruct
tencent/Hunyuan-4B-Instruct
model
resource
https://huggingface.co/tencent/Hunyuan-4B-Instruct
 Note that vllm 0.11.0 has trouble with Hunyuan dense models, it seems to only support the MoE variants, I used 0.10.2 for my evaluations.
text
 Note that vllm 0.11.0 has trouble with Hunyuan dense models, it seems to only support the MoE variants, I used 0.10.2 for my evaluations.
new_line
new_line
In 7th place and worth a mention is 
text
In 7th place and worth a mention is 
https://huggingface.co/HuggingFaceTB/SmolLM3-3B
HuggingFaceTB/SmolLM3-3B
model
resource
https://huggingface.co/HuggingFaceTB/SmolLM3-3B
 which is a very efficient smaller thinker, but it's achilees heel was the strawberry test: it fails both letter counting and word sorting.
text
 which is a very efficient smaller thinker, but it's achilees heel was the strawberry test: it fails both letter counting and word sorting.
new_line
new_line
The last model worth discussing in this context is 
text
The last model worth discussing in this context is 
https://huggingface.co/google/gemma-3-4b-it
google/gemma-3-4b-it
model
resource
https://huggingface.co/google/gemma-3-4b-it
 which is obviously not a reasoning model but when asked to think step-by-step it demonstrated tolerable performance across several tasks with incredibly low token utilization compared to most of these little guys.
text
 which is obviously not a reasoning model but when asked to think step-by-step it demonstrated tolerable performance across several tasks with incredibly low token utilization compared to most of these little guys.
new_line
new_line
Would love to hear from the community!  Do you use any of these models in your day-to-day?  Did I miss any?  Let me know!
text
Would love to hear from the community!  Do you use any of these models in your day-to-day?  Did I miss any?  Let me know!
new_line
new_line
Full leaderboard @ 
text
Full leaderboard @ 
https://reasonscape.com/m12x/leaderboard/
https://reasonscape.com/m12x/leaderboard/
link
Spooky season is coming 👻 and there's nothing scarier then poor LLM evaluation results, right?

The ReasonScape m12x dataset, explorer and leaderboard been updated with 12 additional models to bring up the total to *54* models and over *7.1B thinking tokens* and the groups filter has been split into two: family and size, which lets us take a peek at our first small-models-only reasoning result!

The top performer, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507 has an enormous overthink problem so I would actually give this crown to https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 - simply ask it to think Step-by-Step to bring out it's latent hybrid-COT behavior and enjoy stellar performance at a fraction of the tokens!

Right below the Qwen3-4B set and sitting just above https://huggingface.co/Qwen/Qwen3-1.7B is a model from a family that is somewhat underrated: https://huggingface.co/tencent/Hunyuan-4B-Instruct Note that vllm 0.11.0 has trouble with Hunyuan dense models, it seems to only support the MoE variants, I used 0.10.2 for my evaluations.

In 7th place and worth a mention is https://huggingface.co/HuggingFaceTB/SmolLM3-3B which is a very efficient smaller thinker, but it's achilees heel was the strawberry test: it fails both letter counting and word sorting.

The last model worth discussing in this context is https://huggingface.co/google/gemma-3-4b-it which is obviously not a reasoning model but when asked to think step-by-step it demonstrated tolerable performance across several tasks with incredibly low token utilization compared to most of these little guys.

Would love to hear from the community!  Do you use any of these models in your day-to-day?  Did I miss any?  Let me know!

Full leaderboard @ https://reasonscape.com/m12x/leaderboard/
👍
John6666 sldaasd