measuring the information content of a reasoning trace seems like a straightforward reasoning LLM KPI, but how can we achieve this? 
text
measuring the information content of a reasoning trace seems like a straightforward reasoning LLM KPI, but how can we achieve this? 
new_line
new_line
what if we keep it simple: gzip the resulting text and take the length of the compressed stream... "compressed bytes of information per output token" becomes the KPI
text
what if we keep it simple: gzip the resulting text and take the length of the compressed stream... "compressed bytes of information per output token" becomes the KPI
new_line
new_line
if we split across correct answers vs incorrect answers vs truncated answers and group by difficulty, a whole new world of analysis becomes not just possible but visually intuitive and almost trivial:
text
if we split across correct answers vs incorrect answers vs truncated answers and group by difficulty, a whole new world of analysis becomes not just possible but visually intuitive and almost trivial:
new_line
new_line
1) what is the model's overall reasoning efficiency? this is the slope of the scatterplot curve segments (there may be more then one..)
text
1) what is the model's overall reasoning efficiency? this is the slope of the scatterplot curve segments (there may be more then one..)
new_line
new_line
2) is the model able to apply more test-time compute towards more difficult variations of the task? the two on the left are not, the two on the right are.
text
2) is the model able to apply more test-time compute towards more difficult variations of the task? the two on the left are not, the two on the right are.
new_line
new_line
3) when applying more test-time compute, is that compute useful? this is the curvature of the scatterplot trends - the two in the middle are 'losing their mojo' as answers get longer the information content falls down
text
3) when applying more test-time compute, is that compute useful? this is the curvature of the scatterplot trends - the two in the middle are 'losing their mojo' as answers get longer the information content falls down
new_line
new_line
4) is the model applying multiple approaches to the task? (right) do those approaches change with difficulty?
text
4) is the model applying multiple approaches to the task? (right) do those approaches change with difficulty?
new_line
new_line
5) are truncations because we don't have enough context budget (left) or because the model has lost its mind and gone into a repeat loop (middle two) and does this happen across the board (middle left) or only when the problem is more difficult (middle right)
text
5) are truncations because we don't have enough context budget (left) or because the model has lost its mind and gone into a repeat loop (middle two) and does this happen across the board (middle left) or only when the problem is more difficult (middle right)
new_line
new_line
would love to hear your guys feedback on this kind of analysis, is anyone doing similar work? 
text
would love to hear your guys feedback on this kind of analysis, is anyone doing similar work? 
new_line
new_line
this approach generates 12 plots per model (one for each task) so quite a bit of data and i've been hesitant to publish it so far, consider this post a toe tip.
text
this approach generates 12 plots per model (one for each task) so quite a bit of data and i've been hesitant to publish it so far, consider this post a toe tip.
measuring the information content of a reasoning trace seems like a straightforward reasoning LLM KPI, but how can we achieve this? 

what if we keep it simple: gzip the resulting text and take the length of the compressed stream... "compressed bytes of information per output token" becomes the KPI

if we split across correct answers vs incorrect answers vs truncated answers and group by difficulty, a whole new world of analysis becomes not just possible but visually intuitive and almost trivial:

1) what is the model's overall reasoning efficiency? this is the slope of the scatterplot curve segments (there may be more then one..)

2) is the model able to apply more test-time compute towards more difficult variations of the task? the two on the left are not, the two on the right are.

3) when applying more test-time compute, is that compute useful? this is the curvature of the scatterplot trends - the two in the middle are 'losing their mojo' as answers get longer the information content falls down

4) is the model applying multiple approaches to the task? (right) do those approaches change with difficulty?

5) are truncations because we don't have enough context budget (left) or because the model has lost its mind and gone into a repeat loop (middle two) and does this happen across the board (middle left) or only when the problem is more difficult (middle right)

would love to hear your guys feedback on this kind of analysis, is anyone doing similar work? 

this approach generates 12 plots per model (one for each task) so quite a bit of data and i've been hesitant to publish it so far, consider this post a toe tip.
👀
unmodeled-tyler John6666
🤯
brhodeland