Hugging Face's logo
Hennadii Kutomanov's picture

Hennadii Kutomanov PRO

kutomanovhennadii
8 17
VizualDreamer's profile picture originalpictures's profile picture Twu31's profile picture
·

AI & ML interests

Independent researcher. IEC studies the behavior and degradation of language models under controlled load. Code: github.com/kutomanovhennadii/instruction-capacity

Recent Activity

commentedon a paper 5 days ago
Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
repliedto their post 5 days ago
A model’s context window can hold hundreds of thousands of tokens. But how many instructions can it actually follow at once? Developers working with AI agents run into this problem all the time. You add rules to CLAUDE.md, and the agent starts ignoring some of the earlier ones. As the conversation grows longer, the model forgets the original requirements. You connect dozens of tools, and the quality of its work drops, even if many of those tools are never used. In the IFScale study, models were given up to 500 instructions. Even the top-performing models followed only about 68% of them. We have learned to work around these limitations: shortening rule files, clearing the context, loading only the tools we need. But these are still largely rules of thumb. Yet there is a fundamental gap. We measure model speed, token cost, reasoning performance, and context window size. But we have no generally accepted way to measure how reliably a model follows instructions as the load increases. We cannot say how much additional load a particular model can handle. We cannot reliably compare two models in terms of how their performance degrades under load. We cannot estimate in advance whether a prompt will be too demanding. What if we treated degradation not as an annoying quirk of LLMs, but as a measurable property? What if we could express load in defined units, record the outcome of each instruction, identify the laws governing performance degradation, and determine a model’s limit with a measurement uncertainty? That is what my new article is about: why existing benchmarks are not enough, what it takes to measure these limits, and how we approached the problem. At the end, I introduce IEC, an open-source experimental instrument we built to study model behavior under controlled load. https://huggingface.co/blog/kutomanovhennadii/why-measure-a-models-behavior-under-load Originally written in Russian. English translation assisted by AI.
posted an update 7 days ago
Why can a language model do better on a longer text? Imagine asking an AI to process a document while following 32 instructions. It gets only a third of the required actions right. Then you give it a document several times longer, with the same 32 instructions, and it makes almost no mistakes. A paradox? That's exactly what happened in an experiment with GPT-4o mini. In the IEC instrument, a model rewrites text using simple word-replacement rules like `lake → stone`. In one task, the rules covered all 32 distinct words of the text; in the other, only 32 out of 256. The number of instructions stayed the same, but the replacement density dropped eightfold. Turns out, 32 instructions don't always mean the same load. Ordinary prompts work the same way. Opening the answer with a greeting is one instruction; banning commas from the whole answer is one too. But the first is satisfied once, while the second must be respected at every step of generation. Higher density, higher load. So what actually determines the load? My new article answers that. I explain why some seemingly obvious candidates had to be rejected and how the data revealed five load axes: input text length, word position in the response, replacement density, number of rules, and the rule's position in the list. Why does it matter? Benchmarks usually just add instructions, but as their number grows, so do their positions in the list and the length of the prompt and the response. A single curve mixes all these effects: the model errs more often, but you can't tell why. The axes separate the causes, and each follows its own law of degradation. Code and data are open, so you can check everything yourself. Article: https://huggingface.co/blog/kutomanovhennadii/what-affects-a-models-behavior-under-load Data: https://huggingface.co/datasets/kutomanovhennadii/iec-gpt-4o-mini-2024-07-18 Originally written in Russian. English translation assisted by AI.
View all activity

Organizations

None yet