Instruction limit for AI agents jumped 10x in a year

Source · How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI
AI Engineer · Laurie Voss · Arize AI · 2026-09-09 uploaded · 22min

한국어·English

A year ago, frontier models started dropping instructions around 200 to 300 words in a skills file. Laurie Voss of Arize AI reran the benchmark behind that number and found current models now hold up to 5,000 instructions, but fail in four new, harder-to-detect ways.

Origin of 200 — Voss heard the 200-instruction ceiling cited by Dexter Horthy at AI Engineer in Miami, sourced to the 2025 IFScale paper by Jeroslowitch and co-authors.
How IFScale works — The test asks a model to write a business report containing an exact list of words, then measures accuracy as the percentage of required words that appear.
Replicating old result — Only three of the original paper's 10 models were still reachable by API — GPT-4.1, Claude Sonnet 4, Gemini 2.5 Pro — and Voss's rerun matched the original finding: accuracy fell apart around 200 to 300 rules.
New frontier models — Testing GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and Deepseek V4 Pro, all four scored 100 percent immediately, forcing Voss to expand the word list from 500 up to 10,000.
New ceiling — The new boundary sits around 2,000 instructions for most models and near 5,000 for the best one, roughly a tenfold gain in twelve months.
Deepseek fails quietly — Deepseek V4 Pro starts forgetting instructions around 750 rules and is dropping nearly half of them by 2,000.
Claude refuses — Claude Opus 4.7 sometimes bails at the API level because its safety classifier flags combinations of random words, like anthrax and cyanide, as dangerous.
Gemini overthinks — Gemini 3.1 Pro stays solid to 5,000 instructions but past that spends its entire thinking-token budget verifying instructions and returns almost no output.
GPT quits mid-report — GPT-5.5 hits 99 percent accuracy out to 5,000 rules, then writes about 5,000 words before declaring the task stupid and stopping.
Cost of study — Running roughly 2,300 API calls across seven models cost Voss $29 total.
New research corroborates — Chroma's context-rot work across 18 models found accuracy on long inputs can fall 30 to 50 percent before hitting the context window limit.
The real problem now — Voss argues fitting instructions into a small budget was a compression problem that's solved, while knowing whether the model obeyed is now a verification problem solvable only by evals.

In their words

an agent can follow up to about 200 instructions uh before it starts forgetting those instructions0:53

so I ran GPT 5.5, Claude Opus 4.7 because 4.8 came out a week after I ran this test.5:55

I gave them the same prompt, the same words, the same everything. And I immediately ran into a problem, which is that they aced it. They all scored 100% immediately on this test.6:03

a year ago somewhere around 200 to 300 rules frontier models started falling apart. That is a really low ceiling.5:36

지침 준수 한계선 변화 — Laurie Voss가 IFScale 벤치마크를 재현·확장해 발표한 결과

모델별 실패 시작 지점 — Laurie Voss가 발표에서 언급한 각 모델의 지침 붕괴 시작 구간

Disclosure · Voss is head of developer relations at Arize AI, which sells LLM evaluation tools, and mentions Arize's evals product in the talk.

One thing to add — One thing to add — Voss stresses his benchmark is a proxy, not proof that real conflicting instructions get followed equally well. The four failure modes matter more than the 10x headline since each needs different detection.

One thing to try tonight
Count the instructions in your longest skills file, add back any edge cases you'd previously cut, then run it and read the full output closely for a silent mid-report bail-out.