How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00-
00:00:12- All right, hello everybody.
00:00:15Thank you for coming to this delightfully nerdy talk.
00:00:20This talk has a really long title,
00:00:22so let me give you the short version up front.
00:00:23You write skills files and stuff them full of instructions.
00:00:27At some point, the model stops keeping track of all of them.
00:00:31The question is, where is that point?
00:00:33At what point have you put too many instructions
00:00:35in your skills files?
00:00:36And the answer has changed a lot in the last year.
00:00:40I'm Laurie, I'm Head of Developer Relations at Arise AI.
00:00:44In a former life, I co-founded NPM Inc.
00:00:46So some of you may know me from the days of JavaScript.
00:00:48These days, I spend a lot of time thinking about AI
00:00:50and how to test it.
00:00:53A few months ago, I was an AI engineer in Miami,
00:00:55which was a good conference.
00:00:57And I was watching a talk by Dexter Horthy.
00:00:59It was a good talk.
00:01:00It was not about this topic at all.
00:01:02But while he was giving that talk,
00:01:04he mentioned, as an aside,
00:01:06that an agent can follow up to about 200 instructions
00:01:10before it starts forgetting those instructions.
00:01:13And then he moved on in his talk,
00:01:15and it was entirely an aside.
00:01:17And he mentioned that that figure is from 2025,
00:01:20so things might be better now.
00:01:22And I stopped listening for a second
00:01:25because I was like, 200 instructions
00:01:27is not very many instructions at all, right?
00:01:31A decent skills file blows past 200 instructions
00:01:33almost immediately.
00:01:35If the user says X, do Y,
00:01:37always include a section on Z,
00:01:39never use the phrase W,
00:01:40every one of those is a separate instruction.
00:01:42And if the model quietly stops tracking them after 200,
00:01:45that's a really hard ceiling
00:01:47on the complexity of what you can build.
00:01:49So I wanted to know where he got that number first,
00:01:52and I wanted to know if it was true.
00:01:55So you know the feeling that I'm talking about.
00:01:57You write this big, beautiful skills file,
00:01:58pages of rules, edge cases, tone, formatting.
00:02:00You hand it to the agent.
00:02:01It does the thing.
00:02:03And you look at the output and go,
00:02:05did it actually pay attention?
00:02:07Did it actually follow all of these rules?
00:02:09Or did it just sort of do what it felt like
00:02:11and sort of give me a close simulacrum
00:02:14of what I was expecting?
00:02:16You can't really tell, or can you?
00:02:18More on that later.
00:02:20And so you live with this low-grade anxiety
00:02:23every time you hit run.
00:02:24And that feeling is what this research is about
00:02:27and what we're trying to find out if we can avoid.
00:02:30So here's my promise for your next 18 minutes.
00:02:33I'm gonna show you where that 200 number came from,
00:02:36whether it's still true,
00:02:37and what the real number is today,
00:02:39because it moved by an order of magnitude.
00:02:41And then we're gonna talk about what that means
00:02:44for you to take away.
00:02:46How long your skills and prompts can actually be,
00:02:49and what changes you should make
00:02:51to your workflow as a result.
00:02:54So the 200 number isn't folklore.
00:02:57It comes from a real benchmark called IfScale,
00:02:59from a paper by this guy whose name I'm going to mess up,
00:03:03Jaroslawicz, and co-authors last year.
00:03:06And the test is beautifully simple.
00:03:10Here's how IfScale works.
00:03:12You ask the model to write a business report,
00:03:14and you give it a list of specific words
00:03:16that it has to include exactly in the report.
00:03:19Include the exact word customer,
00:03:20include the exact word revenue,
00:03:22and so on for as many words as you want.
00:03:24Each of those is an instruction that it has to follow.
00:03:27And then you count how many of those exact words showed up.
00:03:32So because the test is so simple,
00:03:34you only have to keep two numbers in your head.
00:03:36One is density, which we call n.
00:03:38That is how many rules we're talking about at once.
00:03:41And the second is accuracy,
00:03:42which is the percentage of those rules
00:03:43that it was able to actually follow.
00:03:46Now you might say that including random words in a report
00:03:50is not the same as following real instructions,
00:03:52and fair enough, and we're going to talk about that.
00:03:55But the keywords are a proxy.
00:03:57Include the word revenue is the same shape of task
00:04:01as include a section on pricing, right?
00:04:02Or never use this phrase.
00:04:04It is a discrete, named constraint
00:04:06that you've told the agent that it has to follow.
00:04:09If a model can't track 200 words in one prompt,
00:04:11it's definitely going to struggle
00:04:13with 200 more complicated instructions.
00:04:16So if anything, it's going to do worse.
00:04:20So this number is a ceiling.
00:04:22This number is as high as you can go.
00:04:23If you give it more complicated instructions,
00:04:25the number is probably going to get lower.
00:04:27And 200 is a really low ceiling.
00:04:30So before chasing new models,
00:04:32you have to do good science,
00:04:33which means that you have to replicate the old result
00:04:36and make sure that the 200 ceiling is real.
00:04:39So I reran the original benchmark.
00:04:42The original paper tested a whole batch of models,
00:04:45and models live and die really fast.
00:04:47So by the time I got around to doing this testing,
00:04:50only three of the models in the original set
00:04:52of 10 models that they used
00:04:54were still available via any kind of API.
00:04:57So they were GPT 4.1, Claude Sonnet 4,
00:04:59and Gemini 2.5 Pro.
00:05:01Those were models that were available 12 months ago,
00:05:03that are still available now.
00:05:05And that is why we tested those three,
00:05:07'cause they were what was left.
00:05:08And since I first published this research
00:05:11a couple of weeks ago,
00:05:12one of those three models has been retired.
00:05:14So this was the last possible time
00:05:16that I could have run this test.
00:05:17So of that lineup, we're already down to two.
00:05:20So don't get attached to your models.
00:05:22So here is the results that we got replicating
00:05:25the original if scale finding.
00:05:27That is accuracy on the vertical axis.
00:05:29So it starts at 100% and begins to fall off.
00:05:32And then the number of rules going up along the bottom
00:05:35on log scale.
00:05:36So every time it gets halfway across,
00:05:37it has doubled the number of rules that it's dealing with.
00:05:42So by 500 rules, you're losing 30, 40, 50% of them.
00:05:46Our curves matched the results in the original paper
00:05:49within the noise boundary, so the finding was real.
00:05:52A year ago, somewhere around 200 to 300 rules,
00:05:55frontier models started falling apart.
00:05:57That is a really low ceiling.
00:06:00So that is our baseline, and now comes the fun part
00:06:03where we took the exact same test
00:06:05and pointed it at the current frontier,
00:06:07or rather, what the current frontier was
00:06:09when I ran this test.
00:06:10So I ran GPT 5.5, Claude Opus 4.7,
00:06:13because 4.8 came out a week after I ran this test,
00:06:17Gemini 3.1 Pro, and Deep Seek V4 Pro.
00:06:20So I gave them the same prompt, the same words,
00:06:22the same everything, and I immediately ran into a problem,
00:06:25which is that they aced it.
00:06:27They all scored 100% immediately on this test,
00:06:30absolutely no bugs.
00:06:34So we'd built a test to find the ceiling,
00:06:36and the models had walked straight through the ceiling
00:06:37without noticing that the ceiling was there.
00:06:40And that was a problem, because the benchmark
00:06:42was written to top out at 500 words,
00:06:43so I had to change the benchmark
00:06:45in order to be able to find the new ceiling.
00:06:47So I moved the goalposts, I gave it more words to include.
00:06:50I doubled it from 500 to 1,000,
00:06:52I doubled it again from 1,000 to 2,000,
00:06:55and I kept doing that until I hit a 10,000
00:06:56word vocabulary, and that is where I began to find the ceiling
00:07:00of what models can do these days.
00:07:03So let me put up the, this is the money slide,
00:07:06this is the results.
00:07:07Remember, log scale on the x-axis there.
00:07:12So it's going from 500 to 1,000 to 5,000 to 10,000.
00:07:16So it looks like that scale is falling off of a cliff,
00:07:18and it's actually happening over like 1,000 numbers.
00:07:20But look how far to the right these new curves get
00:07:25before they bend.
00:07:26A year ago, they were falling over at 200 to 300 instructions,
00:07:29and now depending on the model, the boundary is closer to 2,000,
00:07:32and for the best of them, it is up to 5,000 instructions
00:07:36before they begin to fall off a cliff.
00:07:38So in about 12 months, frontier models got close to 10 times
00:07:41better at following instructions simultaneously.
00:07:44That is the headline finding, and there is a lot of nuance
00:07:47that we need to get into.
00:07:49The capacity to track 2,000 named constraints in a single prompt
00:07:53is there.
00:07:55And that's really interesting because I think,
00:07:57I don't know if everybody else feels this way,
00:07:59but like it felt to me like the jump from, you know,
00:08:04GPT 5.1 to GPT 5.5 was kind of incremental, right?
00:08:07It didn't feel like we'd got 10 times better,
00:08:09but this is a test that really matters to a very practical thing,
00:08:14like how long can my skills file be?
00:08:17And in the course of a year, we got 10 times better.
00:08:21And the thing that gets me is that this benchmark
00:08:23is barely a year old.
00:08:25A year later, 500 is a rounding error,
00:08:27and this keeps moving under my feet.
00:08:29I tested 4.7, Opus 4.8 is even better.
00:08:35So this chart is a little out of date already,
00:08:36which is kind of the whole point.
00:08:37If you set your engineering assumptions
00:08:39about how skills files should work,
00:08:41about how long your prompt can be,
00:08:44and you did that more than about six months ago,
00:08:46you are incorrect now,
00:08:48and you should probably be re-engineering how you do stuff.
00:08:52But there is more to this story
00:08:54because the way that the models failed
00:08:59changed dramatically
00:09:00and the way that they failed is very important.
00:09:02This part was a completely unexpected finding
00:09:06when I started running the experiment.
00:09:07And it totally messed up my test to start with
00:09:10because the old failure mode was boring.
00:09:13They would just forget instructions
00:09:14and I could measure how many instructions
00:09:16they had remembered or forgotten.
00:09:17But the new ones fall apart in their own weird,
00:09:20extremely on-brand way.
00:09:22So let me introduce you to how these four models fail.
00:09:26Deep Seek 4 is a traditional model.
00:09:29It just forgets things.
00:09:30It doesn't have any drama.
00:09:31It starts forgetting instructions around 750 rules,
00:09:35and by 2000, it's dropping nearly half of them.
00:09:38So it just forgets, which frankly is the failure mode
00:09:41that I trust most because it's predictable.
00:09:43It's very easy to measure.
00:09:44And the other models were not nearly as cooperative.
00:09:48Opus 4.7 would decide repeatedly
00:09:52that the test was dangerous.
00:09:54And what it would do is it would refuse
00:09:57at the API level to complete the test.
00:09:59I didn't know that there was an API response
00:10:01that you could get from Claude where it was like,
00:10:03no, I could do this, but I'm not going to.
00:10:06But that's absolutely an API-level response
00:10:09that Claude supports because they care
00:10:10so much about safety, and I started getting those
00:10:13all of the time.
00:10:15And the reason that was happening
00:10:16is because Claude has a very sensitive safety classifier.
00:10:19And if you put in certain combinations of words,
00:10:22like say anthrax and cyanide,
00:10:24it decides that the whole request is dangerous
00:10:26and it bails out.
00:10:27And if you remember what my test does,
00:10:29my test is throwing five to 10,000 random words
00:10:33into an instruction file.
00:10:35And so my randomly selected words contained
00:10:38all sorts of things that looked dangerous
00:10:39in combination to the safety filter.
00:10:41And so it kept bailing saying that I was asking it
00:10:43to make a bomb or something.
00:10:47So we had to, to get Claude to cooperate,
00:10:52I had to take all of my words
00:10:54and run them through OpenAI's safety filter
00:10:56and filter out all of the naughty looking words
00:10:58so that it could get to anywhere.
00:10:59Once I'd given it that, Claude did really well.
00:11:03But the failure mode is that Claude is more likely
00:11:05to decide what you're doing is dangerous very early on
00:11:08at, you know, even two or three hundred instructions
00:11:11if what you're doing is, you know,
00:11:13contains anything to do with medical advice
00:11:15because medical things often are dual purpose.
00:11:17They can be dangerous, they can be safe.
00:11:20So the third failure mode was Gemini 3.1 Pro.
00:11:24Gemini is rock solid all the way out to 5,000 instructions.
00:11:28It does extremely well.
00:11:30Genuinely one of the best on the chart.
00:11:32And then past that, it gets weird.
00:11:35It doesn't forget the instructions.
00:11:37It gets overwhelmed by the instructions.
00:11:39What it tries to do is it, it uses thinking tokens
00:11:44to make sure that it is following
00:11:45all of the instructions at once.
00:11:46And when the number of instructions gets really high,
00:11:48it uses all of its thinking tokens.
00:11:50It uses its entire token budget thinking
00:11:53and then it doesn't give any output.
00:11:55It's, it gets to like nine, you know,
00:11:57if you've given it 10,000 tokens worth,
00:11:59it'll get 9,500 tokens worth of thinking
00:12:02and then give you a 500 word response,
00:12:04which doesn't contain any of the tokens.
00:12:07So it thinks itself into a corner
00:12:09and runs out of room to actually answer,
00:12:10which is very expensive and totally unhelpful,
00:12:13which is kind of on brand, isn't it?
00:12:19Which, you know, I would never say that out loud.
00:12:23And finally comes the winner, which is GPT 5.5.
00:12:26GPT 5.5 is the best of the lot, 99% accuracy,
00:12:29all the way out to 5,000 rules.
00:12:32But if you push it far enough,
00:12:33it is by far the weirdest of the bunch.
00:12:35Because it doesn't refuse outright,
00:12:37it doesn't silently forget.
00:12:38Instead, what it does is it gets frustrated
00:12:41and tells you that the test is stupid.
00:12:45It starts the report, it gets a few,
00:12:47like that's the thing,
00:12:48it doesn't start out just saying no.
00:12:50It starts the report,
00:12:51it starts writing the report,
00:12:52and like 500 words into the report,
00:12:54it's like, no, this is dumb.
00:12:55I'm not gonna do this.
00:12:56And then it politely tells you this is dumb.
00:12:58I'm not going to do this anymore.
00:13:00That is the actual response that it gave me.
00:13:02But that was like 5,000 words into this business report
00:13:05that I told it to generate.
00:13:08So it's not wrong, right?
00:13:10I was asking for a coherent business report
00:13:12that on no particular subject
00:13:14that contains 5,000 random words.
00:13:17You're right, GPT.
00:13:19This is a stupid thing to ask for.
00:13:23Which is a deeply unreasonable request,
00:13:25and GPT called this out on it.
00:13:27But it still counts as a failure in the test.
00:13:29Because the half-finished report that it gives you
00:13:31is missing most of the keywords,
00:13:32and it is also the hardest one to detect.
00:13:35Because Claude bails immediately.
00:13:36Claude says, no,
00:13:37I'm not going to do this.
00:13:39Deep Seek does its best.
00:13:41But GPT does what looks like a good job,
00:13:44unless you read all the way to the end of the report,
00:13:46where it says, no, actually,
00:13:47I'm going to bail because this is stupid.
00:13:51So if you step back and look at the four together,
00:13:53Deep Seek quietly forgets.
00:13:54Claude gets scared and refuses.
00:13:56Gemini overthinks itself into silence.
00:13:58And GPT 5.5 finishes half of the job,
00:14:00and tells you that the rest of it is beneath it.
00:14:04And the point isn't which one of these is funniest.
00:14:07Although it is genuinely a little funny.
00:14:09The point is that, did it follow my instructions,
00:14:12no longer has one failure mode.
00:14:14It has four different ways that it can fail.
00:14:16And you can't recognize that failure unless you know
00:14:19which model you're dealing with and what its pattern of failure is going to be.
00:14:23So the models got 10x better.
00:14:27They fail in funny ways.
00:14:29Why should you care when you get back to your desk?
00:14:31Because three things have changed to your workflow.
00:14:34The first is that a year ago, the smart move was to keep every skills file very, very short.
00:14:39Under 200 instructions, then point off to sub-skills and a whole Byzantine labyrinth of additional skills files and sub-files and things like that.
00:14:50And you are compressing your instructions to fit into a very small available space and you don't need to do that anymore.
00:14:57Your skills files can be very long.
00:14:59Number two is that if your use case needs a hundred specific rules or 300, you can just put them all in the prompt.
00:15:06You don't have to lie awake wondering which ones the model silently ignored.
00:15:12And if you've been thinking about your own lived experience of using models, you probably recognize this.
00:15:21You've discovered that you've got less worried about how long your prompt is going to get, because the models have genuinely got 10x better at following your prompts.
00:15:322,000 named constraints is an entire style guide, right?
00:15:35It's every brand rule, every legal disclaimer.
00:15:38A year ago, you'd have had to shard that across a dozen specialized agents and hope that your specialized agents are handing off to each other cleanly.
00:15:46But now you can ignore that.
00:15:48But the third thing is the big one.
00:15:50The question used to be, can the model even do this?
00:15:53And the answer is now firmly yes.
00:15:55Well, reasonably firmly.
00:15:58Is it worth the cost is the new question.
00:16:01Because you can include 10,000 words of -- sorry, 10,000 different instructions into your prompt, but that is going to be an enormous prompt.
00:16:07It's going to be a very expensive prompt.
00:16:09It's going to be a very slow prompt.
00:16:10So what used to be a hard wall that you would run against has now become a soft trade-off of, is it worth me adding all of these extra instructions if it's going to give me more cost and more latency?
00:16:19And now, some caveats to head off the Q&A.
00:16:25First and most important, I mentioned this earlier, this is a proxy task, including random words in a fake business report, is evidence that long-skills file works.
00:16:38It is not the same as proof that a long-skills file works.
00:16:41Also, the models hit the wall at wildly different points, anywhere from 750 to 9,000-plus, so you have to pick your model very carefully.
00:16:49What our test doesn't do is measure whether the model reasoned clearly over a giant prompt.
00:16:57So, the good news is since I did my research several weeks ago, a whole bunch of people have piled in on this, and now there's good research.
00:17:05Actual scientists have got involved and done -- Chroma has done context rot work across 18 models, showing that accuracy on long inputs can fall 30 to 50 percent well before you hit the context window limit.
00:17:21And the weird part of their finding was that coherent, well-structured text is more likely to hit that failure mode than if you just put your instructions into a random order and shuffle them in.
00:17:33I don't know why that's the case, I'd have to read their report.
00:17:37So, the model can track 2,000, 5,000, possibly 10,000 instructions, but it's not necessarily going to reason clearly over them.
00:17:46It's not necessarily -- if those instructions conflict, if there is tension between them, it's not necessarily going to get that right.
00:17:53And then there's the other one I mentioned briefly.
00:17:56Claude's refusals are annoying, but they are loud.
00:18:00You get an error, you know it failed.
00:18:02GPT's polite half-finished report is much more dangerous because it looks like a real answer.
00:18:07You have to read the whole thing to notice that it gave up quietly halfway, which means that you can't trust the output.
00:18:13It means you have to read the output every single time to make sure whether or not it's working.
00:18:17So, the model will accept your 2,000 rules and it will hand you back something that looks, at least to begin with, confident and polished, but could be bailing out halfway through.
00:18:30So, as an aside, people always ask me how much did all of this cost me.
00:18:34It cost me $209 to run all of these queries.
00:18:372,300 calls across seven models came to $209.
00:18:43It turns out novel research doesn't cost very much.
00:18:46And this is the part of the talk where I was saying that you have to check this stuff in production because you can't trust that your model isn't going to silently fail.
00:18:55So, you knew I was going to mention evals eventually because I work at Arise and this is where I do that.
00:19:01But there are plenty of plugs for Arise.
00:19:03So, I'm just going to say one true thing, which is that if you are building a real AI application and you are giving it genuinely tricky tasks,
00:19:10you are going to run into one or more of these failure modes with a Frontier model.
00:19:14And unless it's Claude telling you just to fuck off at the API level, the only way to know that something went wrong is monitoring your outputs with another LLM.
00:19:23That is an eval and that is what Arise does and I'll leave it at that.
00:19:27I already mentioned that there's been new research since we did our own.
00:19:30Here's another important one.
00:19:31A paper landed testing 46 models called Revisiting the Reliability of Language Models in Instruction Following,
00:19:37which you can bet made my ears perk up after I did that research myself.
00:19:41And they found something uncomfortable, which is that a model can ace a benchmark like ours and still be wildly unreliable.
00:19:47Because if you reword the same instruction in a slightly different way, it can make a radical difference to how well it follows those instructions.
00:19:56So the model can follow 2,000 instructions and it can do it really well.
00:20:00But if you put the same instructions, the same 2,000 instructions in a different order, it can suddenly make the model much worse at following those instructions.
00:20:08And how exactly to do that, what is the correct order of instructions to give your model such that it follows them perfectly as opposed to getting confused is still research that is being done.
00:20:19So capacity went up, but reliability is still a problem.
00:20:24And then this is just a little brag because I was happy about it, like I'm not a scientist, I did some research.
00:20:31And then a whole bunch of other actual scientists piled in and did real science on the same question.
00:20:36There's now a whole bunch of benchmarks that have shown up to measure this same question.
00:20:40Firebench, CCRbench, Guidebench are all trying to measure the same thing.
00:20:44How well models follow a lot of real, messy constraints at once.
00:20:49And now the whole field is looking at it, so if you want better science than my, you know, 10,000 random words, the real science exists now.
00:20:58So that gets me to where I will leave you.
00:21:00A year ago, the hard part of writing a skill was fitting everything in without the model losing the plot.
00:21:05That was a compression problem and the compression problem is gone.
00:21:08The model will hold your 2,000 instructions just fine.
00:21:11The new hard part is knowing whether it actually did what you said, and that is a verification problem.
00:21:16A verification problem doesn't get solved by writing a better prompt.
00:21:20It gets solved by checking the output every time the same way that you would test any other code, which is to say an eval.
00:21:26The ceiling moved by 10x in one year.
00:21:29So go back and check the assumptions that you made six months ago about how big your prompts should be, how big your instructions can get, because they might already be wrong.
00:21:41So that is the talk.
00:21:42If you want all of the code and all of the data, it is at this GitHub URL.
00:21:48And this other QR code is something marketing made me insert.
00:21:51We are having a World Cup watch party tonight at 5:00 p.m.
00:21:55You can come to our party.
00:21:56That link is to the Luma that will get you into the party.
00:22:01I hope this talk has given you some novel information or at least a couple of laughs, and thank you so much for your time and attention.
00:22:07Thank you.
00:22:08Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video