It’s Tokens All The Way Down: How RLMs are Different — Kevin Madura, AlixPartners
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Kevin Madura: Awesome. Thanks everyone for being here. My name is Kevin Madura. I'm from a company
00:00:17called Alex Partners. We're a consulting firm. I'm here to talk to you today about RLMs. Just
00:00:23curious, show of hands, who here is familiar with RLMs so we know how much time to spend on it?
00:00:29Kevin Madura: Okay, so not many. All right, that's good. We'll start with what an RLM is and why it's different. RLM is a recursive language model. Really, the key difference here is that it treats the context as an object that it can interact with symbolically in its environment. It differs from a tool call in the sense that typically when you do a tool call, it's JSON or some type of string that's being sent, being interpreted elsewhere,
00:00:59Kevin Madura: maybe by some other program and that's returning effectively as a string. The key difference here is that it's interacting with a symbolic environment. So typically, that's a REPL, Python REPL.
00:01:11Kevin Madura: So that's key difference number one. Key difference number two is that it has the ability to delegate to another LLM, often to itself. You can specify whether it's the same model or a different model, but fundamentally, because it lives in this environment, you can offload or make a sub call
00:01:28Kevin Madura: to another LM with particular parameters that also lives in that REPL environment. And so you get this ability to recursively decompose problems and apply and have the LLM basically decide how to apply certain logic or certain interpretations or write its own code to solve those problems.
00:01:48Kevin Madura: And then that recurses down. So the sub-LMs can do the same sort of thing in terms of understanding and interpreting what it thinks it needs to do. And I added this last one here.
00:01:58Kevin Madura: It's largely bitter lesson-pilled, in my opinion, right, and shared by Alex and the rest of the creators of it. But as models get better, you should be able to defer more and more to the model for it to kind of figure out on its own what it needs to do.
00:02:15Kevin Madura: So I don't know if this is the actual kind of starting point for our LMs. This is one that I consider to be one of the first kind of inklings of it. This is a tweet from Omar, who
00:02:28who is Alex's advisor for RLMs. And this was a concept that he had come up with, where it was basically an ability to use DSPy and some other techniques to take in arbitrary length inputs.
00:02:43And basically, the use case here would be summarizing an arbitrarily long document and coming up with a table of contents and some summary of that content.
00:02:52Kevin Madura: But at least to me, this is kind of the first inkling of, okay, context windows might not be something you need to deliberately manage, although there's, of course, benefits to doing so.
00:03:01Kevin Madura: There could be ways to exceed the context windows using some of these clever techniques.
00:03:07Kevin Madura: And so if you read the paper and some of the blog posts that are out there from Alex and Omar, I mean, it has demonstrably better performance on some of these long context tasks.
00:03:20Kevin Madura: So Oolong is one benchmark where the intent of the benchmark is to measure model performance on answering questions about excessively long context.
00:03:30Kevin Madura: This is another one, browse comp, where it needs to iterate through a large body and corpus of text and answer particular questions about it.
00:03:38Kevin Madura: You can see the blue line at the top there is the RLM. It's very good performance as compared to some of these other models.
00:03:44Kevin Madura: And even on the price curve, the purple is actually just using tool calling with GBT5 calling a BM25 tool, and that's actually even more expensive for worse performance than an RLM.
00:03:59Kevin Madura: So it's worth reading into if you're interested in some of the benchmarks and how RLMs perform.
00:04:04Kevin Madura: But fundamentally, an RLM, again, takes in your input and you're kind of deferring to the model about how to decompose the process,
00:04:13Kevin Madura: what code it needs to write, and it is very tightly integrated with the REPL itself.
00:04:18Kevin Madura: So it, by itself, defines what it needs to do.
00:04:21Kevin Madura: And so I kind of had this mental model in terms of, and I'm very DSPi-pilled, if you can tell by now,
00:04:27Kevin Madura: basically a student of Omar and the rest of the group there, where you have this relatively deterministic shell of what you want to do.
00:04:35Kevin Madura: Like, what is your intent? What is your actual task that you're trying to accomplish?
00:04:40Kevin Madura: You define that in terms of your inputs and your outputs and some type of guidance or prompt or what have you to the model to say,
00:04:48Kevin Madura: this is generally what I want to achieve. Go off and do it. Here's the things that you can expect as your input. Here's what I want out of it. Go figure out the rest.
00:04:56Kevin Madura: And so this applies for using something like DSPi, but I think it applies to RLMs as well, because you don't have to worry as much now about how the actual implementation works in the middle.
00:05:09Kevin Madura: You can just have some guarantees about the inputs and the outputs, and you can let the model figure out the rest of that part of it.
00:05:16Kevin Madura: So a lot of this comes down to, if you were at, I think it was Code in November in New York City,
00:05:23Kevin Madura: Dex had this great talk about just broader context engineering, and he coined something like the dumb zone,
00:05:28Kevin Madura: which it's kind of great out at the bottom there. But the point is that we all know that there's context rot, right?
00:05:34Kevin Madura: Once you fill up the context window to a certain degree, performance starts to degrade.
00:05:39Kevin Madura: And so RLMs somewhat get around this problem because the context itself doesn't fill up as quickly because you're deferring a lot of the subtasks to the submodels,
00:05:50Kevin Madura: and it's the full kind of context and the inputs aren't exposed to the context window itself.
00:05:57Kevin Madura: It lives as a variable in the REPL, and so the main LM can choose how to access that.
00:06:05Kevin Madura: It can offload some of these subtasks to sub-LMs, and really the only context that it gets back are the things that actually matter.
00:06:12Kevin Madura: So in terms of how it's meaningfully different, RAG, of course, you kind of just stuff the context window, you run into limits there.
00:06:19Kevin Madura: Agents are largely just bringing strings back, and you don't have this tight coupling between the logic, the execution, and the results.
00:06:29Kevin Madura: And so you still run into the same sort of problem there.
00:06:32Kevin Madura: Same thing with tool calling and code act, and then RLMs.
00:06:36Kevin Madura: As I mentioned, the LLM is actually just interacting with the context, the results, as variables in the REPL so that it can do additional computation on versus it trying to attend to all these different tokens in its context window.
00:06:51Kevin Madura: It's a meaningfully different way of the LLM interacting with the actual content itself.
00:06:58Kevin Madura: And so people always say, "Okay, what's the difference between that and coding agents?"
00:07:04Kevin Madura: In my mind, the largest difference is that the way that tool calls are done is passing strings back and forth.
00:07:13Kevin Madura: But you can see with the release recently of Workflows that Anthropic is doing something fairly similar.
00:07:21Kevin Madura: At the CAIS conference, I think it was Tariq or someone similar, mentioned the RLM paper as a key driver of Workflows and how they've implemented it.
00:07:34Kevin Madura: And you can see here the intermediate results for Workflows live in script variables, i.e. a variable in the context.
00:07:43Kevin Madura: So it's driving some of these breakthroughs and some of these techniques from the labs as well.
00:07:48Kevin Madura: I'll skip through this a bit just because I have about 10 minutes left, but generally speaking, when you want to use it, it's obviously for large or dense input context.
00:08:01Kevin Madura: An underexplored area is outputs as well.
00:08:05Kevin Madura: So if you have some type of task where you need to generate hundreds of thousands of lines or whatever it might be, RLMs, I think, would be a good candidate for that as well.
00:08:15Kevin Madura: Obviously tasks that are amenable to some type of decomposition.
00:08:19Kevin Madura: So if you want to look through the entire, I don't know, the whole tax code as an example and try and find loopholes or something, you can't obviously put all of that into context at once.
00:08:29Kevin Madura: You could use an LLM to crunch through all of that and iteratively explore and use subagents to explore interesting areas of the tax code, bring back those sections, and then reason over that.
00:08:41Kevin Madura: And then just generally for longer horizon sessions.
00:08:44Kevin Madura: And when you want to skip it, of course, it would be something that fits in context.
00:08:47Kevin Madura: You want something that's low latency or the model itself is as strong of a coder.
00:08:52Kevin Madura: And our friend Raymond here did some great performance testing on the long chain of thought benchmark.
00:08:59Kevin Madura: I'll leave this link as a leave behind after, but just to give you a sense of how well it performs in some of these tasks, it's a meaningful jump overall from 2.6 to 45.4% accuracy on many of these tasks.
00:09:14Kevin Madura: And you can see it performs really well on things that are amenable to code, so logic puzzles and chess and chemistry and things like that, where it can dynamically write code, bring in only the relevant part of the context, compute that, and then return the result, where the main model is really just harvesting the results from the sub-LMs instead of trying to do that by itself.
00:09:36Kevin Madura: I put together a few just super simple examples.
00:09:41Kevin Madura: I mean, these are kind of somewhat unfair, I suppose, to the base model, but it makes the point that there are certain tasks that base models just aren't really fit to do themselves because they have to attend all these different tokens at once in the context window where you need or want to use some type of coding approach to that.
00:10:01Kevin Madura: So in this random example, summing 12 numbers that are buried across 30,000 tokens, the LLM trying to figure all that out by itself and give you the answer isn't always going to work as well as something that you can write regex for or something similar.
00:10:16Kevin Madura: And then the same sort of thing, particularly for data frames, and we'll walk through a brief example here, where because the LLM can interact with the data frame within the REPL,
00:10:28Kevin Madura: it just has a much better understanding of the content and can iterate through that much more quickly than having to pass tool calls back and forth in terms of like JSON strings and that sort of thing.
00:10:40Kevin Madura: And then I threw this in there in terms of running the same experiments with a coding agent.
00:10:44Kevin Madura: Now, I didn't look into this too deeply.
00:10:46Kevin Madura: There's probably some unfair math going on here, but you can see that it was totally bloated in terms of the way that cloud code tried to solve these tasks.
00:10:56Kevin Madura: So there's more work to be done there, of course, in terms of like running experiments to compare base models versus RLMs versus something like a coding agent.
00:11:06Kevin Madura: But there's just for certain tasks for like production workloads.
00:11:09Kevin Madura: My sense is you probably don't want to just do cloud dash P your prompt and like hope for a good result.
00:11:16Kevin Madura: Like you want more of a structured approach to your inputs, your outputs, and you want to define pipeline for doing so.
00:11:22Kevin Madura: Which reduces your cost, reduces your complexity, reduces your bloat, all that sort of thing where RLMs can shine.
00:11:30Kevin Madura: So in the real world, there are a bunch of different open source libraries that implement RLMs at some level.
00:11:38Kevin Madura: Some of them are more RLM focused, like a predict RLM would be a good example of that versus others are kind of just integrating it into the broader approach or the broader framework.
00:11:50Kevin Madura: That's really interesting work that's being done there.
00:11:56Kevin Madura: Predict RLM is more focused on like knowledge work.
00:11:59Kevin Madura: So it works with spreadsheets and PDFs and that sort of thing.
00:12:02Kevin Madura: And then fast RLM.
00:12:03Kevin Madura: And then there's a tweet yesterday from this guy, Sam Hogan, who runs inference.net.
00:12:09Kevin Madura: He's using an RLM to basically run and extract insights from your particular production workload traces.
00:12:18Kevin Madura: So that they can see what makes sense to defer off to something like a GLM 5.2 and do that iteratively and automatically as your traffic goes through.
00:12:27Kevin Madura: So point being, you don't have to worry about context engineering.
00:12:32Kevin Madura: You can kind of just throw the RLM at it and have it figure it out.
00:12:35Kevin Madura: I only have five minutes left, so we won't go through this whole example and I'll skip to some of the traces because that's probably the most interesting.
00:12:43Kevin Madura: But this is all you would really need to do in terms of a simple, in this case, it's like a cohort retention analysis, something that you might give to a data scientist.
00:12:53Kevin Madura: But this concept of applying an RLM to a complex data structure like a data frame becomes very easy to do.
00:13:02Kevin Madura: And so I think this is all the code you need to do it where I'm feeding in three different data frames.
00:13:06Kevin Madura: I'm saying, these are the sorts of things you need to look for.
00:13:09Kevin Madura: These are the output types that I want.
00:13:11Kevin Madura: And then just let the RLM go on it.
00:13:14Kevin Madura: And I'll show you some of the traces.
00:13:17Kevin Madura: And so it has its own REPL where it can interact with those data frames.
00:13:23Kevin Madura: And you can see it reasoning through, okay, first I need to do this.
00:13:26Kevin Madura: It's writing the code.
00:13:27Kevin Madura: And because it's living in the REPL with the data frame, you don't have this additional bloat of the tool calls back and forth.
00:13:34Kevin Madura: It's actually interacting directly with the data frame as if it was typing in its own Jupyter notebook.
00:13:40Kevin Madura: And there's significant advantages for doing so.
00:13:43Kevin Madura: And so you can see the sorts of outputs that it gets as a result.
00:13:48Kevin Madura: And it by itself will iterate.
00:13:50Kevin Madura: And in this case, it didn't, but it has the option to defer to sub-LMs to do, okay, now I have this big, whatever, this big subset of the data.
00:13:59Kevin Madura: Sub-LM, go off and do this analysis, give me the result.
00:14:02Kevin Madura: And it can do that iteratively over time.
00:14:05Kevin Madura: But the point is that the LM is directly interacting with the data frame in its REPL and kind of iterating through the results.
00:14:15Kevin Madura: And so this platform compound is RLM and DSPy native.
00:14:20Kevin Madura: So it gives you this really nice breakdown of the reasoning.
00:14:23Kevin Madura: It separates out the code that's being generated.
00:14:26Kevin Madura: And ultimately you can see the final output, which is here where it's formatting.
00:14:34Kevin Madura: Okay, here are the key findings that I have, here are the recommendations.
00:14:37Kevin Madura: And then you have this final submit, which is the final answer that gives you the typed outputs that you had to find up front.
00:14:45Kevin Madura: And the key thing here is that the LLM itself is deciding when to stop.
00:14:49Kevin Madura: So you have this, you have a variable of max iteration.
00:14:53Kevin Madura: So you can decide whether you want it to have a maximum of 10 or 100 or whatever it is.
00:14:58Kevin Madura: But it will by itself explore the data, understand what needs to happen.
00:15:03Kevin Madura: And then when it itself is comfortable, it can run submit and give you the final output.
00:15:08Kevin Madura: But again, being better lesson-pilled, this will get better over time.
00:15:12Kevin Madura: You can kind of just defer everything and it will figure out what to do.
00:15:16Kevin Madura: And so the hope would be you don't have to, I mean, we're already, you know, whatever this is, 20 lines of code or something.
00:15:23Kevin Madura: But you can, you can see a world where you can continue to go up levels of abstraction.
00:15:29Kevin Madura: As long as you can define what your objective is and what you want it to do, the model will kind of figure out the rest.
00:15:35Kevin Madura: So we just walked through a bunch of this, but these are the different steps that it took in this example and the code that it wrote.
00:15:43Kevin Madura: And then I'll just breeze through a few real world case studies and where it's actually being used.
00:15:48Kevin Madura: So I mentioned PredictRLM before.
00:15:50Kevin Madura: So the company Trampoline AI, I think it is, they're doing really interesting work in applying RLMs, like I mentioned before, for different pieces of knowledge work.
00:16:01Kevin Madura: So natively interacting with PDFs and spreadsheets and that sort of thing.
00:16:06Kevin Madura: So in this relatively simple example, okay, I have a bunch of, I have a directory of invoices that I need to create one consolidated inventory out of.
00:16:15Kevin Madura: As we all know, invoices can be complicated, they can be very long, they can kind of be all over the place.
00:16:22Kevin Madura: To do that today without RLMs or this sort of like framework gets very complicated very quickly.
00:16:30Kevin Madura: I have a lot of battle scars to prove it.
00:16:32Kevin Madura: But with something like an RLM, you don't need to worry as much about, okay, if I have a 200 page invoice or contract or whatever it is, you can let the RLM just churn through all of that and give you the result.
00:16:44Kevin Madura: Instead of having to worry about chunking and embedding maybe and doing all these different strategies to try and get around the context window management that we've all had to do previously.
00:16:56Kevin Madura: So it allows you, the point there is that you can focus on the abstractions and what you actually want to do instead of the context engineering itself, which I think is a really helpful output of all this.
00:17:09Kevin Madura: An interesting tidbit for all the DSPy fans in the room, predict RLM uses DSPy to determine the schemas between the main LM and the sub LM calls, which I personally think is a nice feature because you have a lot more readability and maintainability.
00:17:27Kevin Madura: So you understand exactly what the model is trying to achieve and the model can be much more precise and prescriptive about the types of data that it's looking for from the sub LM.
00:17:37Kevin Madura: And I would want to do some experiments to test this out.
00:17:41Kevin Madura: But I would think that this would improve performance for cheaper models like a Quinn or some of the other ones because you're specifying the inputs and outputs and you're enforcing those types coming back.
00:17:51Kevin Madura: And so you get all the benefits of the RLM being able to churn through all this information, but you have a lot more of the structure in between where, when it's handing off to a sub LM, it enforces some of those schemas.
00:18:05Kevin Madura: This is an example from an AWS engineer from a couple of days ago.
00:18:12Kevin Madura: We were just kind of playing around with it.
00:18:14Kevin Madura: But I just thought it was a nice example of you can kind of just throw arbitrary data at RLMs.
00:18:21Kevin Madura: In this case, it was a bunch of log data to surface some interesting results and he found it useful.
00:18:30Kevin Madura: There's a project called Halo, which uses an RLM to look at traces of different agent tasks.
00:18:40Kevin Madura: And basically the promise of Halo is that instead of optimizing a particular workflow or DSPy or other framework structure itself, it's actually iterating on the harness.
00:18:55Kevin Madura: So it's like a meta abstraction almost or meta optimization of the harness itself and it uses an RLM because as we all know, tracing can get very long and complicated.
00:19:06Kevin Madura: So the RLM can not only take in all of that context, but also leverage the structure of those traces to recommend a better harness.
00:19:18Kevin Madura: And then this last one, this is all the code you need.
00:19:22Kevin Madura: I ran this little experiment.
00:19:24Kevin Madura: There's an intentionally vulnerable application called, it's from OWASP, but basically it's a web app with a bunch of vulnerabilities in it.
00:19:32Kevin Madura: This is all the code you need on the right hand side to run basically an agent to run through whatever it is, 500,000 lines of code to generate some type of security report.
00:19:42Kevin Madura: That's just an arbitrary example.
00:19:44Kevin Madura: But the point is you don't need a lot of context engineering.
00:19:48Kevin Madura: You don't need a lot of structure around it to achieve what you want to do.
00:19:51Kevin Madura: And so you can feed in an arbitrary size code base into this and get some type of insights out.
00:19:58Kevin Madura: So you can imagine that being applied to other areas as well.
00:20:01Kevin Madura: So I know I rushed through everything a little bit, but I'm happy to answer questions afterwards.
00:20:07Kevin Madura: I'll leave you with this, the biggest promise I see here is just imagine a world where the models are actually post-trained and actually like RLM aware.
00:20:17Kevin Madura: I think things will get pretty crazy pretty quick when they actually know how to use and kind of take advantage of the RLM methodology natively.
00:20:27Kevin Madura: So thank you so much for your time.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video