Where RL Will Take Search — Maximilian-David Rumpf, SID.ai

English
AAI Engineer
컴퓨터/소프트웨어창업/스타트업AI/미래기술

스크립트

00:00:00-
00:00:12- Okay, today I'm gonna talk about
00:00:14where reinforcement learning will take search.
00:00:18And some background on me.
00:00:22I'm the founder and CEO of Cit.AI.
00:00:24We're a stealth-ish AI lab for search.
00:00:27We're backed by some pretty amazing people,
00:00:30and we're hiring.
00:00:35Okay, agents are a new paradigm for search.
00:00:40You can now get vastly higher quality results.
00:00:45Twice as likely to find the right documents.
00:00:49But it's incredibly expensive.
00:00:52About 100 to 1,000 times more expensive
00:00:54than what you'd get out of a classical search query.
00:00:58And it is extremely slow.
00:00:59You're looking at minutes and not milliseconds,
00:01:02like Turbo Puffer.
00:01:05And what this means is that agents spend 30 to 50%
00:01:10of their tokens on searching.
00:01:11And this is usually at the beginning of some task.
00:01:14It finds the right context to then do whatever
00:01:17what you ask it to do.
00:01:19And the idea here is quite simple.
00:01:21First, instead of having the main agent do the searching,
00:01:25you pass the searching to a sub-agent,
00:01:28and you train a model to be a great sub-agent.
00:01:32And the question here that we'll answer today
00:01:35is how much cheaper and faster can we make this
00:01:38with reinforcement learning?
00:01:40And this is really our target.
00:01:42So this is a benchmark across legal, finance,
00:01:45knowledge bases, science, email,
00:01:48a bunch of different tasks, some academic benchmarks,
00:01:50some internal benchmarks.
00:01:52And this is where you currently are.
00:01:58You see Reranker and Vector-only performance
00:02:00at the bottom.
00:02:01And you can see frontier models essentially
00:02:03kind of like go through here at the cost of spending
00:02:07many, many minutes per question.
00:02:10And can we get a model to kind of like be extremely accurate,
00:02:14have extremely high recall,
00:02:16but also be incredibly fast and cheap?
00:02:19And let's quickly look at classical search.
00:02:23This is the pipeline that many of you guys
00:02:25will be familiar with.
00:02:26A question comes in, you might have an LLM
00:02:29that rewrites the question.
00:02:30You then execute that on a search backend.
00:02:33You might have a Reranker
00:02:35and you get your results at the end of the day.
00:02:38It is essentially a pipeline of chained,
00:02:40locally optimized models.
00:02:43And all of the decisions are baked in at design time.
00:02:47And you expand a fixed amount of compute per question.
00:02:51And this one is really important.
00:02:53The Reranker might know that the results
00:02:55are insufficient at answering the question,
00:02:58but the Reranker can't take action.
00:03:00It can only essentially return the results
00:03:02even when they're bad.
00:03:03And what this means is that in practice,
00:03:06a pipeline like this accrues a long tail of failure
00:03:09where unexpected questions come
00:03:11that the designer didn't have something for.
00:03:15And in practice, this usually means people add
00:03:17lots of edge cases to essentially fix these.
00:03:21But of course, you can't design infinite edge cases,
00:03:23and you can't add infinite tweaks.
00:03:27And so the strategy here is one that we've seen before.
00:03:32Machine design outperforms human design.
00:03:36And we saw this in computer vision
00:03:38where you had your primitive edge detection algorithms.
00:03:42You then had the box-around-a-dog generation of models
00:03:45that were very good at this very narrow task
00:03:49and locally optimized for it.
00:03:50And then you had VLMs that were extremely good
00:03:53at all parts of the search pipeline.
00:03:54You saw this again with chess, with IBM Deep Blue
00:03:58being largely a collection of human-written rules,
00:04:01Stockfish bridging the two, and then Alpha Zero and Mu Zero
00:04:04essentially completely just putting it all inside of the model.
00:04:07And we're gonna see something similar happen to search
00:04:11where we have our existing algorithms
00:04:13like BM25 and PageRank.
00:04:16Then we had an evolution from that with small models
00:04:19that did some task very well, like vectors and re-rankers.
00:04:23And now essentially this new paradigm of pure RL
00:04:28where we actually don't make any design decisions
00:04:30into the model.
00:04:32And what this looks like in practice is something like this.
00:04:37So you have one model, it goes back and forth
00:04:41with the database, it can search, it can read results,
00:04:43it can iterate, it can search again until it is happy.
00:04:47It can set metadata filters on the fly,
00:04:50it can constrain its search, it can try as much as it wants.
00:04:55And in the end it produces a ranked list of results.
00:05:00And what you get is a model that makes all of the decisions
00:05:03and can adapt to any question on the fly,
00:05:06and for example, use much more compute
00:05:07if a user asks a very difficult question.
00:05:11And what helps us here is that search is verifiable.
00:05:15And reinforcement learning needs rewards
00:05:17that are verifiable and grindable.
00:05:19Verifiable here means for a given question,
00:05:22did you find the correct document?
00:05:23And we can design this and tell this quite easily.
00:05:28And is there an environment where the model can attempt this question
00:05:32loads and loads of times?
00:05:33And in practice for us this means thousands of times per second
00:05:37during a training run.
00:05:40And the second part is kind of like,
00:05:42can we turn the models that are currently very general
00:05:45and very general purpose into something that is much more specialized?
00:05:49And it turns out we don't actually need most of the parts
00:05:51of a language model to be extremely performant at search.
00:05:55And similarly, with like CPUs and GPUs and ASICs,
00:05:58a CPU in theory can do anything that a GPU can do,
00:06:02but you would never want to use a CPU to do LLM inference, for example,
00:06:06because the much more specialized version is much more effective.
00:06:10And this really makes search an ideal target for RL.
00:06:15And this is what happens when you train a model on this task.
00:06:19And so again, here we added the vector and Reranker-only baselines.
00:06:22This is of an earlier task.
00:06:24And what we see is that search quality increases
00:06:26very predictably with compute.
00:06:28And we can mix in other rewards like latency
00:06:30and different kind of like retrieval strategies
00:06:33to make it even more performant.
00:06:35Importantly, we don't really tell the model what to do.
00:06:38Much like in AlphaZero and chess,
00:06:40we want it to discover its own strategies
00:06:42and its own tricks to essentially use search well.
00:06:45And we don't know whether this method has no ceiling,
00:06:50but we're definitely not yet seeing a ceiling to this approach.
00:06:54And these are the results.
00:06:56So this is the same chart as before,
00:06:58and this is SID1 and then SID1
00:07:00with some parallel execution on the left-hand side.
00:07:04And so what this ends up meaning is
00:07:06you're about 20 times faster.
00:07:09So instead of taking around two minutes,
00:07:10you take around five seconds on average,
00:07:12and it's about 100 times cheaper
00:07:14than using a frontier model for this task.
00:07:18There's some more detail here,
00:07:19but like, yeah, the cost and kind of like speed
00:07:22are just completely incomparable.
00:07:25You can, yeah.
00:07:27It's not quite at the latency of a vector and re-ranker pipeline,
00:07:32but in practice, we think we can get there quite quickly.
00:07:36And how does this look like in production?
00:07:38So this is a usual kind of like agent execution trace.
00:07:41The agent does some searching here.
00:07:42It finds some good stuff.
00:07:43It finds some bad stuff.
00:07:45But all of the bad stuff that it finds
00:07:47is essentially polluting its own context window.
00:07:50And what we can instead do
00:07:51is use a sub-agent here
00:07:54that does all of the searching and thinking
00:07:56and iterating for the main agent,
00:07:57and the main agent only ever sees great results.
00:08:00And this means that the main agent sees more good stuff,
00:08:03which means it's more likely to be correct,
00:08:06and it is also extremely cost-effective,
00:08:10where those 30 to 50% tokens
00:08:14that were earlier used by the main agent
00:08:16to do searching can now be passed off
00:08:17to this just, you know, 100x cheaper search sub-agent.
00:08:25And where will this take us?
00:08:29Scaling RL will give us arbitrarily good search in any domain.
00:08:35And RL models will become even faster,
00:08:38which will allow them to be used
00:08:39in things like voice and e-commerce.
00:08:42They'll become even cheaper than we are currently.
00:08:45So the charts that you saw there,
00:08:47I think we can move even further.
00:08:50And better search will unlock more knowledge work tasks.
00:08:55The web is actually quite small
00:08:56in comparison to the entirety of data that is out there.
00:09:03And the most valuable information is not on the internet.
00:09:06For example, how to run JP Morgan is nowhere on the web,
00:09:09but it is deep inside of the databases at JP Morgan.
00:09:14So that's it for me.
00:09:16Thank you.
00:09:16Thank you.

핵심 요약

Applying reinforcement learning to search sub-agents reduces retrieval costs by 100x and speeds up execution from minutes to 5 seconds.

하이라이트

  • Agents spend 30% to 50% of their tokens on searching, mostly at the beginning of tasks.

  • Reinforcement learning models achieve search latency of about 5 seconds instead of 2 minutes.

  • Reinforcement learning search models run 100 times cheaper than frontier models.

  • Reinforcement learning enables sub-agents to handle search operations, keeping bad results out of the main agent's context window.

타임라인

Limitations of current agentic search

  • Agents deliver higher quality results but cost 100 to 1,000 times more than classical search queries.
  • Agent searches take minutes instead of milliseconds.
  • Agents consume 30% to 50% of their tokens purely on initial context searches.

Current agentic search systems struggle with high compute costs and long execution times. Passing search tasks to a dedicated sub-agent addresses these inefficiencies, forming the core target for reinforcement learning improvements across finance, legal, science, and knowledge base benchmarks.

Comparison with classical search pipelines

  • Classical search relies on a chained pipeline of locally optimized models designed at compile time.
  • Rerankers cannot take action when results are insufficient.
  • Pure reinforcement learning removes manual design decisions by allowing the model to interact dynamically with the database.

Traditional pipelines fix all decisions at design time and expand a static amount of compute per question, creating long tails of failure for unexpected queries. Machine design outperforms human design, mirroring the evolution seen in computer vision and chess engines from human rules to pure reinforcement learning models.

Reinforcement learning training and performance

  • Search provides verifiable and grindable rewards necessary for reinforcement learning.
  • Search quality increases predictably with compute scaling.
  • Trained models achieve a 20x speedup and a 100x cost reduction compared to frontier models.

Reinforcement learning models discover their own retrieval strategies by attempting questions thousands of times per second during training. Parallel execution brings average task times down from two minutes to about five seconds while cutting costs drastically.

Production architecture and future outlook

  • Sub-agents handle search and iteration while passing only high-quality results to the main agent.
  • Scaling reinforcement learning provides arbitrarily good search across any domain.
  • Improved search unlocks internal corporate databases containing the most valuable enterprise information.

Isolating search inside a cheaper sub-agent prevents context pollution and reduces token waste in production. Future iterations of reinforcement learning models will operate fast enough for voice and e-commerce applications, extracting valuable insights from private enterprise databases beyond the public web.

커뮤니티 글

아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!

이 영상에 대해 글쓰기