스크립트
00:00:00Hi, everybody. Thank you for coming today. Welcome to a talk about nothing. Sorry, a talk about
00:00:21retrieval. My name is Yuval. I work at AI21, which is essentially an AI research lab. And today,
00:00:29I want to talk to you about something that most people don't want to talk about, which is chunking.
00:00:37And I hope to convince you by the end that chunking isn't dead and there is something to do with that.
00:00:45And really, if you are at Axe, LinkedIn, wherever, you've probably seen that Rugg is dead, right?
00:00:53I think people also killed MCP lately. And Rugg is dead again. Long live the organic retrieval,
00:01:00organic search. And there is come a time where you have to ask yourself, how many times can Rugg die?
00:01:07Right? And even when someone says, well, Rugg isn't dead, like Jerry, the CEO of Llama Index,
00:01:14they still have to kill something. And apparently, this something is chunking. Like, don't invest in it.
00:01:23Don't do it. And this is the reason that people said that chunking is dead, because everybody is using
00:01:30agentic search now, right? You have greps, you have ls, you have finds. All of these are great,
00:01:36but these are still not enough if you have a lot of data and you have various amount of queries.
00:01:46Just a second. Okay. And I think that the main reason that a lot of people don't like to talk
00:01:51about chunking, it's because it's not the fun part, right? In every Rugg or files system,
00:02:00we have two stages. The first stage is the boring one, as you may. The one you do in the beginning,
00:02:07you have a lot of data. You have to preprocess it. You have to decide on the chunk size. And then you
00:02:13have to store everything in a vector DB. The other part is the retrieval part, essentially the one
00:02:20that happens per query. This is something which is much easier to do, right? It's much easier to
00:02:25optimize. You can use all your queries and then you can play with the max K, top K, sorry, you can play
00:02:32with the hybrid search maybe, those kind of things. Much more fun to do retrieval tuning, right?
00:02:40So I will claim that if we have to kill something, if something has to be dead, then it's probably
00:02:47retrieval tuning. And yes, agentic search probably killed that. But still, agentic search, even if we
00:02:55can accept the fact that it killed retrieval tuning, it's still not good enough when you have a lot of
00:03:02data. It costs a lot of money. I don't think I have to mention that anymore. Token maxing
00:03:09is like something that everybody's talking about. And the thing underneath, which is if the data itself
00:03:17is not ordered in a right way in your folders, in your directories, you still get something which is
00:03:25inefficient. So let's try to think of a timely example, right? The FIFA World Cup is now. And let's
00:03:33imagine that we have a data set that contains all the FIFA World Cup. So every directory is, let's say,
00:03:40the 98 one, the 2002 one, and so on, and so on. But if your query asks which team won the most World
00:03:50Cup, you can't just go to a folder and access that. You have to go to every folder, see who won, and
00:03:57then aggregate this together, which is very inefficient. The answer right away is Brazil, I hope, at least
00:04:03according to the time that this conversation is happening. So retrieval didn't actually die. We're not
00:04:13killing anything in this lecture. It got devoted into plumbing. And I think that everybody who worked on
00:04:21any rug system know the feeling. Day one, or week one, or maybe even month one, if you're very thorough,
00:04:29you're picking some sort of a chunk size. Let's say 512. And maybe you're probably putting some overlap,
00:04:36right, 10%, 20%, so on, indexing everything, and forget all about it. And you can, right,
00:04:43we talk a lot about the fixed chunking strategies, where if your chunk is something which is too big,
00:04:49right, so you get the whole picture, which is nice, but you're losing a lot of the nuance. And all the
00:04:54chunks will not get meaningful embeddings. Where if you will choose your chunks to be too small,
00:05:00you're getting the big picture lost. And really, it won't be as efficient. So what this tells us is
00:05:07that chunking is essentially a lossy compression. No matter what we're doing, we're losing something.
00:05:15And I will claim that there is no right chunk size. And a lot of you who worked on data will say,
00:05:23no, but we have this corpus, we have this data set, and we really used and we optimized our system to
00:05:29work very, very well on this data. And we thought so, too. We had a lot of experience with it,
00:05:35with a lot of different types of agents and systems and workflows that you can really,
00:05:40and, right, you think about benchmarks, how easy it is to overfit your model to a benchmark.
00:05:48But not with RUG. It doesn't happen there. And you cannot really optimize it per dataset. And I will
00:05:54show you how to make sure that it is query dependent. And how can I be so sure? How can I claim such a
00:06:00thing? Because we ran experiments, and we tested it, and now I'm going to present it to you. So what we
00:06:06did, instead of saying what is the best chunk size per data, let's find out. Let's actually take a
00:06:14data set and duplicate this data set several times. In this case, six times. In every duplication,
00:06:23in every instance, the chunk size is different. So we have a database with a chunk size of 2,000,
00:06:28a database with a chunk size of 1,000, and so on, and so on. And we did it with several data sets,
00:06:36so QMSUM, which is a meeting transcript dataset. Narrative QA, which is question answering on
00:06:41novels. And Seinfeld dataset, which is a trivia about nothing. Not really. It's trivia questions
00:06:49about the transcripts of Seinfeld. It's kind of a trolling dataset that we build in-house. We also
00:06:56published it if anybody wants the link at the end. And we tested on all of them to see what happens.
00:07:03And first of all, we just wanted to see, for every dataset, which chunk size is the best. And what
00:07:10we're seeing here is an example from the Seinfeld dataset, where essentially two queries, which are
00:07:16different by nature, get different results based on that chunk size. So the first question, what is the
00:07:24name for Jerry's favorite shirt? You can see this is a very focused question, a very specific question.
00:07:28The answer to it is probably very contained. And this is something that a smaller chunk size
00:07:33we'll do best in. And you can see rank one versus rank below 50. Between 100 tokens fixed
00:07:43chunk size to 100. Whereas a question like, who does Jerry describe as his nemesis and pure evil,
00:07:50which I'm not even that big of a Seinfeld fan, and I know it's Newman. But if you look at the
00:07:56transcript, it's not something you can find that easily. And you can see that it really changes, right? If you use
00:08:02small chunk size, you will not get the answer. And what we did to really -- after we ran
00:08:10all of these things and all of these things, we've noticed that, we said, what if we had an oracle, or a genie, if you
00:08:19want, that can tell us, for every query, what is the best chunk size to do retrieval for. This essentially is the
00:08:25oracle experiment. This is what we wanted to know to see the potential. This is not -- we already have the
00:08:33ability to be building a system here. We just want to see what is the potential that we have here. And what you can
00:08:38see here, okay, in this graph, all the blue -- first of all, the y-axis is the recall. Higher is better. The
00:08:46x-axis is the number of retrieved chunks. So it's recall at k versus k. You can see all the blue lines, probably
00:08:52indistinguishable, but each of them is the performance for a fixed chunk size. Whereas the orange one is the
00:09:01oracle line. This is -- for every query, we took the best one out of these. And you can see it happens
00:09:09across several data sets. In a lot of them, you can actually see that the blue lines intersect with
00:09:16each other, meaning that indeed for a lot of the data sets, no chunk size actually dominates. And what's
00:09:24more interesting is that there is a lot of potential. The gap, which you can see between the orange line
00:09:30and all the blue lines, is big. And when I say big, it's something like 20 to 40 percent just from
00:09:38doing strategy on chunking. And very simple strategy, may I add. And this is -- this gap, this is what the
00:09:47choice of 512 or 1,000 or whatever, right? This number is just arbitrary. This is what it costs you. And I
00:09:56think that the problem here is -- it's a bit tricky because it's kind of like an information problem that we don't have the
00:10:07information that we need at every stage. And what do I mean by that? If I'm looking at the indexing part,
00:10:12where I do have control over the chunk size, I don't know what the queries will be.
00:10:18I can guess. I can maybe estimate. I can try. But I don't know what the queries will be, so I cannot
00:10:25adjust my chunk size accordingly. And the retrieval part, where I do have my queries,
00:10:31I cannot control the chunk size, right? It's already fixed. And I obviously will not do the entire
00:10:37process per query from the beginning. So we looked at prior works, such as notably entropic,
00:10:47contextual retrieval, where they enrich every chunk, and others that essentially try to improve the latent
00:10:54space of every chunk. But this is not the direction that we went. All of them just stayed in the model
00:11:01of let's work with a fixed chunk size, whereas we took a different approach. And we said, why commit to
00:11:08one where we can commit to several? And we call it the multi-scale indexing. Essentially, we're just
00:11:16doing what we've seen before. So we're checking the database. We duplicate it and chunk it with several
00:11:26chunk sizes or window sizes. And then, this is what happens at the indexing. And then at retrieval time,
00:11:34we are querying all of them. So if we had n duplicates of the database and window sizes, we now have to run
00:11:43six different retrieval calls per query. Sorry, six as n. And how do we combine them? We obviously cannot
00:11:52use the oracle, right? The oracle is something that we have just for potential. In real life,
00:11:57we don't know the answer. But what we can do is to find some sort of merging algorithm. Now, you would say,
00:12:06when we look at it like this, what can be the issue? The fact that we have n ranking,
00:12:13but the rankings are for chunks. And chunks with different sizes are not really comparable,
00:12:19right? So instead, we opted to do something which is pretty popular these days. And a lot of the
00:12:25RUG systems actually work like this, that instead of just retrieving the chunk, when we're getting a
00:12:30chunk, we're retrieving the entire document, right? When context window grows, we want to give more and
00:12:35more context. And now, in this case, we have n rankings of the same documents, because they're
00:12:44not chunks anymore. And this, we can compare. And in this case, you can think of retrieval as essentially
00:12:50just voting. All right? So it's not purely ranking. We don't have run ranking, and then we're doing it
00:12:56re-rank. We're having n different ranks of the relevant documents, and we want to aggregate them
00:13:03all into one. That's why we are using something called RRF, Reciprocal Rank Fusion, which is pretty
00:13:11much a simple formula. We tried several things. This worked the best. And as you can see, it's not a
00:13:17model. It's not something that you have to do specifically. Like, especially, this is just a
00:13:23simple script that takes really no time. And this is how the full system looks like. So we have the
00:13:32indexing n times, then we query each query from every database, and we're using RRF to combine them
00:13:40all. And the results, you can guess that they're good. Otherwise, I would not be standing here and
00:13:48being way too much confident. All right? But you can see, we tested across several data sets, QMSum,
00:13:56Narrative QA, Seinfeld, and also Finance Bench. We took all of them, and it matches our bits the best
00:14:05fixed size. Let's see it in a graph. It's a bit hard to see here, so I'll walk it slowly. Every row
00:14:12here is chunk size. So you can see 50, 100, and so on. The bottom row is our method. This one, the one
00:14:21that you do from all of them and then combine. And every column is recall at something. So recall at one,
00:14:31two, three, up until 10. What you can see here is that two things, right? First of all, that across
00:14:39like recall at whatever, our method still wins, which you can think is very easy, but the fact that you
00:14:46have to combine all of them is not something which is very trivial. And also you can see that the quality
00:14:53actually increases. The heat map where you can see it becomes much greener. And again, this was just
00:14:58something that I wanted to show in large. Here you can see all four of the datasets where we do achieve
00:15:06better results. Really quite like 20, 30, 40 percent even in a lot of the things. Also, there are results
00:15:14that I did not show you here which are on MTab. You can see in our blog, I will put the link later, we're getting
00:15:22there also a lot of improvements somewhere between 10 to 40 percent depending on the dataset.
00:15:30Now, I'm not naive. I'm not going to claim here that this costs nothing. Obviously, there is a cost,
00:15:36right? No free launches. Everything has to come with something. And yes, this costs with extra memory.
00:15:43It costs something between 2 to 5, 0 of 1, right? A constant of additional memory where you have to keep
00:15:51all of those copies of the database. However, if you think about it, latency-wise, it doesn't really
00:15:59affect that because you can do all the retrieval part parallelly. And also, the RRF part doesn't really take a lot of time.
00:16:10I will say that this was a very nice research project that we did and we got really, really cool results.
00:16:16There are things to do, right? There are places to improve. There are future work to do. More precisely,
00:16:23we want to understand how many chunk sizes do we want and which, right? The fact that we worked with 50, 100,
00:16:32200, and so on was pretty arbitrary, to be honest. So we do need to figure out how to compute this and
00:16:40how to know how many copies exactly do you need. Also, go beyond RRF, right? The fact that we're
00:16:46using RRF is because it worked the best from the methods that we used, but it doesn't mean that there is
00:16:52no better method. And if I need to leave you with something, I would say that agents didn't kill
00:16:59retrieval. Nothing died. Come on. It's just infrastructure. And the bad part is that it's
00:17:07infrastructure from 2022. And with really simple methods, you can take your RAG system or anything that has to
00:17:17do with storing data and then retrieve it with 20 to 40 percent, again, without any something too
00:17:26sophisticated. So if you want to read more about it, you can read the blog. There is also an example
00:17:34code there and the Seinfeld data set. And that's it. I'm Yuval. Thank you so much for being here.
00:17:47I'll see you next time.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기