Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher
AAI Engineer
Computing/SoftwareBusiness NewsInternet Technology
Transcript
00:00:00So good afternoon everyone. My name is Harshal Jain and he is Tanmisha. And we
00:00:21would like to welcome you all in this two-hour workshop on the LLM inference.
00:00:26So the goal of this workshop is to understand this domain from the first
00:00:33principles, dive deeper into it and like understand what's going on throughout the
00:00:39industry. A bit of background about us. So I am a senior software engineer at
00:00:47Audible. I have been building MLAI data platforms for the past five years and on
00:00:53the size I have been writing this open source handbook on LLM inference. And
00:00:59Tanmay, he is the senior quantitative modeler at Xi'an's Bank Corporation. He
00:01:06recently completed his PhD and he has been actively doing research in the agent
00:01:12verifiers and the world models. So a quick show of hands here. WooHair is like brand new to the LLM inference.
00:01:24Okay, great. And WooHair has like deployed these models in production. They have been
00:01:33tuning it, they have been serving the production traffic. Okay, great. So this
00:01:42workshop is targeted towards the beginner and the intermediate level. And all of the
00:01:48slides and exercises they are in the repo I will share that soon. Here is the quick
00:01:56agenda for the workshop. We will start with the problem statement. We will try to
00:02:01understand a few of the pain points around LLM inference. Then we understand what causes those
00:02:08pain points and build our foundations from there. Then we will dive into like two
00:02:14kind of the optimizations that we do, like the model optimizations and the serving
00:02:19optimizations. And then we start learning about different serving engines that are
00:02:25available to deploy our LLM inference solutions in production. And we will showcase some benchmarks
00:02:32benchmarks and the decision chart on like which engine to use.
00:02:39Cool. So to understand the pain points, first we need to know what is like an LLM inference. So
00:02:47I'm probably a lot of us already know this. But yeah, anything that you ask your AI to do,
00:02:55like whether it be generate a video, audio, analyze any text, analyze your medical reports or like your
00:03:02tax bills, all of that is like an LLM inference. And this market is like approximately $23 billion today.
00:03:12Semi-analysis recently shared that if you want to model like a Google search queries with LLMs, you need like a profit drain of like $36 billion.
00:03:25And query cost has to be less than 0.5 cents to keep your search business profitable.
00:03:33On the other hand, the business insider mentioned like your AI has to be put on right and everyone has to start auditing and budgeting their token usage.
00:03:43And all of this is happening. Why? Because your hardware is limited, compute is expensive, your inference is expensive.
00:03:50And with the growing need of like more and more AI usage, this inference cost is rising more and more.
00:04:03So this stat, it's an old stat from the open AI, but it's still true.
00:04:11So if you look at the like training costs of the GPT-3, it was like around $4.6 million. It was a one-time cost.
00:04:19But if you see the inference costs that has been like, it's a recurring cost because it's a operating cost that scales with every user that comes in that every token that comes in every session that is being initiated on the like AI.
00:04:36And there are only two ways to basically counter this one way is you reduce your token usage.
00:04:47Alternative is you should try to optimize your inference solutions as an inference service provider for your customers and for yourself.
00:04:58And so this, we have been seeing like lot and lot of like new solutions coming out every then and now.
00:05:06And so the idea would be like, okay, we will try to build those foundations that will help us understand and evaluate like whatever ships next.
00:05:18So, yeah, to get started, like we will do a quick demo, like it's a short demo of like what are the different pain points around inference.
00:05:28And so this is a repo.
00:05:31I mean, you can pull it or you can also open it on the GitHub.
00:05:36It's called LLM inference at scale.
00:05:39A bit of background here, like four months back when I didn't knew anything on the LLM inference, I started learning it.
00:05:46I saw like a lot of resources were scattered.
00:05:49So we started putting it like all together in one place so that it could benefit people.
00:05:55Yeah, so let me actually get out of this slide show mode and probably go into this extended mode.
00:06:11Yeah, so in this repository, if you see a readme file, there is like a link to the slides.
00:06:27So this, it will be like this folder where you have like a PPTX and there is like a benchmark report in there.
00:06:34You can always like download it and then for the demo purposes, we have a couple of Jupyter notebooks.
00:06:43We have like collaborated with Molab who are the like Google collab alternative and what they basically provide you is like a free RTX 6000 GPU.
00:06:54So it's a 100 GB RAM GPU and we have like already set up these notebooks so that it becomes easy to like experiment with and like all of the assets and everything are preset for you.
00:07:09So we will start with like a simple demo A.
00:07:16Probably, probably, let me just see.
00:07:31Yeah, so when it comes to the inference, you need to do an inference on a certain model, right?
00:07:40So, for the workshop purposes, we are using a simple Mistral 7B model.
00:07:45It's a small model of around 15 GB in size.
00:07:49So we are going to like load that into the GPU.
00:07:53So, and we would look like some of the GPU stats as well.
00:07:58So we see like, okay, we are working on the 6000 Blackwell.
00:08:01So, and you might be thinking I'm not running the cells because I don't trust the Wi-Fi at conferences.
00:08:08So, yeah, so I would probably be just going over the results that we kind of ran previously.
00:08:18Yeah, so we have like a GPU which is like 102 GB.
00:08:22Now, the first thing that comes to my mind is like what's my memory consumption looks like when I do the LLM inference.
00:08:31So I load this model and I see like, okay, I have like a 15 GB here.
00:08:37So I have roughly like 87.5 GB.
00:08:40And now when I do the like inference here, what I notice is like the more the number of inputs I pass, more is the memory that I need.
00:08:52And it's increasing slowly, but it's still increasing.
00:08:56So imagine like if you have a context length of like around 4,000 or 16,000 or 32,000 tokens.
00:09:03So this memory could like really grow big and you could actually get like all of those out of memory issues.
00:09:12So definitely this is like your problem one, like your memory increasing with the increase in tokens.
00:09:18So in form of like a simple visualization, it looks like this.
00:09:24The second problem that you would see is like the time to your first token, it's very, very slow.
00:09:32We measure it by a metric called TTFT.
00:09:35It's a short form of it.
00:09:37And when you try to like measure the TTFT with the like input size, you would see like longer the context, you would see like this TTFT being slow.
00:09:52So now there are two problems.
00:09:54Your memory increases with the token size.
00:09:56Your TTFT increases with the token size.
00:09:59Sorry, not the token size, the context size.
00:10:07And then the third is the like throughput.
00:10:09The throughput is like how many tokens can you serve per second?
00:10:13And then how many users can you serve per second?
00:10:16So if you take a very, very vanilla implementation on your local system, it would be like very sequential.
00:10:24So if you send like five requests, all those five requests would be catered like sequentially rather than parallel.
00:10:31And so like your request basically takes more time to complete if you have like multiple users.
00:10:40So these are the like three problems.
00:10:43There is a fourth one.
00:10:44I haven't described it here.
00:10:45Probably we will build that intuition as we move forward.
00:10:48But let's remember like these are the three problems, the memory, TTFT and the throughput.
00:10:55Cool.
00:10:56I will go back to the slides.
00:11:02Okay.
00:11:03Perfect.
00:11:04So it should be this.
00:11:17Is it visible?
00:11:18Yeah.
00:11:19It's visible.
00:11:20It's visible.
00:11:21Yeah.
00:11:22It's visible.
00:11:23It's visible.
00:11:24It's visible.
00:11:25Yeah.
00:11:26It's visible.
00:11:27It's visible.
00:11:28Yeah.
00:11:29It's visible.
00:11:30Yeah.
00:11:31It's visible.
00:11:32Yeah.
00:11:33It's visible.
00:11:34Yeah.
00:11:35It's visible.
00:11:36It's visible.
00:11:37Yeah.
00:11:38It's visible.
00:11:39Yeah.
00:11:40It's visible.
00:11:41Yeah.
00:11:42It's visible.
00:11:43Yeah.
00:11:44It's visible.
00:11:45Yeah.
00:11:46Yeah.
00:11:47So within that repository, if you see a workshop folder, you see that README and then the README
00:11:55has all the links, the slides and the demos.
00:12:01Does that work?
00:12:06Okay.
00:12:07Perfect.
00:12:09Okay.
00:12:10Perfect.
00:12:16Okay.
00:12:17So let's start working through the foundations.
00:12:19Let's start understanding what are the reasons behind those pain points.
00:12:25And for that, like we have to look at this inference pipeline.
00:12:32So we get like an input text.
00:12:35That text could have like any number of words.
00:12:38You convert those into the tokens.
00:12:41So for simplicity, you can assume one word equal to one token.
00:12:46Then you kind of convert them into like the embeddings and then you send it to the like transformers.
00:12:53Like there are 32 layers of transformers, but that's specific to the Mistral 7.
00:12:58The different models have different kind, number of layers.
00:13:01And then you generate a new token.
00:13:04And that token basically goes back to the input.
00:13:06Then you generate another token and that keeps on going.
00:13:09Now in this entire pipeline, you would see like 95% of your compute is like taken by these transformer layers.
00:13:18So it's worth looking at like what goes within this transformer layer.
00:13:24Within this transformer layer, you would have like more layers.
00:13:28You have like a normalization layer.
00:13:30You have an attention layer.
00:13:32You have a feed forward layer and all.
00:13:35And attention layer is the one I think that has been very, very famous.
00:13:40Attention is all you need people.
00:13:42I think that's very well known.
00:13:44So attention is the most compute intensive layer.
00:13:48And we need to understand what goes within that attention layer.
00:13:53So what does attention do?
00:13:56So if you have an input text, it needs to find the attention scores of every token with respect to all of the previous tokens.
00:14:05And to do that, what it needs to do is like it needs to project every token into like a key query and the value space.
00:14:14So in like in a simpler terms, just understand this.
00:14:20Like if you have 10 tokens, then it needs like the 10 different query key and the value vectors.
00:14:26If there are 100 tokens, you would need 100 key and the value vectors.
00:14:31If there are 1000 tokens, you would need 1000 key value vectors.
00:14:35And so like your number of the key and the value vectors, they increase as you increase the input size.
00:14:45And if you calculate the like KV size per token, for a mistral 7b, it comes out to be 131 KV.
00:14:54This is because like you have two vectors, K and V.
00:14:59You have to multiply the size.
00:15:02One vector is like 128 dimensions.
00:15:04You have to multiply it by 32 transformer layers.
00:15:08And then you have to multiply it by the KV heads.
00:15:11For mistral 7b, it's KV heads.
00:15:15It's not like 32 because it uses a different kind of an attention mechanism, which we will talk about for sure.
00:15:23But yeah, so the KV size per token is like your 131 KV.
00:15:30Now imagine if you have 4K context, so that size becomes like half a GB.
00:15:37If you do like 16K context, that size becomes 2.1 GB.
00:15:42Now multiply it by the users.
00:15:45Like assume you can serve multiple users together.
00:15:49At the same time within that GPU, you could have like 42 GB with a 4K context and 80 users.
00:15:58And if your GPU is only like, let's say 24 GB, you are already running out of the memory.
00:16:04So you cannot serve that many users with that many contexts.
00:16:11To visualize this, look at a GPU memory.
00:16:14So the GPU memory has like a model weights, which are pretty fixed.
00:16:19These are pre-trained weights.
00:16:21There is like an overhead that is also fixed.
00:16:24That also, like that changes, but it does not change that much.
00:16:29Overall, you can assume it's fixed.
00:16:32And then there is like a leftover memory.
00:16:35So this leftover memory is what being used by your KV memory, like key and the value vectors.
00:16:41So, assume like you have one user.
00:16:46You can only serve that many key and the value vectors or that many tokens, which can like fit in this entire ATGB, like memory that is left.
00:17:00So we can show this with a simple demo too.
00:17:07Okay, great.
00:17:08Okay, great.
00:17:08Let me see if I can actually run this.
00:17:14Okay, great.
00:17:15Let me see if I can actually run this.
00:17:15Where the heck is this?
00:17:15Okay, great.
00:17:16Let me see if I can actually run this.
00:17:21Where the heck is this?
00:17:22Okay, great.
00:17:23Okay, great.
00:17:24Yeah.
00:17:25Yeah.
00:17:25So, you would see like let me see if I can actually run this.
00:17:31Let me see if I can actually run this.
00:17:33Let me see if I can run this.
00:17:34Let me see if I can run this.
00:17:35Let me see if I can run this.
00:17:36Let me see if I can run this.
00:17:37Okay, great.
00:17:38Yeah.
00:17:39So, you would see like the GPU is attached.
00:17:43Okay, great.
00:17:44Yeah.
00:17:45So, you would see like the GPU is attached.
00:17:56So, here we are just trying to confirm the like memory based on the maths and based on the intuition
00:18:13that we have built.
00:18:14So, the model memory is like let's say if you have a 7 billion parameters, you are doing
00:18:19a 16-bit precision.
00:18:20Your total memory comes out to be 14.6 GB.
00:18:23You can basically verify that with the maths.
00:18:27So, if you do all that maths, that comes out to be the 14.6 GB.
00:18:33Now comes the KV and the KV size.
00:18:37So, this KV size is like your 131 KV per token.
00:18:42And if you do that maths and you try to like visualize this.
00:18:47Oh, sure.
00:18:52Wait.
00:18:53Okay.
00:18:54And then let's just visualize this.
00:19:07Okay, great.
00:19:10Great.
00:19:11Yeah.
00:19:12So, this is the like memory chart.
00:19:15So, if you see like as your context increases, your memory keeps increasing.
00:19:21Then another thing to realize is like as your users increase, then also your memory increases.
00:19:28So, if you want to serve like 160 users on a GPU, you can support like, you can only support
00:19:37like a lesser context length.
00:19:40So, there is always a trade-off between what context length you can serve versus how much cost
00:19:47you can save by like putting your multiple users or the concurrent users into like a single GPU.
00:19:54So, you have to always take that trade-off.
00:19:57And we will go through that like in couple of more slides.
00:20:06Can you repeat, please?
00:20:10I'm sorry.
00:20:11I cannot hear you.
00:20:12Do you see a different pool with different context length so you can serve the light memory
00:20:25of the right?
00:20:26Yeah.
00:20:27Cool.
00:20:28Cool.
00:20:29Okay.
00:20:30Good.
00:20:31So, let me pull back.
00:20:32So, that was a light memory.
00:20:33We need to understand why we had like a slower time to first token when we increased the context
00:20:53length.
00:20:54So, for that like we need to understand the two phases of inference.
00:20:58And those phases are like the pre-fill and the decode phase.
00:21:01I think you would all, it seemed like a lot of articles, but we just wanted to explain it.
00:21:06So, when you send like a lot of, like when you send these input tokens, what you want to do is, you want to build those key and the value vectors that I mentioned for all the tokens.
00:21:20Then you want to compute the attention scores of every token with respect to the previous token.
00:21:27All this operation that you do, it's a very, very metrics heavy.
00:21:31It's a very, very compute heavy operation.
00:21:34And we all know like the GPUs, they are like very well suited for a heavy compute workload.
00:21:41So, we call like a pre-fill to be like a compute bound.
00:21:46And it does take some time to complete.
00:21:49So, whatever time that this phase takes to complete, that's your time to the first token.
00:21:55So, if you have like more input tokens, you have to generate more key value vectors.
00:22:02You have to do a lot more attention math.
00:22:05And because of that, your TTFT becomes more and more slower.
00:22:12Whereas if once you generate one token, you need to keep doing this to generate another tokens, sequentially one after another.
00:22:20But in that process, every time you have to build the key and the value vectors of all the previous tokens, which is same as pre-fill.
00:22:30Like you were building key value vectors there also, here also.
00:22:34But in decode phase, you are only computing the attention math for the new token.
00:22:40And that is why it's a very less, it's lesser compute oriented.
00:22:45And it's also called as memory bound.
00:22:48We will see it shortly why it's called as memory bound.
00:22:54So, in a classic timeline, you would see pre-fill and decode phase like this.
00:22:59So, time taken by pre-fill, that's your time to first token.
00:23:03And then your time taken by every decode step, that's your basically your inter-token latency.
00:23:12So, that's like the fourth metric that you need to worry about.
00:23:18Like what's the time being taken by your decode step?
00:23:21Okay.
00:23:22Okay.
00:23:23Cool.
00:23:24Now, why does the like decode step or why does decode takes time?
00:23:34And why it's being called as like a memory bound operation?
00:23:38Let's try to understand that.
00:23:40To understand that, we need to look at how the metrics map basically works on the GPU on a high level.
00:23:47So, GPU has two kinds of memories.
00:23:50You have a high bandwidth memory.
00:23:52You have a shared memory.
00:23:55So, the high bandwidth memory is a larger size, but a lower bandwidth.
00:24:01By lower bandwidth, I mean like you can transfer data out of it at a lower rate.
00:24:06Compared to the shared memory, so the shared memory is smaller in size, but it has a very, very high bandwidth.
00:24:13That means you can transfer data in and out of it at a very first thing.
00:24:19So, when you have to do a metrics map, so you have to pick the data in chunks from the high bandwidth memory.
00:24:27You have to put it into the shared memory.
00:24:30Do that math.
00:24:32Write back the result into the high bandwidth memory.
00:24:38For the pre-fill phase, when you have to do this, you have to do this metrics math only once.
00:24:46But for the decode phase, you have to do this metrics math again and again because you are generating each and every token sequentially.
00:24:59And so, it doesn't matter how fast is your decode because now you can transfer your data out of the high bandwidth memory into the shared memory at a certain speed.
00:25:14Because you are limited by the high bandwidth memory bandwidth speed.
00:25:18And so, that governs your token ceiling, like at what rate can you actually generate tokens out of the decode step.
00:25:32If you look at this in the roofline plot, so there is a left section which is called to be a memory bound.
00:25:42Mathematically, it's governed by the arithmetic intensity.
00:25:47Arithmetic intensity is the number of flip-flop operations that you perform per byte of data being transferred.
00:25:55So, for the decode step, since you are transferring a lot of data, like the key and the value vectors of all the previous tokens, the model weights,
00:26:08but you are doing less computation because you are computing attention math for only one token.
00:26:13So, it's arithmetic intensity is very low.
00:26:18But for a pre-filled phase, you are transferring the data once, but then like you are doing this heavy computation.
00:26:27And so, it's arithmetic intensity is very high.
00:26:30So, now you know like in terms of mathematics, like why the computer, like why the arithmetic intensity of pre-fill is very high compared to your decode.
00:26:44Okay.
00:26:46So, this is like another small, small demo.
00:26:51Okay.
00:26:52Every time I have to.
00:26:53Okay.
00:26:54Okay.
00:26:55Great.
00:26:56I hope this is already.
00:26:58So, yeah.
00:26:59Again, we are loading the model.
00:27:04Now, this is like the pre-filled cost.
00:27:09So, what we are basically doing is we are getting the like the input text and then we are trying to generate this, the pre-filled step, the amount of time it takes.
00:27:22We see like as we increase the like size of the input tokens, this pre-fill is increasing.
00:27:28So, you know, and this is the reason why your DTFT increases.
00:27:33And then like your decode time.
00:27:36So, the decode time is like on average, it stays about the same.
00:27:41And so, if it is like assuming like you ignore the like cold start, your decode time is like approximately around the average line.
00:27:50It is still impacted by like the input size.
00:27:57It's not like it's a constant time.
00:28:00And it is because it still needs to pull the key and the value vectors from the memory for all the previous tokens.
00:28:07So, there is still like that basically a small increase in time that you would see with the decode step.
00:28:15And then this is the like classic roof line plot.
00:28:20Okay.
00:28:22Okay.
00:28:23Presentation.
00:28:24Okay.
00:28:25Okay.
00:28:26Okay.
00:28:27So, now let's try to understand like the throughput dimension.
00:28:43You want to understand how many users you can actually serve.
00:28:48And I think we saw like a diagram of the GPU memory where we saw, okay, there is some memory that is free for the key and the value vectors to grow.
00:28:56Okay.
00:28:57So, assume like you have just a single user.
00:29:01What's the total KV size that you have that you can basically support?
00:29:09It's defined by your context limit.
00:29:11The max users that you can support is like whatever is your GPU availability, like whatever is the memory that is available in the GPU, you divide it by the key and the value size per user.
00:29:25And when you do that, like it comes out to be like the concurrent users.
00:29:32Now, assume like your GPU is fixed, your model is fixed, so your KV size per token is fixed.
00:29:46There are only two dimensions that are left here, which is context and your concurrent users.
00:29:53If you want to serve more concurrent users, you have to reduce the context length.
00:29:57If you reduce the context length, you could impact your quality.
00:30:02So, these are the two dimensions right now that we are trading off.
00:30:09Then if we, but can you actually serve the like max number of concurrent users?
00:30:17In an ideal world, probably not because every business has like a latent CSLO that we have to meet.
00:30:27So, if you remember like in the decode step I said, the time for the decode still increases if you have more inputs.
00:30:37It also increases if you have more users.
00:30:40So, ultimately your inter-token latency also gets impacted if you have like a higher batch size.
00:30:51And your TTFT also gets impacted.
00:30:53So, now there is a third dimension you have to worry about, which is like your latency.
00:30:58So, the three dimensions that you have is like a quality, latency, and the throughput.
00:31:04So, it comes out to be like this tradeoff triangle where you have to choose between the two.
00:31:11So, for a premium chat application, you would want to prioritize definitely the quality and you want to prioritize the latency.
00:31:21You would not want your users to wait for the, like, not infinitely, but probably for the larger latency.
00:31:30You can always sacrifice the number of users you can support on the GPU and probably take that cost rate, being more customer-obsessed.
00:31:39So, in the form of like, and like if you consider like an agent, sorry, the async agent workload, you would want to like prioritize definitely quality and the throughput.
00:31:53Because these are the long running tasks.
00:31:56And you would want to like serve as many as concurrent tasks as possible, but with a very, very higher quality.
00:32:06And often like, we think like, okay, if the GPU is like a very expensive GPU, that might not be a good fit for us.
00:32:19But it turns out that could actually serve you the lowest cost per million tokens.
00:32:27But you really have to trust your kind of calculations on the max users that you want and like, you really have to make those estimations correctly.
00:32:40So, we do have like, let me just, okay, great.
00:32:55So, for the capacity calculator, there is like a link to the collab because I was familiar with it.
00:33:01I had to migrate out of the whole widget library and I didn't have time.
00:33:16So, being lazy, I just picked collab there.
00:33:20Apologies to my lab.
00:33:23So, my V-RAM is connected.
00:33:34Okay.
00:33:45So, Wi-Fi probably.
00:33:46Okay, great.
00:34:02So, what we have done over here is we have like shaded some like the GPUs with their V-RAMs and bandwidths, the flip-flops and the cost per hours.
00:34:11Then we kind of like build this simple like capacity calculator.
00:34:19This is just a KV visualizer where you kind of like, when you increase the number of tokens, you see like a KV size, it increases.
00:34:29And when you increase the number of users, your size is like increasing at a much faster rate.
00:34:37And then, in this capacity calculator, let it run.
00:34:47So, we have like a model which is like a 7 billion parameter model that we selected.
00:34:56We set the like precision to be FP16.
00:35:01Now, we decide, the way we basically go by the GPU decision is you have to decide what's your, like you have to fix one dimension first, which you care about the most.
00:35:15So, for premium chat, I mentioned like latency is definitely the one.
00:35:21And then like for the async workloads, the batch, the minimum batch size that you want to solve for, from like a single GPU, that is the second dimension.
00:35:31So, you want to fix these first.
00:35:34So, I will go about like in a premium chat application.
00:35:39So, I can go ahead with like 10 milliseconds latency.
00:35:44A minimum batch size, I don't care.
00:35:46Like I can, so I am okay with like probably two.
00:35:53Okay.
00:35:54So, probably with the seven concurrent users on a single GPU.
00:35:59And like my context limit is very important to me because I want to focus on the quality as well.
00:36:07And so, like I do see like some of the GPUs.
00:36:10So, the H-108 GB, it's like a $8 per hour.
00:36:15But like, am I 300x?
00:36:19Is it?
00:36:20Yeah.
00:36:21So, it's like around $10 per hour.
00:36:24So, if you do all that throughput math that we shared in the mathematics before, you could find like your cost per million dollar tokens.
00:36:34That could be very, very, that could be like lesser.
00:36:37So, you need to do such calculations by fixing those dimensions and you need to decide your GPU to like reduce your kind of inference costs.
00:36:49So, this is at least the first step that you can take towards optimizing the inference.
00:36:56Okay.
00:37:01So, the next slide.
00:37:04Let me.
00:37:08Okay, great.
00:37:10And so, like now the next thing is about the model optimization.
00:37:15So, we are now basically have built that foundation where we understood some of the pain points, reason behind those pain points, why those were happening.
00:37:23How we could like address that GPU capacity thing.
00:37:29We need to understand what can we do, like what can we further do about it.
00:37:33So, it is about model optimization and I think I would like to invite Tanmay.
00:37:39He can talk more about these model optimizations provided he has work on this like during his research times.
00:37:47Okay.
00:37:48Okay.
00:37:49I can control.
00:37:50I can control.
00:37:54Yeah.
00:37:55Here.
00:37:56Okay.
00:37:57Hi everyone.
00:37:58Mic check.
00:37:59Am I audible at last?
00:38:00Yeah.
00:38:01Okay.
00:38:02So, hi.
00:38:03I am Tanmay Shah.
00:38:04I work as a senior quant modeler and also I am an AI researcher.
00:38:08So, focus is on agent verification and right now building world models.
00:38:13So, for this one, model optimization.
00:38:16Before we start model optimization, so I created a research template so that it will be easy for us to understand all these complex things.
00:38:25So, our template is simple.
00:38:28First, we will identify the problem.
00:38:30Second step, we will solve the problem using two algorithms.
00:38:34These are just fake algorithms.
00:38:35So, first algorithm is called ostrich algorithm.
00:38:39Whenever we see, just like ostrich, whenever we see a problem, ostrich put their head into the sand.
00:38:46So, same thing we will do.
00:38:47Whenever we face a problem, we will just ignore it.
00:38:51So, this is an important algorithm we should follow.
00:38:55Second one is created, it is called world cup algorithm.
00:38:59For example, we don't know who will win this FIFA world cup.
00:39:03So, what organizers did, they break the 48 teams into 12 groups.
00:39:11Then round 32, so round 32 right now is currently going on.
00:39:16Then round 16, then quarterfinals, then semifinals and finals.
00:39:21So, what they are doing is that they are breaking it into smaller problems and the useful results are moving forward.
00:39:30So, same analogy or same algorithm we will use to understand this model optimization, all those things.
00:39:38So, yeah, let's start.
00:39:41So, I have one H100 GPU.
00:39:46I have to use this open source model, what is called GPT OSS 120 billion parameter model.
00:39:54So, right now I think it's, so they have trained it on BF float 16 and weight is 240 gigabytes.
00:40:02What should I do?
00:40:04This is the problem we have.
00:40:07First thing, what we have to do is that 240 gigabytes and 80 gigabytes H100.
00:40:16So, and I have to fit only in one GPU, not in multiple GPU.
00:40:21So, what can we do?
00:40:22I think simple step is that just compress it.
00:40:27But how should we compress it?
00:40:29That's another challenge.
00:40:30So, if we compress BF float 16 to FP8, then it will be around 120 gigabytes.
00:40:38But our GPU H100 is still 80 gigabytes.
00:40:42So, what I think they did is that they compressed it into further MXFP4.
00:40:49And I think size is around 65 gigabytes.
00:40:53So, this is something we can do, compress, but question.
00:40:59So, and we will use over this ostrich algorithm.
00:41:03We are assuming that there is no loss in compressing a bigger model into a smaller size.
00:41:10So, second thing, in this one, okay, yeah.
00:41:16So, in this one, in this slide, we have used this Mistral 7B.
00:41:20So, seven billion parameters.
00:41:22So, it's a small model, seven billion parameters.
00:41:25So, if you multiply it by two bytes, so weight of it's around is 14.
00:41:3114.5 gigabytes, which can easily fit into H100 or even A40.
00:41:38So, next, what we can do is that like Mistral 7B, instead of compressing it of floating point 16, we can apply different techniques like int8 or int4 or nf4.
00:41:53So, basically, we have to just use ostrich algorithm and just believe that there is no quality loss kind of things.
00:42:01But, somehow, we also have to mathematically prove that by doing some kind of testing on some external benchmark that whether it is working or not.
00:42:10So, and this comes under post-training quantization kind of thing.
00:42:16One can also do this one during fine-tuning.
00:42:20One can also do this kind of quantization.
00:42:22This comes under a quant-aware training kind of thing.
00:42:25So, let's move to our next problem.
00:42:31So, we have this huge matrices.
00:42:37So, just imagine 1,000 by 1,000 dimension, matrix A, and another matrix 1,000 by 1,000.
00:42:51So, if you multiply by these two matrices, so number of operations will be 1,000 raised to the power Q.
00:43:00And this is kind of a problem in terms of computing.
00:43:06So, we wondered our matrix multiplication should be fast and it should save memory.
00:43:13So, what should we do?
00:43:15We have a giant matrix.
00:43:17Okay, let's take this one, Mr. 4096 by 4096.
00:43:23What should we do to solve our problem of speeding up the things and saving the memory 4096 by 4096.
00:43:33So, first thing is that we will use just our World Cup algorithm.
00:43:37We can decide a random number, just break the block vertically.
00:43:42It does not matter what you are choosing it.
00:43:45So, let's say we have 4096 columns.
00:43:51We will break it into a group of 128 columns each.
00:43:57So, 128, 128, 128, 128, 128 vertically.
00:44:03So, we will get this 32 blocks if we divide this 4096.
00:44:09Then, what will happen by doing this thing?
00:44:13So, if we just divide this one vertically, then we can use a multiple GPU to speed up the process.
00:44:21So, this kind of thing is called multi-head attention.
00:44:26So, what else can we do?
00:44:29We have a big matrix like, as I have mentioned that ostrich algorithm.
00:44:36So, our main problem is sizing.
00:44:39So, what we can do is that instead of having all those 32 vertical blocks, we will throw away 31 blocks.
00:44:49And we will assume that one block is sufficient enough that all the queries can handle those blocks.
00:44:57Our loss will be almost negligible.
00:45:00And we come up with this algorithm.
00:45:04And this algorithm is called multi-query attention.
00:45:08So, as we can see, right now we are at two spectrums.
00:45:12One is multi-head attention, where we split it into 32 blocks and use different GPUs or do some parallel processing.
00:45:22And at the same time, we are just throwing 31 blocks.
00:45:26And we are calling this as a multi-query attention.
00:45:31So, at both extremes, we should come up with a middle ground like something.
00:45:38We can say that instead of throwing all the 31, maybe we can group some of the blocks together.
00:45:49And we can assume that similar blocks will attend to a similar kind of queries.
00:45:58So, this kind of technique comes under grouped query attention, which is very popular right now.
00:46:04Even in Mistral or in other models, this grouped query attention works.
00:46:11So, right now, we have understand that we have a big matrix.
00:46:16We can divide it the way we want and doing some mathematical calculation,
00:46:19prove that the loss is almost negligible kind of thing.
00:46:23So, what else we can do?
00:46:25So, after that, after this grouped query attention,
00:46:34see, we have a big matrix.
00:46:38One is key and one is value.
00:46:42Let's compress that matrix into a latent vector.
00:46:47And then come up with some algorithm to reconstruct from latent vector to our original matrix.
00:46:55So, this kind of strategy comes under this one.
00:46:59Multi-head latent attention.
00:47:02But again, it has some problems with rope because rope is position-dependent and it is position-independent kind of thing.
00:47:10So, one needs to also include some index for keys also, so that one can map it.
00:47:16But again, the main problem is that why we are multiplying all those big matrices.
00:47:25So, because that's how this attention mechanism works, that each token will pay attention to every token.
00:47:33So, how about, let's, don't pay attention to all the previous token, only pay attention to the important tokens, which is important for us.
00:47:42So, this is kind of, this kind of field is evolving.
00:47:46So, this comes under sparse, deep seek sparse attention.
00:47:51So, yeah.
00:47:53And, yeah, so, okay.
00:47:56Next.
00:47:59Yeah, so next one is flash attention.
00:48:02So, in flash attention, so main problem is that,
00:48:09so currently, so currently, so currently, not currently, so right now, almost everyone uses flash attention, but way in 2022 or 2023.
00:48:20So, that's how it works.
00:48:23That's how it works is that, so, this Q, K, query and key matrices, they were in HBM.
00:48:34It loads, it, first, it loads into this one, our tensor core, and it do some, it do some calculation, and then it will write it back to HBM.
00:48:47And then, this process goes on multiple times.
00:48:51So, in flash attention, what they did is that, is that instead of multiplying the whole matrices,
00:48:59so they just divided it into, like, our World Cup algorithm, divided the bigger matrices into a small tile,
00:49:05and only put those small tiles into a SBM, so that it can process multiplication fast,
00:49:13and just keep, keeping track of this, some three variables, so that they can calculate this online softmax.
00:49:20So, yeah, so this is just mathematics, so if we have a multi-head attention, if it is for 524 kV,
00:49:34then it depends upon how much, how much grouping we want, and so, if instead of 32 kV head, we only want to use 8 kV heads,
00:49:47so, so, so, so we can get a compression of 4x times, and this multi-head latent attention.
00:49:54This formula depends on the model to model how many layers your model have.
00:50:00So, in the original DeepSeq paper, I think they have some 128 dimension, 128, one point, I don't remember the exact dimension, but according to that,
00:50:11they have used this one latent vector, in which they have used 512 as a dimension, and some 64 for rope index,
00:50:23and then they show that it is 56x more compressed than multi-head attention.
00:50:32Okay.
00:50:33Yeah, so, so yeah, so this is, so this is the, this is the trade-off, trade-off diagram, so here I think we have not talked about this linear attention or Mamba.
00:50:50So, main problem is just all this matrix multiplication, right now everyone is using attention.
00:50:56Suppose in future, if we don't want to use attention, rather than generating tokens sequentially, just use maybe diffusion models,
00:51:07where we can generate everything simultaneously, so all these algorithms will change also.
00:51:14But here, I think they have two more, one is linear attention and one is Mamba.
00:51:19So, according to this slide, so, if we are not compressing anything, so MHA is just, we are parallelizing the process.
00:51:29So, there is no quality loss, so it's a good, and then this group query attention, which is, I think, almost every model is using just a GQA and DSA kind of thing.
00:51:42Yeah.
00:51:43I think same thing we are providing in the attention mechanism scorecard, so, so I think this one, MHA quality is good, throughput is, throughput is okay.
00:51:56And for grouped query attention, it depends upon your use case also, though, yeah, though, though quality is almost similar to multi-head attention, but use case also matters a lot.
00:52:10Yeah, multi-query attention is just one extreme.
00:52:13We are, I don't know why, but we are just assuming that we only need one block, and all the queries will attend to that smaller, smaller blocks.
00:52:24So, quality is not that great for MQA.
00:52:29And this multi-head latent attention, so, yeah, if you have tried some, these are deep-seed models, so I think they are doing great job, yeah, in quality-wise, besides that, sliding window.
00:52:44So, all these are some techniques which, yeah, all these are some techniques, like, adjust the windows, all those things, and instead of, yeah, instead of multiplying everything, so linear attention is just saying that, summarize everything first, and then look up into it, and then Mamba, this is just a straight-space model, yeah.
00:53:09So, for the model, like, optimizations, we also have, like, two notebooks here.
00:53:28So, there will be, I have to go to this.
00:53:35Okay, so for the quantization, like the demo, this is, is this already run? No.
00:53:51Let me just run this.
00:53:54Okay, so we are loading the model, which is, like, a Mistral 7B.
00:54:10So, this one is, like, with the FP16 baseline.
00:54:20Wait.
00:54:21Did it run?
00:54:21Wait.
00:54:22Did it run?
00:54:23Okay.
00:54:24So, it's, uh, two milliseconds, it's run.
00:54:27Did this run?
00:54:28Okay.
00:54:29So, yeah, this time, it's fetching that model with the FP16 precision.
00:54:35Okay.
00:54:36So, it's, uh, two milliseconds, it's run.
00:54:40Did this run?
00:54:41Okay.
00:54:42So, yeah, this time, it's fetching that model with the FP16 precision.
00:54:47The Wi-Fi.
00:54:58It's gonna take time.
00:55:03Okay.
00:55:08Yeah, it, because it's downloading the weights from the hugging face.
00:55:14Huh?
00:55:17Yeah, so, Molab is, like, running online.
00:55:22Yeah.
00:55:23Because it needs to make the network call through the hugging face and, like, fetching.
00:55:30I don't know, like, but it's taking time to download, probably.
00:55:35Okay.
00:55:36Okay.
00:55:37Okay.
00:55:38Okay.
00:55:39Okay.
00:55:40So, here we see, like, the memory size is, like, 15 GB around, approximately, with the FP16 precision.
00:56:00We are trying to do the 2x compression, as Talmud talked about, with the int 8.
00:56:15Okay.
00:56:16So, we do see, like, your memory size is now, like, 0.7, 0.5 GB.
00:56:21What that means is now you have more memory for your KV to basically grow.
00:56:27That means you can either solve higher context limit or you can solve the higher concurrent users there.
00:56:36So, if you do the, like, in 4, you're basically, you are doing the 4x compression.
00:56:41So, that, with the 4x compression, it would be more lower.
00:56:45It would be, I think, around 3 to 4 GB.
00:56:50Yeah, 4.5 GB.
00:56:51And, yeah.
00:56:52So, this is, wait.
00:56:53So, this is just a basic plot of, like, so, these are the, like, theoretical numbers.
00:57:09We are not doing the, like, any throughput tests here.
00:57:12But, usually, you would see, like, your memory increases.
00:57:15So, you would also have, like, a bit of higher throughput.
00:57:21From some of the benchmarks that we studied, we saw, like, the int 8 compression.
00:57:26It does have, like, a lower throughput.
00:57:32Okay.
00:57:33And then, there is, like, a demo on the, like, the attention mechanisms.
00:57:42So, for the attention, okay.
00:57:45I have to run this.
00:57:58Okay.
00:57:59So, it has run.
00:58:02Oh.
00:58:03Wait.
00:58:04Why does it say no GPU detected?
00:58:09I should say the GPU should be detected.
00:58:18Oh.
00:58:19Okay.
00:58:29Wait.
00:58:30Wait.
00:58:30Wait.
00:58:50This is surprising.
00:58:57I guess it's not, like, able to detect the GPU for some reason.
00:59:05We do have, like, a GPU here.
00:59:11Okay.
00:59:12Never mind.
00:59:14Yeah.
00:59:14So, but the, like, basic idea here was more, like, as you try to move towards, like, compressing the computation, like, by using different attention
00:59:26mechanisms, like moving from the multi-head to the grouped query attention and then to the MLA.
00:59:33You would start seeing some optimizations.
00:59:38I think yesterday night we were doing some benchmarking.
00:59:43I wanted to correct this part.
00:59:45So, it wasn't, like, 56x.
00:59:48It was 14x.
00:59:50Basically, the demo had a mistake of, like, a computation where it did not multiply the number of layers.
01:00:02Yeah.
01:00:03So, apologies for that.
01:00:04So, this MLA is, like, a 14x savings worth in comparison to, like, your multi-head attention.
01:00:15So, now that we have understanding of the pain points, the foundations, the one side of the optimizations, which is the model optimizations, we want to talk about what can you do on the, like, the serving side.
01:00:31So, the first thing is we saw, like, when you perform, like, a simple decode step, you are pulling it, you are basically pulling the model weights and then you are recomputing the key and the value vectors for all the previous tokens.
01:00:50Even though you already computed those vectors for all the tokens.
01:00:55So, there is definitely, like, a lot of compute wastage.
01:01:02And if you kind of analyze the time complexity of it, it would come out to be O of N squared.
01:01:07And the way to resolve that is, like, a classic trade-off against the memory.
01:01:12You can maintain a memory of those vectors against the tokens and you can reference that memory.
01:01:19So, that memory was called as, like, KVCache.
01:01:24And, like, the flow looks something like this.
01:01:27And then, based on this KVCache, there were, like, four optimizations that were really possible.
01:01:35The first one is about the page detention.
01:01:42So, what's the difference, what's the problem today?
01:01:45So, when you send, like, multiple requests as the input to the GPU, these requests are in a batch.
01:01:52Every request is allocated, like, a continuous memory storage.
01:01:58Let's say of, I'm just taking an example, like, let's say, 2 KV.
01:02:05However, like, your request needed only, let's say, 1 KV.
01:02:11So, there is, like, a 50% of that memory fragmentation.
01:02:19And this fragmentation basically leads to the memory wastage.
01:02:24That means there was a space in the memory where you could have served more requests, but you could not because you were looking for that contiguous block of the memory.
01:02:36So, an inspiration tool was being taken from, like, how the OS works.
01:02:41Like, you maintain a logical memory and you basically have a physical memory.
01:02:47So, in the logical memory, it would still feel like that the KV vector for the, like, every token is, like, a contiguous.
01:02:59But it will be mapping to a different physical address.
01:03:06So, that really helped, like, saving a lot of memory.
01:03:11And it was only possible because they consider, like, memory as a set of blocks.
01:03:18And you would be dynamically allocating those blocks as the requests need.
01:03:22As the, like, new tokens comes in and they need that kind of memory.
01:03:27Another lever is, like, when you are sending multiple requests in the batch,
01:03:39GPU is, like, taking those requests.
01:03:42But it does not accept the new batch unless all the requests in that batch gets completed.
01:03:49So, the diagram looks more like a paid retention.
01:03:52But here it is more about, like, when is GPU available to take the next batch.
01:03:59So, there is a time period where GPU is, like, sitting really idle.
01:04:04And you want to, like, resolve for that.
01:04:08And for that, like, the idea was, like, okay, let's do that continuous batching.
01:04:17So, the continuous batching also really helped with, like, throughput because now you can ship more requests pretty quickly.
01:04:24Keep making sure, like, GPU always gets, it's always, like, occupied and it's not, like, sitting idle.
01:04:33So, you are saving on that compute.
01:04:36The third is the, like, prefix caching.
01:04:39So, you remember, like, the KV cache helped you save the computation for a single request across the tokens.
01:04:46But what if, like, you have the same tokens across multiple requests?
01:04:53How do you basically save against that?
01:04:55So, the prefix caching, which was introduced by the VLLM, exactly counters that.
01:05:04And then the third is, like, we talked about the fourth, actually.
01:05:10So, we talked about quantizing the model, but you could also, you can also, like, quantize the KV weights.
01:05:21So, that means now you need, like, a lesser space for your key and the value vectors.
01:05:28That means you can serve more key and the value vectors in the memory.
01:05:32And that means, like, you can serve more tokens.
01:05:34That means you can serve more context limit.
01:05:37And that means, like, you can serve more model quality.
01:05:41And all of this is, like, already present in the VLLM.
01:05:51You don't really need to reinvent that wheel.
01:05:56And you can, like, deploy this VLLM in production and you could see that, basically, growth.
01:06:03So, next, we have, like, a benchmark that we did.
01:06:08So, this benchmark was, let me see if I have that.
01:06:14Here.
01:06:17The demos.
01:06:22So, doing this benchmark takes, like, around one hour because you have to continuously stop and, like, restart the VLLM servers and you have to load the models and all.
01:06:35So, it does take a lot of time in doing the testing, but I can, like, really tell you here what we are doing.
01:06:41So, we have kept the model the same, like the MISTRL 7B.
01:06:46And then, we have, like, the set of input questions that we are sending.
01:06:53Consider them as the prompts.
01:06:56Then, we have a couple of helper functions here, like checking the server is up or not.
01:07:01This server is the VLLM server.
01:07:04Then, there are helper functions to get the VLLM metrics.
01:07:09And I will talk about, like, what those metrics are.
01:07:14Then, there are, like, a lot of the benchmarks and all.
01:07:17And then, you have to measure the KV usage and all.
01:07:22So, these are the, like, helper functions.
01:07:24So, the baseline is very simple.
01:07:26Like, we have a hugging face baseline.
01:07:29This is a raw, like, sending the text to the LLM, getting back the response.
01:07:36We see some results here.
01:07:38We saw, like, hugging face as a throughput of, like, around 51 tokens per second.
01:07:44Time to first token was, like, 54.
01:07:46And then, the inter-token latency was 19.
01:07:49This was all run on the H100.
01:07:56And then, we start, like, a very default VLLM server.
01:08:00So, by default, VLLM provides you the page retention, continuous batching, and the KV caching.
01:08:08So, three things are present by default.
01:08:13And then, when you try to compare those benchmarks, you see your throughput is, like, almost 15x.
01:08:21You are able to serve more tokens per second.
01:08:24Then, your time to the first token, that also rises.
01:08:34And then, the inter-token latency, it kind of goes down.
01:08:38And then, your KV versus users and the versus context, it increases for sure.
01:08:43Now, when you apply the prefix caching to it.
01:08:51So, with the prefix caching, you see, like, your throughput increases more.
01:08:57Your TTFT decreases.
01:08:59Your inter-token latency is approximately same.
01:09:02And then, your KV cache usage versus the users, it's kind of going down.
01:09:09The versus the context, it's not going down.
01:09:12It's approximately same.
01:09:14I think this is also approximately same.
01:09:16It's, like, not that big of a deal.
01:09:19When you apply the, like, KV quantization on top of it.
01:09:25So, it becomes, like, so, you see, like, the throughput is, like, almost similar.
01:09:33Your time to first token is similar.
01:09:36Your token latency is similar.
01:09:39But then, your KV usage actually goes down.
01:09:42This is because, like, you have quantized your key value space.
01:09:49And then, there is a concept of speculative decoding that Tanmay will talk about.
01:09:54So, when you try to benchmark those, so, you also see, like, there is, like, a bit of, like, the less KV usage there.
01:10:06Although, like, the results are approximately same.
01:10:08So, yeah, I mean, overall, like, these are the, like, the metrics across.
01:10:21Probably I should zoom out.
01:10:25Okay.
01:10:26It's not.
01:10:27Zoom out.
01:10:28It's not working.
01:10:29Great.
01:10:30So, yeah, this is the, like, VLLM benchmarks.
01:10:35It's your production default, by the way.
01:10:38We will also share that decision tree when we try to talk about, like, the other engines.
01:10:48So, yeah, so, we should talk about, like, what are some of the other inference optimizations we can do on top of it.
01:10:58And what were some of the other solutions that came out.
01:11:02So, I would like to, again, invite Tanmay.
01:11:07He is going to talk about, like, some of these optimizations.
01:11:11Oh, sorry.
01:11:12I'm so sorry.
01:11:13I didn't enable the slides.
01:11:26What was the?
01:11:27Okay.
01:11:28Great.
01:11:29Perfect.
01:11:30Which one?
01:11:31The speculative decoding.
01:11:32Yeah.
01:11:33Thank you, Harshal.
01:11:34Yeah.
01:11:35So, all these are, like, speculative decoding.
01:11:40All these are the, so, what we say, different flavors of same kind of soda.
01:11:46So, this technique comes under decoding accelerators.
01:11:51So, first one, so, we are only talking about this speculative decoding, but there are other variants, like self-speculative, Eagle, Medusa.
01:12:01I only like, I think, this one, Eagle algorithm.
01:12:05So, let's start with speculative decoding.
01:12:06Okay.
01:12:07Okay.
01:12:08So, let's start with what is speculative decoding.
01:12:09Main problem is that in transformer architecture, all these tokens are generated sequentially, one by one by one.
01:12:26How about, just use a smaller model and let a smaller model to generate maybe, let's say, four or five tokens.
01:12:37And this teacher model, or we can say, according to our World Cup algorithm, we can say referee.
01:12:43So, referee will decide how many tokens it accepts.
01:12:48And this loop keeps on going on.
01:12:51And our assumption is that there are certain domain where this kind of things will work.
01:12:59Like maybe in decode, maybe in coding, where almost there is no creativity.
01:13:05Each code or syntax is almost similar.
01:13:08So, maybe it can help it.
01:13:10But, based on personal testing, I didn't find this speculative decoding useful at all.
01:13:18But, other techniques like self-speculative decoding, where teacher model also have one head, auxiliary head, and it will do similar kind of things about this base model or small model is doing it.
01:13:35But, then this EGLE came, EGLE 1, 2, 3, I don't know how many versions are, but it is just saying that instead of creating, instead of generating tokens, let's train a small model and just take our features from one of its, one of main models' layers so that instead of generating tokens, it will generate this feature.
01:14:02So, EGLE is better compared to this other kind of technologies.
01:14:10And then another one is MEDUSA, which is just saying that, just generate all these tokens parallelly.
01:14:17Okay, so here, so here in this slide.
01:14:21Yeah.
01:14:22The next slide.
01:14:24Okay.
01:14:25Okay.
01:14:26Okay, yeah.
01:14:27Okay.
01:14:28Now we come to, now we will come to this one prefix caching.
01:14:34So, I don't know whether people are using this one static prefix caching or not.
01:14:39But, the thing is that, main problem with prefix caching is that, sometimes we type and make a small kind of mistake.
01:14:47And this standard static prefix caching is basically, it takes a prompt, do some hashing.
01:14:53And then next time when user asks similar kind of question, it will try to match the hash.
01:14:58So, if hash is equal, then it will, instead of recomputing all those k and b, it will just take it from the storage.
01:15:09But, you know that sometimes we make a mistake or maybe we can just change a word or letter, something like that.
01:15:15Then we have a very higher cache miss hit rate.
01:15:21So, that's why this one, Reddix tree.
01:15:25So, Reddix tree is becoming very popular and also because of agent.
01:15:30So, I think almost everyone is doing agent and most of the computation is going during test time, inference kind of thing.
01:15:38Where we keep on asking same kind of questions and prompt.
01:15:42For example, you are an expert software engineer multiplied by 200 times.
01:15:48This kind of loop keeps on going inside this agentic kind of things.
01:15:55Where it is necessary to keep or store similar kind of things in a Reddix tree.
01:16:03So, Reddix tree is just an advanced version of this prefix tree where we will just collapse a node if it does not have any branch.
01:16:17And for this kind of work where we keep on repeating same thing.
01:16:24This Reddix tree helps a lot and sglang use this kind of algorithm for prefix caching.
01:16:35Okay, yeah, then there is another thing.
01:16:38One is tensor, RT, LLM.
01:16:41This is very confusing.
01:16:42When I first started, I was just confused.
01:16:46What is tensor, RT, LLM?
01:16:49So, yeah, so, tensor, RT is just a standard SDK kind of thing.
01:16:55Tensor, RT, LLM is just an inference engine.
01:16:59Just like VLM, sglang.
01:17:01But problem is that it is related to NVIDIA.
01:17:05They optimized each and every layer and every problem.
01:17:10As I mentioned in our World Cup algorithm, they just break everything and optimized everything at hardware level also.
01:17:18So, yeah, okay, next.
01:17:23Yeah, so, for this workshop, we also did some benchmarking, like which is best.
01:17:33So, our setup was something similar.
01:17:36So, we did two kinds of testing.
01:17:39First one is without agentic testing, where we just...
01:17:44So, we use the shared GPT, this one data set, and just ask those questions using VLM and sglang.
01:17:57Okay.
01:18:05Yeah, okay.
01:18:07And let me just zoom it up.
01:18:12Okay, great.
01:18:13Okay, yeah, so, yeah, for this workshop, we used H100, and our first testing was that we just asked...
01:18:23We take questions from shared GPT and put it into VLM, sglang, and we found that actually there's no statistical difference between which one is better.
01:18:34So, both have almost similar kind...
01:18:37So, both are fulfilling similar kind of request per second, RTTFT and latency.
01:18:43So, but only difference we have seen during agentic branching.
01:18:50So, what we did was that we asked that similar kind of question that you are the best, this one, software engineer in the world.
01:18:59So, just solve the problem of traffic congestion in the city kind of thing.
01:19:04Then, we put this into LLM.
01:19:08LLM generates some output.
01:19:10Then we did another round two also.
01:19:13So, once this LLM generates this output, then in round two, we have especially mentioned that...
01:19:22provide, review the proposal and give ratings from one to ten.
01:19:28So, these are two turns we did, and this loop keeps on repeating it.
01:19:35What we found is that for this kind of workflow where everything is standard, all those prompts and context engineering comes into the picture.
01:19:47So, if we do proper this agentic branching, then I think this HGLang is three to four times better.
01:19:54But again, this depends upon the different setup maybe.
01:19:58If you do it, you may get different results.
01:20:02Okay.
01:20:03Yeah.
01:20:04So, I think...
01:20:06Did we upload it on GitHub?
01:20:08Yeah.
01:20:09Okay.
01:20:10Yeah.
01:20:11So, the PDF is like also in the drive.
01:20:15It's the same link as the slides.
01:20:18So, a quick summary here.
01:20:22So, on a standard API workload throughput, you would see like a VLLM and the SGLang would be the same.
01:20:31So, if you don't have...
01:20:33If you have like a standard workload, definitely go with VLLM.
01:20:36It's the production default anyways.
01:20:38But what Tanmay was also saying is when you try to like make it like agentic workloads, that is where like your SGLang really shines.
01:20:50And it kind of like provides you all the benefits.
01:20:55So, yeah.
01:20:58Keep like VLLM as a default.
01:21:00But if you have agentic workloads, probably try to move towards the SGLang.
01:21:05If you're not happy with the VLLM part.
01:21:09Okay.
01:21:10Let me...
01:21:15Wait.
01:21:18Okay.
01:21:21And then like there is like the...
01:21:29Like a comparison that is done at the 120 billion.
01:21:33Like for the GPT OSS 120 billion.
01:21:36This is a benchmark that was prepared by PlayPy.
01:21:42So, there is like a blog link here.
01:21:46Oh, nice.
01:21:48Okay.
01:21:49Yeah.
01:21:50So, they did the similar benchmark and they included like a TensorRT LLM in it.
01:21:57Definitely, you can always go through these benchmarks and try to understand which basically suits your use case.
01:22:05As we mentioned like TensorRT, they try to optimize the hardware side as well, having the peak hardware performance.
01:22:16And then like in terms of when you want to depict like your engines, once you figure out like between VLLM, SGLang, TencerRT, so that there are some new engines that are popping up.
01:22:29NVIDIA Dynamo for sure.
01:22:43So, they are also for the agentic session routing.
01:22:49Hugging faces are always there.
01:22:51It's a simple nose over.
01:22:53Then there is like an MSTAR engine that was recently proposed by Stanford.
01:23:00NVIDIA Dynamo for like multi-model.
01:23:05So, definitely you could explore those.
01:23:08And when you try to basically, just to like give a quick summary, we start with like a baseline.
01:23:15We try to find what model could fit our use cases.
01:23:22So, you could pick like a DeepSeq.
01:23:27You could pick like, don't pick like a Mistral 7B.
01:23:30I mean, it's not good.
01:23:32But, yeah.
01:23:35So, you pick your model and you want to like have a smaller memory and you want to try to fit that bigger model into smaller memory.
01:23:44So, that you could save cost on the GPU cost.
01:23:47So, you can do like all those quantization.
01:23:51Then you can apply all those serving optimizations by using the right serving engine under the hood.
01:23:58So, that can really provide you that throughput that you really want.
01:24:07And now, something that you can do after going back home probably because we cannot like actually go over all the material here is definitely reading about some of the source information like different attention mechanisms, different like these engines.
01:24:27Like try to just read the different benchmarks which are present online as well.
01:24:34And then, there are a lot of like in-depth guides or the next phases of it which is like learning about some KV eviction strategies.
01:24:45So, the world is moving towards having a separate KV cache engineering domain.
01:24:50So, you want to understand what's going on in there.
01:24:52So, KV eviction, cache compressions, hybrid memories.
01:24:57So, there are like a lot of solutions that are happening around there.
01:25:01So, always try to stick to those foundations or like the fundamentals or the first principles.
01:25:08And try to see like which solution basically solves what problem and whether you actually need that problem to be solved for your use case.
01:25:17And then, there is like distributed LLM inference which is like different pin point altogether.
01:25:26You would probably need like a two-hour workshop there as well to like go over like all the internals, do all the hands-on.
01:25:40Yeah, and this is something we are trying to propose for the AI engineer New York session which is to like dive deeper into the advanced sections of the LLM inference.
01:25:51So, this workshop was more for the like beginner and the intermediate level.
01:25:55So, in this form we do have like a feedback as well plus also the interest.
01:26:02If you think like we need certain improvements in certain sections, definitely give that feedback as well.
01:26:09And if you want to see this workshop in like New York Fair, I mean definitely feel free to enroll your interest.
01:26:22Huh?
01:26:24Oh, how is it possible?
01:26:27Boom.
01:26:32Let me just check.
01:26:37Okay.
01:26:38Huh?
01:26:39Yeah.
01:26:40URL works right?
01:26:41Yeah.
01:26:42Not the QR code?
01:26:43Okay.
01:26:44Probably I forgot to link those two together.
01:26:45Okay.
01:26:46Cool.
01:26:46Yeah.
01:26:47So, if you can give that.
01:26:49Okay.
01:26:50Cool.
01:26:51Yeah.
01:26:52So, if you can give that.
01:26:53Okay.
01:26:54Yeah.
01:26:55Okay.
01:26:56Cool.
01:26:57Yeah.
01:26:58So, if you can give that.
01:26:59Let me just.
01:27:00Okay.
01:27:00Okay.
01:27:01Cool.
01:27:02Yeah.
01:27:02So, if you can give that.
01:27:03Let me just.
01:27:04Okay.
01:27:05That will be fine.
01:27:06Um, and yeah, I think we would like to wrap this workshop then.
01:27:13And I'm sure like a lot of you would be having a lot of questions.
01:27:14So, we can take all those like offline.
01:27:15Uh, we can meet, uh, and we can like talk about those questions.
01:27:16Yeah.
01:27:17Sure.
01:27:18Sure.
01:27:19Uh, thank you everyone.
01:27:20Thanks for joining.
01:27:21Uh, I think it was really.
01:27:22Uh, thank you everyone.
01:27:23Uh, thank you everyone.
01:27:24Thanks for joining.
01:27:25Uh, thank you everyone.
01:27:26Thanks for joining.
01:27:27Uh, I think it was really.
01:27:28Oh, okay.
01:27:29Uh, okay.
01:27:30That will be fine.
01:27:31Uh, and yeah, I think we would like to wrap this workshop then.
01:27:33Uh, and yeah, I think we would like to wrap this workshop then.
01:27:36And I'm sure like a lot of you would be having a lot of questions.
01:27:39So, we can take all those like offline.
01:27:41Uh, we can meet, uh, and we can, uh, like talk about those questions.
01:27:42Yeah, sure.
01:27:43Uh, thank you everyone.
01:27:44Uh, I think it was really meaningful and all of you like came here.
01:27:49Uh, thanks a lot.
01:27:50Yeah, thanks.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video