Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Ashoq Chandasekar: Hi, everyone.
00:00:14Welcome to our talk on our LLM performance benchmarks
00:00:18reliable.
00:00:21A little bit about us, I'm Ashoq Chandasekar.
00:00:23I'm a staff software engineer at Google.
00:00:26I work on inference performance evaluation
00:00:28and optimization.
00:00:30And I lead a couple of open source projects.
00:00:32One is called inference perf, which is a benchmarking tool
00:00:35to do reliable performance benchmarks.
00:00:37And I'm also the sig lead for LLMD sig benchmarking.
00:00:41LLMD is a distributed inference framework
00:00:44that makes production scale inference possible.
00:00:47Hi, everyone.
00:00:48I'm Jason Kronberger.
00:00:49I'm a software engineer at Google.
00:00:52I'm also a co-maintainer of inference perf
00:00:55and a few of the sub projects that Ashoq brought up.
00:00:58And I work on inference performance and benchmarking.
00:01:03Okay, let's get started.
00:01:05Let's look a little bit about how the benchmark ecosystem looks
00:01:08like, right?
00:01:09You have your model server frameworks.
00:01:11These are VLLM, SGLang, and other model servers.
00:01:15And all of these have some benchmark capability within them,
00:01:18right?
00:01:18These are primarily Python scripts and our developer-focused benchmarks
00:01:21to see how you can measure the performance of your model server itself.
00:01:25And then you have your competitive analysis tools.
00:01:28These are MLPerf, semi-analysis, artificial analysis, and so on, right?
00:01:33They mainly aim for competitive performance benchmarks
00:01:36to compare, like, chip and accelerator performance.
00:01:39And then you have your typical web benchmarks.
00:01:42These are like Locus, Grafana K6, and so on.
00:01:46These mainly focus on high-scale HTTP benchmarks, right?
00:01:50And then you have your last segment,
00:01:52which is the production-scale LLM benchmarks, right?
00:01:55These are to actually benchmark your production inference serving stack.
00:01:59And that is our focus today, right?
00:02:03We'll be focusing mainly on inference perf
00:02:05and how we solve this production-scale benchmark problem.
00:02:11So if you have run a benchmark before,
00:02:13it typically looks like this, right?
00:02:15You have some sort of benchmark harness,
00:02:18and then you specify what model you are benchmarking,
00:02:21the number of prompts you want to run,
00:02:23what is the input-output sequence length,
00:02:25and the request rate of the load you want to send, right?
00:02:29And your output looks something like what is on the right.
00:02:32This is basically your input-token throughput,
00:02:35output-token throughput, some latency metrics,
00:02:38time-to-first token, time-per-output token, and so on.
00:02:42So what are some issues with a simple benchmark like this, right?
00:02:47So if you want to actually benchmark production-scale workloads,
00:02:51here I have LLMD inference stack as an example, right?
00:02:55You can have online serving, you can have batch workloads,
00:02:58and if you see the inference pool below,
00:03:00there are a lot of servers that are running, right?
00:03:04And then you have complex configurations like pre-fill decode disaggregation
00:03:09and workload auto-scaling and other things that are going on under the hood,
00:03:12and usually the scale is much larger, right?
00:03:15So your normal benchmark harnesses runs into issues
00:03:18when you try to benchmark a setup like this.
00:03:22And if you look at the key characteristics of what we want out of a production-scale benchmark,
00:03:26we need to be able to do high load, which is limited in a lot of tools out there.
00:03:31We need to be able to simulate real-world workloads, right?
00:03:34What use it is if it is just some synthetic workload that is not accurately representing
00:03:39what your customers are going to run.
00:03:41And then metrics fidelity is very important, right?
00:03:44Are the metrics accurate and how will they work?
00:03:49This is like the set of metrics that LLMD measures by default.
00:03:54I just pulled it from the website there.
00:03:57As you can see, it's not just like a single QPS that you're running, right?
00:04:01You are sweeping a list of various loads and you try to measure what the baseline is
00:04:08and what optimizations you are making and what the difference there is.
00:04:12You need to find the right point where the server gets saturated
00:04:16so you know the right optimal point to run your servers on
00:04:19to maximize performance and to save costs.
00:04:23And things like SLOs become more important, right?
00:04:26What is your time to first token P90 SLO and are you conforming to that SLO?
00:04:34So when you run like a normal benchmark like we saw before,
00:04:38what are some of the pitfalls that you run into, right?
00:04:41We have been running benchmarks for a couple of years,
00:04:44so we run into all sort of different results that people share
00:04:47and a lot of times we aren't able to reproduce the results
00:04:50that are shared by other people, right?
00:04:51So that is what motivated this talk.
00:04:55So these four common things that we see as an issue, right?
00:04:59One is accurate metrics and two, observability into your benchmark tool itself.
00:05:05Do you know if your benchmark harness is actually failing?
00:05:08Is it not able to maintain the load?
00:05:11And three, reproducibility.
00:05:12There is some inherent randomness in like the data sets that you use.
00:05:17So how do you make sure it is reproducible?
00:05:19And four, the data set quality itself.
00:05:24So this is an experiment we ran.
00:05:27We asked like a different benchmark harness to generate 200 QPS, and this was the result, right?
00:05:33So a couple of things I want to point out.
00:05:37Python has this global interpreter lock, GIL.
00:05:40If you have been working with Python, you know that, which makes everything sort of single-threaded.
00:05:44So even when you have like a multi-CPU mission, a lot of times you are limited by the performance
00:05:50of a single CPU, when you are CPU-bound especially, right?
00:05:54So this kind of shows the single-process benchmark harness and the multi-process harness,
00:05:59and how the QPS you are able to achieve differs based on it, right?
00:06:03When you run with a really small shared core mission, you can see that even when you request
00:06:08200 QPS, you are only getting 38 QPS.
00:06:11And then you give it a bigger mission, and then some of these single-process harness,
00:06:14they cap out at like 170 QPS, right?
00:06:17This is a much more powerful mission, but it is a problem because you ask for 200 QPS,
00:06:22and then you don't know whether it actually delivered it.
00:06:25It will just say, I ran it, these are the numbers.
00:06:27So you think, okay, you ran 200 QPS, but in fact, you have not.
00:06:33The other issue that comes out of it is the latency inflation, right?
00:06:36If your server is saying, okay, this is how much QPS I was able to run,
00:06:39and this was the accurate numbers, that is one thing.
00:06:42But if your benchmark harness is actually inflating latency, right?
00:06:46Because it's thrashing, trying to collect all the streaming token requests.
00:06:50In one of the tests, we noticed the delay was up to 58 seconds.
00:06:54So you might look at this and go, oh, my server is bottlenecked, right?
00:06:57It's not able to handle all the requests.
00:06:59But in fact, it's actually your benchmark client that is inflating the latency, right?
00:07:04We ran like a thousand QPS test.
00:07:06When your benchmark harness is actually able to scale out,
00:07:10you can see there is very minimal latency, right?
00:07:12There's a simulated server, so there shouldn't be any latency at all.
00:07:16And like I said, there are other variables that go into it, right?
00:07:23In one of the benchmarks, someone shared, and they said,
00:07:26hey, we are getting 20% better throughput.
00:07:28Then we looked into it, and we found out the benchmark harness
00:07:31was setting the model temperature to zero, right?
00:07:33Which means your model outputs are a lot more deterministic,
00:07:36and it was able to churn out a higher throughput
00:07:39than what you would normally see in like a real workload, right?
00:07:42Where your model temperature is somewhere around 0.7.
00:07:47Another thing is like we used a shared GPT data set, the same data set across two different benchmark harnesses,
00:07:52and they produce different input tokens, right?
00:07:55This is because they sample them differently, they truncate them differently.
00:07:59So as a user, you don't have insight into this, right?
00:08:02You run it, you trust the numbers it produces, but they are wildly different.
00:08:06And there is much more, right?
00:08:09Do you actually force it to generate until the end of sequence?
00:08:13Are you looking at prefix cache rates?
00:08:15How do you do multi-turn replay via benchmarks?
00:08:18And how do you actually get high fidelity on the actual workload
00:08:23that would resemble your production workload, right?
00:08:27So the main thing I wanted to convey here is like a lot of times
00:08:30you diagnose it as your server or inference stack problem,
00:08:34but in a lot of cases it could be your benchmark harness.
00:08:39So what is the solution to this?
00:08:40How do we actually do reproducible benchmarks?
00:08:43Jason here will take care of that.
00:08:45Cool.
00:08:46Thanks, Ashok.
00:08:48So yeah, how do you solve these problems?
00:08:53We pulled together inference perf.
00:08:57It's a CNCF project spanned out of Kubernetes working group serving
00:09:02to provide like a standardized place for us to work with the community
00:09:07and solve some of these issues together.
00:09:12It enables the ability to have like a user defined declarative configuration
00:09:19that allows you to have clear reproducibility across runs.
00:09:25We also added a load generator that solves the GIL problem in Python
00:09:33across multiple processes and reports those client metrics back
00:09:39along with server metrics to ensure that you have the highest metric fidelity
00:09:43and you're able to actually observe when your tool is having an issue versus your system under test.
00:09:52So first, going over the load generator, you see that the main process actually queues requests
00:10:02based off of the planned time that they need to execute, which is based off of your configuration.
00:10:08And this may be in some poison process or constant rate
00:10:13or maintaining a constant number of concurrent requests.
00:10:17This request queue channel is then spread across multiple processes,
00:10:22which pull and ensure that they execute with minimum overhead,
00:10:27but then also observability about when they execute versus their planned time.
00:10:34And you can see this working at scale.
00:10:36So this is a comparison across other tools,
00:10:40some being the HTTP scale tools like K6.
00:10:44But you see that even at 5,000 QPS,
00:10:48inference perf was able to keep up due to this architecture.
00:10:52And most importantly, actually report that it was able to keep up.
00:10:59Other portion is configuration.
00:11:01So earlier, I showed a brief example of how you might simply run a benchmarking tool.
00:11:09And here on the left, you can see a simple example running a random data set
00:11:13against an endpoint.
00:11:15But the actual configuration that we have in front of inference perf is very detailed
00:11:21with a lot of knobs that allow you to accurately test your configuration off of your workloads.
00:11:30You can see here on the right that this is a configuration for a conversation replay,
00:11:35where you're able to configure not only like the input output length, but their distributions, etc.
00:11:44And further beyond just the ability to configure a single run,
00:11:49we've actually worked together to have a published set of some of these workloads
00:11:55and configuration of inference perf that are tied to state-of-the-art inference workloads.
00:12:01For example, here in the workload catalog that we've put out,
00:12:05you're able to access standard multi-turn generation,
00:12:11tree-of-thought agentic generation, as well as batch summarization and others.
00:12:19Each one of these has a simple definition, kind of in natural language that allows you to understand
00:12:27what the scenario is.
00:12:29But beyond that, there's also pretty detailed configuration metrics,
00:12:34not only in inference perfs configuration, but in generic terms so that this can actually be shared
00:12:41across tools and have a place for standardization of these workloads.
00:12:50So, kind of the culmination of these things
00:12:54leads us to actual results that we can clearly display.
00:12:58And here's a screenshot from Prism, which is a UI we have for sharing not only those workloads
00:13:04that I showed before, but also some benchmarking results.
00:13:08And Prism is a part of the LL of D project.
00:13:13Here, you can see a benchmark result for the agentic code generation workload that we showed before
00:13:20on TPUs.
00:13:23These three lines you see here
00:13:26show you the difference between combined optimizations, as the green line, and a baseline
00:13:34that is just a simple Kubernetes service instead of in front of multiple model server replicas.
00:13:41It's important to note here is that this is at production scale with eight replicas.
00:13:48And you can see that the combined optimizations were measured to be much higher than the
00:13:53baseline scaling into almost hundreds of thousands of tokens per second.
00:14:02So, the takeaways, kind of the principles for benchmarking validity based off of the pitfalls
00:14:08that Ashok brought up earlier.
00:14:11At production scale, you need client concurrency and you need observability into your client's behavior
00:14:18and its ability to meet your configuration.
00:14:21The metric fidelity allows you to actually observe your client's behavior as well as your system under
00:14:28test and understand that your scenario was accurately executed and your performance results were valid.
00:14:35The stochastic variables and non-determinism or determinism that you set amongst your run
00:14:43needs to reflect your real-world demands for your workload.
00:14:49And most importantly, your datasets do truly matter.
00:14:54Your workloads need to be as close to what you are intending to test as possible.
00:15:00And we have examples in the workload catalog.
00:15:05So, here we have three links to some of the things that we've presented on here before.
00:15:11Inference Perf is our benchmarking tool and there's the Git repo for it.
00:15:17LLMD is a project that we work under and Inference Perf under it in LLMD benchmark.
00:15:23LLMD is for production scale imprints.
00:15:27And then LLMD prism, which was the UI we showed for the benchmarking results.
00:15:34And the workload catalog that defines some of these workloads.
00:15:41So, that will answer questions after but we appreciate your time.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video