Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
AAI Engineer
컴퓨터/소프트웨어AI/미래기술
스크립트
00:00:00Good morning everyone. Welcome to the first Influence Talk on the last day of AI Engineering
00:00:17World Fair. My name is Nishan Gupta and I'm joined today by my co-speaker Naman Ahuja.
00:00:23We work on building the efficiency, training, and influence infrastructure at Meta.
00:00:27Today, we're going to be talking about how do you operate distributed inference systems at scale.
00:00:32As we all know, inference is no longer just a research artifact boiled to a product.
00:00:37It's a foundational hyperscale infrastructure workload which is growing at a tremendous rate.
00:00:43The inference traffic already outpaces the largest microservices in the world
00:00:47and the rate of growth is fastest of any workload we have ever seen.
00:00:54Let's rewind back to around 2008 and try to compare the AI era with the cloud era.
00:01:00In around 2008, the cloud started as virtual machine offerings.
00:01:05The interesting engineering was virtualization.
00:01:07Then, over time, the value moved up to the stack, to the schedulers, ThinkBorg, Kubernetes, Mesos.
00:01:14Then to service meshes, then to autoscalers, and then to various platforms that were built on top of it.
00:01:18The orchestration layer is what actually captured the value and the complexity.
00:01:24AI is on the exact same trajectory, but just compressed into the last few years instead of a decade.
00:01:30We started with simple models running on GPUs.
00:01:33Then we saw the evolution of model-serving frameworks like VLLM, TOTSERV, Triton, and now we are watching the orchestration layer emerging in real-time, addressing the complex challenges of routing, KV cache management, pre-field decode desegregation, and multi-model multiplexing.
00:01:51In this next phase of AI era, it's not just about the best models or the kernels or the optimizations.
00:01:57It's about the whole ecosystem.
00:01:58It's about the control plane and the orchestration, and this is what we're going to be focusing in this race talk.
00:02:06So, let's talk a little bit about agentic demand explosion.
00:02:09In the classical web serving before the AI inference workloads kicked off, the capacity scaled roughly linearly with users, depending upon the workload type.
00:02:17Double the users, mostly double the QPS, double the infrastructure fleet, if there are no optimizations, and capacity planning was mostly a spreadsheet exercise.
00:02:28In this new agentic serving, capacity is scaling with number of users' time, number of calls per users, times number of tokens, which varies depending on the model, the optimizations, the hardware SKU you have.
00:02:40A chatbot can have one model call per turn, which is evolving now with 10 to 20 for co-pilots and 50 for research agents, and now it's thousands of such calls for autonomous workloads with no human in the loop.
00:02:54The key takeaway that you cannot plan capacity for the agents the same way we did for microservices.
00:03:00We need to think about elasticity and implement workload-aware scheduling and admission control.
00:03:08Now, let's try to dive a little bit deep into the differences, pros and cons, differences between the traditional microservice serving versus the modern inference serving across these key dimensions.
00:03:19The request shape.
00:03:20Microservices assume short, uniform request, wherever the LLM requests we are seeing in our workloads, they can vary from 50 tokens to 100,000 tokens, with vastly different compute profiles between pre-fill and decode stage.
00:03:33For batching, classical stacks for microservices was doing stacking, batching mostly at the load-balance layer, if at all.
00:03:40However, LLM serving requires continuous in-flight batching, otherwise the throughput collapses by an order of magnitude or more.
00:03:48State.
00:03:49Most of the classical microservices were stateless when we're not talking about storage layer.
00:03:54However, LLM serving does require a huge per-request state, the KVCache, which is very expensive to build and even more expensive to throw away.
00:04:02Scaling units.
00:04:03Scaling units.
00:04:05When we think about traditional microservices, we could run them in cheap CPUs, in pods.
00:04:11However, for modern inferencing, we require to run on GPUs, which are 100 times more expensive, which are 10 times slower to acquire, and we cannot over-provision them casually.
00:04:20Otherwise, it will lead to a huge wastage.
00:04:23Failure mode.
00:04:25When you think about traditional microservices like most of us have built over the last couple of years, we could, even if a host crashed or a pod crashed, we could restart it.
00:04:34We could rebuild the state if needed be.
00:04:36However, for the model inferencing, it takes a huge amount of time to go from cold to hot startup.
00:04:42And these, and if a GPU is mid-decode, it can drop thousands of in-flight tokens, and it can lead to a queue buildup.
00:04:50The takeaway is that the bottleneck is not just the model.
00:04:53It's the orchestration itself.
00:04:58Now, as we can see in these hidden decisions behind a prompt, when we go to an agentic application, it requires a bunch of steps, which are behind the scenes.
00:05:06We have to authenticate.
00:05:07We have to choose a model, depending on the request type.
00:05:09We have to select the region where it goes.
00:05:11We have to do admission control.
00:05:12We have to do caching lookup.
00:05:14We have to do the, we have to run it on the GPU.
00:05:17We have to do batching.
00:05:18And there's a bunch of other steps involved.
00:05:19And as we can see, out of all these steps, only one step requires the model, which is the pre-filled decode.
00:05:25If it's disaggregated inference, or just if it's not disaggregated inference.
00:05:29Other steps require infrastructure.
00:05:31The intelligence might lie in the model, but the economics, the reliability, and the user experience are all in the infrastructure.
00:05:38And this is why a lot of platform teams across a lot of companies are having much more impact on the product quality and the success, much more than before.
00:05:50Most of us in this room have a deep expertise across one or two or three layers.
00:05:54We might own kernels or kernel optimizations.
00:05:57We might own routing or the product itself.
00:05:59Or we might be operating the GPU infrastructure or the cluster itself.
00:06:03But very few of us have operated the whole stack or thought about it end to end.
00:06:08As you can see in these layers, these layers are not new.
00:06:11They have been around for 20 years or more.
00:06:13What is new is the combination and the coupling between them.
00:06:18A decision at the routing layer can change the cache hit rate at the model layer, which can change the batch composition, which can change the GPU utilization, which can change the auto-scaling decision because of the change in GPU utilization.
00:06:32So everything is entangled.
00:06:33When we have any regressions in our inference workloads, it's not just about understanding what happened at the caching layer or admission control.
00:06:41We need to think about the stack top to bottom.
00:06:43And when we see any bottleneck, it's very, very important to understand at which layer is that bottleneck so that we can invest properly.
00:06:54Now, diving a little bit deep into how a prompt works, when we have a prompt for any application, be it just if you want to generate an image or if you have a research task or if you have complex multi-agent orchestration, more or less, it involves a bunch of these steps.
00:07:08The prompt goes to the gateway.
00:07:10Then it goes to the router, after which it does the cache lookup if the request was already seen before.
00:07:17Then it goes to the schedulers, which decides on which GPU cluster it should run, on which hardware.
00:07:22It can be on NVIDIA or AMD or your in-house silicon chip, which then goes to the appropriate serving runtime, VLL, MSGLang, or whatever we are working on.
00:07:30And then we stream the response back to the user according to the SLO profiles of time-to-first token and time-between each token, and while making sure the throughput is what the user desired.
00:07:41Now, as we can see, this inference behaves like a distributed transaction.
00:07:45Each arrow in this diagram is a network hop.
00:07:47Every one of these hops can retry.
00:07:50It can time out.
00:07:51It can fall back.
00:07:53It can even fail.
00:07:54And each of these hops will have an SLOs, and it is streaming back to the user.
00:08:00So if it fails, the partial philisemantics are much harder to deal with than for a regular RPC call.
00:08:06Think about what happens if we have already streamed 200 tokens back to the user, and suddenly a GPU host is preempted due to a scheduled or a planned or an unplanned
00:08:15maintenance event.
00:08:16We cannot just retry.
00:08:17We have to think about it holistically.
00:08:19This is where, this is why reliability, we cannot build reliability at the edge.
00:08:23It has to be a property of the control plane, because the control plane is the one which sees the whole workflow.
00:08:31Now, let's talk about schedulers and some of the optimizations and how we can think about it.
00:08:37So for the traditional microservices, we used to think about pin packing of traditional microservices across three, four dimensions.
00:08:43Could be across the socialization, CPU memory, or could be across four domains, depending upon if you're using AWS, if you're using your own in-house cloud providers.
00:08:52But for inference, the scheduler needs to be aware of across at least seven access when we schedule a particular request.
00:08:59It has to be aware of the GPU type.
00:09:00There can be n number of heterogeneous hardwares in your cluster, H100 versus E100 versus B200, with different network topologies.
00:09:09It has to be aware of the HPM headroom, KV cache state.
00:09:13The model waits, whether they're already loaded, whether they're cold, or we have to, whether they've already warmed up, we have to cold start it.
00:09:19It has to be aware of the tenant priority.
00:09:21There can be n number of tenants running on that multi-tenant cluster with different SLO profiles.
00:09:27We have to also be aware of the workflow context.
00:09:30Are we in the third step of reasoning that has already spent X dollar, or are we at the initial stages and we can terminate the workflow if we are over-provisioned?
00:09:39We have to also think about latency budget, depending upon the type of agentic application we are building.
00:09:44So this brings us to the idea that we have to make sure that we implement agentic-aware scheduling.
00:09:49We have to make sure that we place a work which will finish in the fastest and the cheapest time, as opposed to just placing it on a random GPU.
00:09:58A concrete example might be that the scheduler needs to be aware that request R is at step three of a five-in-a-work flow,
00:10:05and it has already, in step one and two, has already spent X plus Y dollar.
00:10:09So if a step three fails, the whole workflow will be terminated, and we have wasted all that compute resources.
00:10:14That's why workflow-aware orchestration is very, very important,
00:10:18because it will change the admission decisions, the priority, and how we retry.
00:10:25Now let's talk about optimizations.
00:10:27I will not go too deep into a lot of optimizations.
00:10:29There's a lot of research which has already been done outside, but I would like to share a framework,
00:10:34which at least I like to use when it comes to it, and we can place it into four quadrants.
00:10:40First, can we avoid the work?
00:10:42Meaning, can we skip it entirely through caching, through techniques like prefix caching, response caching, semantic caching?
00:10:48The second, can we share the work?
00:10:51Can multiple requests share compute through batching?
00:10:54Think continuous batching, prefill decode, chunk prefill, specular decoding.
00:10:59Third, can we move away the work?
00:11:01Can we send it somewhere to a cheaper model or closer to the user?
00:11:05Through mostly routing, can we route it to a smaller model or a cheaper region or some other techniques?
00:11:12And lastly, can we delay the work?
00:11:13Can we wait for a better moment through admission control and queuing, which requires us to understand the priority classes of these requests and implement deadline-aware scheduling?
00:11:25Now this framework is very powerful because it transfers to various stacks.
00:11:29You might be using VLM or SG-Lang or TenserRT, but every technique fits into one of these quadrants.
00:11:35So whenever we think about any optimization to our model, we have to do a comparison and contrast with the previous techniques
00:11:41and see how all these stack with each other.
00:11:47Now whenever we think about scale, it's not just important to think about the performance of the model.
00:11:51We have to think about the cost, the economics as well.
00:11:54This is where it's important to understand what metric we are trying to optimize.
00:11:58Because the cost is not just the cost of the GPU or the model.
00:12:02It is the cost of all these parameters, retries, storage, failures, network, and of course the operational cost of development and all that stuff.
00:12:09What is important is to understand what is the key performance indicator for your product, which will add value to the users.
00:12:17So it's not important to optimize just cost per token or cost per request.
00:12:22We have to optimize cost per successful task because this is what actually users care about.
00:12:26And if you're able to optimize that, the cost for the overall product decreases and the users are much more happier.
00:12:36Now let's talk about reliability, a little bit about reliability and what it means to prevent cascading failures.
00:12:43Now the failure story is never a GPU is preempted or a GPU has died.
00:12:46The interesting story is the feedback loop that follows.
00:12:49So a GPU can degrade, the latency can rise, the client retries, the queue depth increases, the healthy GPUs will saturate, which will follow more retries, much more full regional failures.
00:13:02This is the classic cascading failures, but with a twist for a genetic application, the KV cache.
00:13:09We cannot just casually restart or reroute to a different cluster.
00:13:13A cold pool has to warm up before it can absorb traffic, during which the hot pool has to take on all those requests.
00:13:20This is why it's important to design the loop breakers very deliberately.
00:13:24The circuit breakers at the routing layer, the admission control, which is rather than just queuing, the load shedding tied to queue depth, not just CPU or memory utilization.
00:13:34And we have to also think about retry budgets because if we don't think about all these things, the cost can scale much, much more quickly.
00:13:43Now I'll hand it over to my co-speaker Naman to talk about the remaining talk.
00:13:50Thank you.
00:14:20I'll hand it over to my co-speaker Naman.
00:14:22I'll hand it over to my co-speaker Naman.
00:14:25I'll hand it over to my co-speaker Naman.
00:14:26I'll hand it over to my co-speaker Naman.
00:14:27I'll hand it over to my co-speaker Naman.
00:14:28I'll hand it over to my co-speaker Naman.
00:14:29I'll hand it over to my co-speaker Naman.
00:14:30I'll hand it over to my co-speaker Naman.
00:14:31I'll hand it over to my co-speaker Naman.
00:14:32I'll hand it over to my co-speaker Naman.
00:14:33I'll hand it over to my co-speaker Naman.
00:14:34I'll hand it over to my co-speaker Naman.
00:14:35I'll hand it over to my co-speaker Naman.
00:14:36I'll hand it over to my co-speaker Naman.
00:14:54OK, this works, I guess.
00:14:57Sorry for .
00:14:58So once inference reach a production scale,
00:15:00it starts looking much more like a distributed system.
00:15:03We are not longer just calling a model.
00:15:06It's more like a plastic distributed system problem.
00:15:09So in distributed system, we talk about queues, scheduling,
00:15:13auto-scaling, fault isolation.
00:15:14These are some of the dimensions.
00:15:16Inference has all these problems,
00:15:18but there are new constraints now.
00:15:20Instead of CPU memory alone, we have CPU, HVM, KV cache,
00:15:25and cost per successful task.
00:15:27So the operating question becomes,
00:15:28how do the platform know what to do next?
00:15:31And that's where observability comes into play.
00:15:34It's not just about dashboards.
00:15:35It's about how to provide input signal to the control loop.
00:15:39Telemetry, fields, analysis, analysis drive decisions,
00:15:43issues and changes of scheduling and routing,
00:15:45and finally, we just rinse and repeat.
00:15:47I'll just give an overview of what are some important metrics.
00:15:50First one is time to first token,
00:15:52which tells me about how much time does it really take
00:15:56to get the first response.
00:15:58Then we have a utilization ratio,
00:16:00which tells me whether memory or compute is the bottleneck.
00:16:04We have success per dollar that tells us
00:16:06whether the platform is actually delivering
00:16:08and working efficiently.
00:16:10And finally, we have end-to-end race latency
00:16:12that tells us what's the time being spent
00:16:15across the full request path.
00:16:19There's a core trader between latency, cost, and throughput.
00:16:22You cannot just get all of them.
00:16:24It's pretty analogous to cap theorem.
00:16:26If I increase the batch size,
00:16:28I improve the throughput and cost efficiency,
00:16:31but I may hurt tail latency.
00:16:33If I use speculative decoding,
00:16:35I may improve latency, but there are some extra compute.
00:16:38Ultimately, you know, we increase the cost for token.
00:16:41And finally, I can use a simple model, smaller model.
00:16:44I can reduce the latency and cost,
00:16:45but the response will be of low quality.
00:16:48Ultimately, I'll do failure analysis, do retries,
00:16:50which brings the cost back up.
00:16:52So every serving decision moves
00:16:54system somewhere in this triangle.
00:16:56And our job is to find the perfect setting.
00:16:58It's just an optimization problem now.
00:17:03This is where the industry is heading right now.
00:17:05Inference needs its own control plane.
00:17:07Everything we discussed-- routing, batching, caching,
00:17:10scheduling, reliability--
00:17:12they cannot be a separate knob now.
00:17:15They're converging into a logical layer.
00:17:17Let's call it an inference control plane.
00:17:19We used to manage VMs before in distributed system.
00:17:21We have auto-scaling schedulers.
00:17:23And we had Kubernetes, which turned these into control plane.
00:17:27Inference is going through the same transition now.
00:17:30Models are becoming resources.
00:17:32GPU, kvcache, token, latency, cost are now scheduled around.
00:17:36The control plane decides which models serve which request
00:17:40and how it is batched.
00:17:42So whether we build this layer internally,
00:17:44use open source, or from a vendor,
00:17:46the key design is to assume layer will exist.
00:17:50Now let's discuss about whether some of the operating lessons
00:17:53we have experienced in AI infra
00:17:55and how they are applicable here.
00:17:57The first lesson is that infrastructure bottlenecks
00:18:00usually show up before model bottlenecks.
00:18:02In production, many failures can come,
00:18:04but they can be just about scheduling and routing
00:18:07or capacity breakdowns.
00:18:08So these are not related to inference.
00:18:10It's about infrastructure problems.
00:18:12Then we have elasticity.
00:18:14We need elasticity in the system.
00:18:15We can have more GPUs, but this will not really solve the problem.
00:18:18We are just hiding the problem.
00:18:20Then our solution to schedule the decision,
00:18:24overpowered efficiency.
00:18:26The same fleet can deliver very optimally,
00:18:28depending on how you're scheduling it
00:18:30or how you're badging it.
00:18:31Then we have control loops, which beats the manual process.
00:18:35The platform has to sense, detect,
00:18:37and automatically adapt to the system.
00:18:39So main takeaways do not optimize for tokens.
00:18:43Optimize for successful tasks.
00:18:48So this is a broader shift I want to leave you with.
00:18:51The first phase of AI infrastructure was about better models.
00:18:54We invested a lot of time in improving our models,
00:18:56smarting, making the models smarter,
00:18:58and coming with better benchmarks.
00:19:00The current phase right now is faster inference,
00:19:03lower latency, better batching,
00:19:05better batching, and better GPU utilization.
00:19:08But the next phase is about orchestration.
00:19:10That means GPU, memory, cache, and everything,
00:19:13these are just resources,
00:19:14and they need to be scheduled and controlled.
00:19:17The teams that understand this early on
00:19:19will build infrastructure for the future.
00:19:21So the closing idea is,
00:19:23infrastructure is no longer a survey problem.
00:19:25It's an orchestration problem.
00:19:27Thank you.
00:19:29Thank you.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기