Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

EnglishالعربيةDeutschEspañolFrançaisहिन्दी日本語한국어PortuguêsРусский中文
AAI Engineer
컴퓨터/소프트웨어AI/미래기술

스크립트

00:00:00Good afternoon, everybody. I'm Sitan Shu from Corvi. I'm going to be talking about vertical
00:00:19mobility. It's quite a fancy topic, the title that we came up with, but basically going to
00:00:26be talking about the inference platform that we have at Corvi that we are building to serve
00:00:30small to big models and various different types of workloads. A quick intro about me, I joined
00:00:38Corvi just about four months back, leading all of inference over there. And before this,
00:00:44I was managing everything at AWS Annapurna Labs for training, and before that, inference
00:00:49and training at Sambanova. So quite a bit of experience in this particular space. What
00:00:56I will, the way I'll be taking you through is explaining to you the consumption models
00:01:00that we have, and from that, how we have derived what the platform should look like so that
00:01:04we do not need to keep changing the platform and we keep making enhancements in the platform
00:01:09that we have for serving inference, and how and why performance plays such an important role
00:01:14over there. I think a little bit of this might be common with the previous topic that was discussed
00:01:20over here. Consumption models. So we have at large two biggest consumption models. One is a serverless,
00:01:29which is where the customers can come in, consumers can come in, do not need to worry about managing the
00:01:35hardware themselves, do not need to worry about managing the clusters, orchestration, anything at all.
00:01:40There's an API, there's UI, you come in, you pay per token, and you get your model served.
00:01:47Biggest thing over here is that the type of models that we serve in the catalog, that is the breadth of
00:01:54models that the customer will be able to go through. I'll talk about dedicated and then I'll come back to serverless,
00:02:01because there is one thing unique on the serverless side. Dedicated inference service that we provide
00:02:05is more for customers who want to know exactly what hardware they are going to be using and running on,
00:02:11but the model deployment also depends on them. So they use our service, they use our orchestration layers,
00:02:18but the model deployment depends on them. Model performance also depends on them, as long as we provide
00:02:25in the platform the capability and the knobs to serve those features. Coming back to serverless,
00:02:30one of the interesting pieces over here is typically serverless models are noisy neighbor problems,
00:02:38where if let's say everyone is banging on the exact same model, then you might be timing out quite a
00:02:43bit, depending on how much capacity I have behind it. So another feature that we have on the serverless
00:02:48side is what we are calling provisioned throughput. So as a customer, if you know your traffic profile,
00:02:55and if you can let us know about that, we can carve it out specifically for you,
00:03:00behind the scenes. You still do not need to worry about what hardware it is exactly running on,
00:03:04as long as your throughput, your SLAs are maintained.
00:03:07So that is another one on the serverless side, and that is still charged by the token,
00:03:12but you know that you're not running into the noisy neighbor problem over there.
00:03:19Let me take a quick stab at a few different types of workloads, workload shapes that we have,
00:03:27that we are seeing, and the ratio between these is like continuously changing, though agentic is
00:03:32like really high up there. Agentic and chat, kind of very similar, super high on the input sequence
00:03:39lengths, very low on the output sequence lengths typically. But the biggest difference between agentic and
00:03:43chat being the fact that the multi-turns in agentic are super low latency versus in chats, because when
00:03:50you get the response of the user, you have to read the answer and then you respond to it. So there are
00:03:56there are differences over there. And that big difference ultimately converts into something
00:04:01related to the KVCache management. But these two are both real-time and another real-time workload is
00:04:09your voice and videos, which are steady streaming and super latency sensitive. On the agentic and chat
00:04:16side, largely the requirements are from throughput point of view, not so much from latency, but real-time
00:04:23voice and videos are absolutely latency sensitive. Coming to batch, batch is where the SLAs are like super
00:04:32loose. They run into like seconds and minutes, sometimes for some customers actually even in hours.
00:04:37They're like, I'll just throw, give me 10 to 12 hours of workload capability and I'll throw whatever I can,
00:04:44process it whenever you can. These batch workloads, the way they come into the picture over here in
00:04:52deciding, sorry, being the requirement for some of the design choices that we make in the stack. Imagine
00:04:59these four different types of workload shapes. In the time dimension, you have to play the game
00:05:04of characteristics on how you can fit it in to utilize the underlying infrastructure the most.
00:05:12I'll give a high level on how our stack is shaped right now. And I'll walk you through a bit of a
00:05:19request flow over here. So for both serverless and dedicated, if you look at the right hand side of
00:05:24the screen, you'll see that on the platform side, you'll go to the control plane to have your
00:05:31authorizations, your rate limitings and your usage being tracked, etc. So that you can be built accordingly.
00:05:37And observability so that we can make sure that we are not violating the SLAs that have been signed,
00:05:45right? Underlying on the platform, I've shown at our super high level that we have these different
00:05:52inference engines, VLLMs, SGLangs, and tensed RTLL, but there's quite a bit of detail over here that I'll touch up on.
00:06:00And underlying that, what I'm trying to show over here in green is various different pieces of hardware.
00:06:09So it's not that, so the platform needs to be capable enough of sharing, of having the workload
00:06:15getting distributed across various different generations of these GPUs, specifically NVIDIA GPUs
00:06:21that we use, right? So let's take a few examples over here. Let's say the request originates from the
00:06:31client side through apps or notebooks, any of those, or through the agents, right? It hits the gateway.
00:06:36Once it hits the gateway, then, like I mentioned on the control plane, goes through authentication,
00:06:42etc., etc., etc., but then comes either the serverless or dedicated. So in the case of serverless,
00:06:48it'll be paper tokens, so the token usage would be monitored over here, not the exact tokens,
00:06:53but just the token usage, because we maintain ZDR, zero data retention policies.
00:06:59It is, depending on the multi-tenancy or the provision, if it is provisioned, then we know
00:07:05underlying for the router, it needs to go in and target the explicit deployments for the provision
00:07:12throughput customers. For the multi-tenant customers, there are separate deployments.
00:07:17Router over here, specifically, the router is very important, since the router is responsible for
00:07:26making kvcache-aware routing choices. Why is it important? Because, like I mentioned when we were
00:07:31discussing the workload profiles, the agentic use cases are typically super heavy on the input sequence
00:07:39lengths, and bulk of the input sequence length, about 80 to 90 percent, depending on which company it is,
00:07:44depending on the customers, 80 to 90 percent of it is the same for various different requests.
00:07:51So there is no point in going in and recomputing the pre-fill or redoing the pre-fill for that.
00:07:57Pre-fill is super compute bound, very expensive, that's why as much as you can hit the cash,
00:08:04more you can save, which is why if you look at the token pricing anywhere, there's a specific price for
00:08:11input tokens and there's a way cheaper price for the cash input tokens. So caching becomes really important
00:08:18over here. Underlying how you want to split the hardware is totally dependent on the choice in the
00:08:26platform and we provide the capability to do either. Either do a pre-fill decode disaggregation if the
00:08:31use case desires it or do not do it because pre-fill decode disaggregation is not cheap for every type of use case.
00:08:41Let's take another request flow over here. Let's see if when it was a dedicated customer, then what will happen.
00:08:48A dedicated customer again will go through the gateways that have been set up for them with proper isolations.
00:08:55Billing is not based on tokens, billing is based on usage of per GPU per hour.
00:09:03It's a private gateway so that there is no noisy neighbor problem, no one else can get in.
00:09:09Same router logic over here so that if there are requests which are very similar, then it hits the cache most.
00:09:18And depending on the deployment that the customer makes in there on their dedicated GPUs,
00:09:24they can decide if they want to do pre-fill decode disaggregation or not. They can decide which
00:09:29which engine to use, VLM or SG-Lang or Tensile RTLM. And given the bulk of capacity that the customer
00:09:35has reserved, they can decide if they want to have just one deployment with the ability to scale
00:09:42through the whole cluster or they want to have multiple different models, multiple different deployments.
00:09:47One thing that I do want to mention about the router over here is the fact that
00:09:54heterogeneous capacity across different zones and regions
00:09:59is supported. It is a little, it's quite a bit of a hard problem to load balance across that.
00:10:04So the priority order that we typically take is first KV cache locality and then the least loaded fallback.
00:10:16That's that. Another request flow that I want to go over here which
00:10:21might be a little hard to see from the diagram is I want to take the batch workflow.
00:10:25For the batch workflow what we would actually do is the underlying capacity that the customer has,
00:10:31let's say the same dedicated inference customer, during US daytime they're running their real-time
00:10:37workloads and from evening to night they want to run batch workloads, the same capacity after time can
00:10:43be scheduled to run the batch workloads. So we provide the capability in the API to tell when to scale up and
00:10:51when to scale down and as per schedule, if we can, if they tell us that we can have to scale down,
00:10:56we will scale down and open it up for batch processing through the night.
00:11:05I think I've spoken quite a bit about optimizations on the KV cache side but I do want to repeat a little
00:11:10bit because this is one of the most interesting pieces. If we can hit on the cache more you can,
00:11:19you will avoid the cost of pre-fill which is the most expensive piece over here.
00:11:26Reusing the KV cache across multiple different turns in your agentic workloads, between turns also there is
00:11:31a lot of similar pre-fill that comes in the input sequence length.
00:11:38Think about the chat workloads which is where offloading KV cache also becomes extremely important
00:11:43because with the chat workloads we have a lot of latency between different, between multiple turns
00:11:49that we as users put in. But if we completely evict whatever we had in our particular conversation,
00:11:57then the next time we ask a question in the same chat it's going to take a little bit longer.
00:12:02So instead of actually completely evicting and redoing the pre-fill again, what the the techniques
00:12:08being used are maybe using, we are using our own but externally we know about LM cache and Mooncakes.
00:12:17What we do is we will offload the KV cache to a high bandwidth storage
00:12:21so that we can store a lot of these pre-fills such that whenever the accompanying request comes
00:12:28for that particular conversation, it can be loaded in right away into the HPM.
00:12:37On the performance liver, I would, I just want to mention a few performance livers that we've
00:12:41discussed the PD DSAG that is one but quantization and speculative decoding are others and how to
00:12:48carefully choose the parallelization degrees and the strategies that is actually very important.
00:12:54Two of the biggest livers that we have been working with are quantization to NVFE4 and spec deck.
00:13:01We do provide capability where if the customer has their data set and they want us to train speculators
00:13:07for their data sets for better acceptance lens, which will ultimately make the output throughput
00:13:12significantly higher. We do have that as well. So, but that happens async. We, we get the data async,
00:13:20we train the speculators async and then we deploy the speculators into the customer deployments
00:13:26if that's what they wanted. You see three screenshots over here. I've posted them from
00:13:32the last one month, one month's worth of work that some of us in my team have done. You can see we came
00:13:39quickly on top of the leaderboard on Kimi 2.6, 2.7 and those are, those are from artificial analysis
00:13:46and going back to the session before this, can we trust that? That's why for GLM, I have the results
00:13:52from open routers. So artificial analysis, when they run benchmarks, they're running very specific workloads.
00:13:58Open router is actual user traffic and you can see on the open router side, weights and biases. So the
00:14:05branding is different, but weights and biases are basically curvy. We bought weights and biases
00:14:09about a year back. You can see the speed over here that we have from our deployment is pretty close
00:14:16to what Fireworks is providing as Fireworks fast, right? But underlying techniques that we are using is
00:14:21what I want to emphasize the most over here for performance optimization. That becomes critical because
00:14:27ultimately what you want to serve to the customer, what we want to serve to the customer is price performance benefit.
00:14:36Quick recap, single platform is what I've been trying to emphasize is what I'm trying to show.
00:14:44Two different consumption models, serverless and dedicated for customers and within serverless,
00:14:49I describe two different consumption models as well, pay as you go and provision throughput if you care about
00:14:53that and ultimately compounding the gains through performance optimizations in the stack.
00:15:01That's all. Thank you, folks.

핵심 요약

A unified inference architecture combining KV-cache-aware routing, hardware offloading, and quantization scales efficiently across serverless, provisioned, and dedicated GPU workloads.

하이라이트

  • Serverless inference offers both pay-per-token models and provisioned throughput options to eliminate noisy neighbor latency spikes.

  • Input sequence pre-fill processing is compute-bound, making KV cache locality routing critical since 80% to 90% of agentic prompt tokens are identical across turns.

  • Offloading KV cache to high-bandwidth storage prevents conversational latency delays without needing full pre-fill recomputation on every turn.

  • Dedicated inference gateways isolate traffic and charge per GPU per hour, enabling users to schedule off-peak batch processing on the same hardware during low-use windows.

  • Quantization to NVFP4 and asynchronous custom speculator training significantly boost output token throughput across large language models.

타임라인

Inference Consumption Models and Noisy Neighbor Mitigation

  • Serverless inference abstracts hardware management and charges strictly per token.
  • Dedicated inference services grant direct hardware transparency while charging per GPU per hour.
  • Provisioned throughput carves out dedicated capacity within a serverless billing structure to bypass noisy neighbor issues.

Inference platforms serve two core delivery profiles: serverless pay-as-you-go APIs and dedicated hardware allocations. Serverless setups abstract cluster orchestration and charge per token, but heavy shared usage risks high latency. Dedicated instances guarantee hardware access and billing per GPU hour, leaving model deployment settings up to the user. Provisioned throughput blends these paradigms by reserving capacity behind the scenes to meet latency SLAs while retaining per-token pricing.

Workload Shapes and KVCache Routing Strategy

  • Agentic and chat interactions feature long input sequence lengths relative to output token lengths.
  • Pre-fill operations are compute-bound and represent the highest cost driver during token generation.
  • Routers prioritize KV cache locality over pure load balance metrics across heterogeneous regions.

Agentic and chat workloads feature asymmetrical sequence ratios where 80% to 90% of prompt tokens overlap between iterations. Because initial token pre-fill is compute-heavy, repeating it inflates operational costs. The routing layer checks for existing KV cache locality before falling back to the least-loaded GPU, ensuring optimal cache reuse. Platform features allow users to enable or disable pre-fill and decode disaggregation depending on specific latency requirements.

Hardware Reallocation and KV Cache Offloading

  • Dedicated instances can shift capacity from real-time serving during daytime to batch workloads overnight.
  • Offloading KV cache to high-bandwidth storage maintains conversational memory without occupying high-bandwidth GPU memory continuously.
  • Pre-fill avoidance significantly lowers token serving costs.

Time-dimension scheduling allows dedicated GPU capacity to scale up during peak hours for real-time traffic and scale down at night to execute low-priority batch workloads. For chat applications with long pauses between turns, keeping prompt states inside HBM wastes memory. Storing idle KV cache data in high-bandwidth external storage and re-hydrating it upon the next user request eliminates repeated pre-fills while keeping GPU memory clear.

Performance Optimization Levers and Execution Speed

  • Quantization to NVFP4 speeds up output throughput without sacrificing performance quality.
  • Asynchronous speculator training yields higher speculative decoding acceptance rates for custom customer datasets.
  • Stack optimizations compound speed benefits to deliver maximum price-performance gains.

Optimizing output generation relies heavily on NVFP4 quantization, fine-tuned parallelization strategies, and speculative decoding. Asynchronous pipelines can ingest customer datasets to train tailored speculators, increasing overall token acceptance rates during deployment. Benchmark results demonstrate that integrating these stack-level performance enhancements achieves serving speeds competitive with specialized fast-inference providers.

커뮤니티 글

아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!

이 영상에 대해 글쓰기