Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia한국어PortuguêsРусский中文
AAI Engineer
Internet TechnologyComputing/Software

Transcript

00:00:00- Hi, everyone, thanks for joining our talk.
00:00:14I'm Lu, and this is my colleague, Chen Ru.
00:00:17So, today we're gonna talk about the,
00:00:20we both work on the inference team at OpenAI,
00:00:23and today we are going to talk about
00:00:24routing IIM inference in production.
00:00:27Specifically, how our system evolved
00:00:29from routing based on feedback loops
00:00:32driven by engine signals to a more explicit
00:00:35and a predictable policy,
00:00:37which is still informed by engine signals.
00:00:39However, it's more like the way we use it is different.
00:00:43So, for the agenda today,
00:00:44we're gonna begin by introducing
00:00:46the inference load balancer,
00:00:48what it is, what it does, and how it has evolved.
00:00:51And then, Chen Ru will walk us through
00:00:53the newer control plane and the data plane
00:00:56driven architecture, what are the responsibilities of each
00:00:59and followed by a concrete case study
00:01:01of how we reduce the global network overhead.
00:01:05And in the end, I return to discuss
00:01:07the protection mechanisms that help keep the system
00:01:10stable under production level stress.
00:01:13So, to begin with, what is the inference load balancer?
00:01:17And where it sits.
00:01:20So, this is a very high-level diagram of the system we are talking about.
00:01:25On the left-hand side are the front-end clusters.
00:01:29Those are the GPU clusters, sorry, those are the CPU clusters
00:01:33that act as gateways into our system.
00:01:36And they receive user requests, then prepare them into the inference request
00:01:42that can be processed by the inference engines.
00:01:44And on the right-hand side are the engine clusters,
00:01:47which are usually GPU clusters and each hosting multiple inference engines.
00:01:53So, that's why we got the name of engine clusters.
00:01:56And as you may have already heard, nowadays GPUs are pretty popular and expensive.
00:02:02So, sitting in the middle, it is the IRB or inference load balancer.
00:02:07It actually runs on the front-end clusters, but it's also a bridge into our inference stack.
00:02:14It has two main responsibilities: select an engine and approximate the request.
00:02:19For this talk, we are going to focus on the engine selection part.
00:02:23So, in some ways, IRB resembles a very traditional load balancer
00:02:28because a request usually targets a model, and a model is backed by multiple engines.
00:02:35They may live on different clusters in different regions or even across the continents
00:02:40because that gives us a good resiliency towards localized degradation or cluster failures.
00:02:46However, the inference stack or the uniqueness of the inference introduces a lot of nuances,
00:02:53like it has to consider a bunch of signals reported in real time,
00:02:59like the well-known time-to-first token, TTFT, time-between-output tokens,
00:03:05also known as token-throughput or time-between tokens,
00:03:08and other harshness and utilization signals.
00:03:11Besides, there's an important concept of a Kiwi cache, which is also well-known.
00:03:16But, for example, when the conversation already has a lot of the useful context cached in one engine,
00:03:22sending the follow-up turns of the same conversation back to the same engine
00:03:27will avoid recomputation, improve efficiency, and reduce latency.
00:03:32So the combination of performance, reliability, locality, cache awareness is what makes it such an interesting problem.
00:03:39So how we attempted in the problem, let's take a look at the early days.
00:03:45And to be honest, early days in this industry sounds a lot more historic than it really is.
00:03:51And the routing process at that time began with a filtering of, like, each request may not be served by all the engines
00:04:01because of constraints such as capabilities or geo-restrictions due to compute or data residency.
00:04:08And among the remaining engines, IRB used a weighted consistent hashing to select the best destination engine for a request for a certain user.
00:04:19Then the important question becomes, where are the weights come from?
00:04:24So they were generated by a periodic feedback loop.
00:04:27The inference engines, as mentioned earlier, reports all kind of the signals we care about.
00:04:33And the controller will periodically smooth out those signals and compute a performance score.
00:04:40The performance score then will be compared against the fleet average.
00:04:45Then the weight will be adjusted basically for each engine if the weight goes up,
00:04:52if the performance is better, or it goes down when the performance is worse than the fleet average.
00:04:57And this generated weight will impact the routing and then it's basically a control loop.
00:05:05Conceptually, it's very similar to the PID controller.
00:05:10And no, this PID controller will not help you cure a Linux process.
00:05:13But instead, it's a classic control theory technique that continuously steering the system towards its desired state.
00:05:20And we just borrowed this important concept, the proportional part of it, and applied it into the system.
00:05:27So it has a lot of nice properties.
00:05:33For example, it could combine the useful signals we care about into the single routing decision.
00:05:40And because it adapts to the observed performance, as we mentioned earlier, there's a lot of constraints.
00:05:45And those constraints might have some engines basier because they can serve more kind of requests than the remaining.
00:05:53But those basier signals will be fit into the next loop and resulting in the less constrained request that can go to more of those kind of engines.
00:06:03So basically, they're self-balanced out.
00:06:05And to some extent, this just means we don't need to do a lot of manual intervention.
00:06:11And it just works.
00:06:12However, that kind of adaptability comes with big trade-offs.
00:06:16Because of the same reason that it combines so many signals, it's also very hard to reason about
00:06:22a particular routing decision, or why such engines get higher weight than we expect.
00:06:28And every time we want to fine-tune towards some aspect, it's almost impossible to not impact something else.
00:06:36And the load is not always very evenly distributed, because sometimes a model is served by engines on different GPU skills,
00:06:48and they have different characteristics, then the problem becomes a lot more trickier.
00:06:53And the feedback loop sometimes creates bad oscillations, because when you shift an engine away from traffic,
00:07:03the engine turns a bit cooler, and this signal gets fit to the controller.
00:07:08The controller now thinks, "Hey, this engine can take a lot more traffic."
00:07:12Then some traffic is going to be shifted back and forth between a few engines and disrupting the KVCache utilization.
00:07:20So all those limitations motivated us to rethink about the architecture and see if we have new ways to address the problem.
00:07:29So I'm going to hand it over to Qianru to deep dive into the new architecture we tried out.
00:07:38Yeah, thank you, Lu.
00:07:40So I'm going to talk about the architecture of the load balancer, and how do we reduce the overall overhead with our routing algorithm.
00:07:50The load balancer answers one question.
00:07:53So for each request from a CPU cluster, which engine should serve it?
00:07:59One most naive baseline might be round robin, which send requests across engine evenly.
00:08:06But if you think a little bit more, that doesn't make sense.
00:08:09Because engines are not homogeneous, they can have different hardware and capacity,
00:08:15different health, and also different distance from CPU cluster.
00:08:19Also, run robin could break cache locality.
00:08:24Related requests that could reuse the same engine cache might be sent to different engines.
00:08:30A probably better solution might be for each CPU cluster, it choose the best engine from its own local view.
00:08:39But that's not enough either.
00:08:41Think about one extreme case, multiple CPU cluster route traffic to the same engines independently,
00:08:48which could overload that engine while leave other engines underutilized.
00:08:53So what we need is a globally optimized solution.
00:08:57A control plan that has a global view for all the CPU cluster and GPU engines,
00:09:02and could compute a globally optimized routing answers.
00:09:06And the data plane can make a routing decision quickly based on the answer pulled from the control plane.
00:09:18Now, let's look inside the control plane and data plane.
00:09:22In the data plane, there is an engine selector, which selects engine for each request.
00:09:28It reads the local routing state, which includes the candidate engines and the routing weights for each candidate engine.
00:09:37Both of them are refreshed asynchronously in the background.
00:09:41So we don't need to ask the control plane before we make a routing decision for each request.
00:09:48Also, the data plane collects real-time engine signal, such as number of ready replicas, engine house, etc.,
00:09:56to service fast local guardrails.
00:10:00In the control plane, the data loader combines those live engine signals and never overhead.
00:10:06And with offline regressions of capacity, TDFT, and TBOT, the optimizer could turn those data
00:10:14into routing weights into routing weights.
00:10:16And the control plane will publish the routing weight for each data plane to pull.
00:10:21In this way, no request need to wait on the data plane.
00:10:26The control plane continuously computes the next globally optimized routing weight snapshot,
00:10:32while the data plane makes a routing decision based on the latest snapshot already installed locally.
00:10:42In summary, there are three important paths through the system.
00:10:46The first path is the inference request path.
00:10:49The request arrives to the CPU cluster, and the data plane inside the CPU cluster will select
00:10:55engine for that request based on the local routing state, and forward the request to the selected engines.
00:11:03The second path is the engine signal path.
00:11:06The system continuously collects real-time engine signal, such as TTFT, TBOT, number of ready replica,
00:11:15and engine house, etc.
00:11:17Both planes need those real-time engine signals.
00:11:20The control plane need them to compute a globally optimized routing weight, while the data plane need
00:11:26them to service fast local guardrails.
00:11:29And the third path is the routing weight path.
00:11:32The control plane compute and publish the routing weight, and the data plane
00:11:38pull the updates to its local cache.
00:11:41So only the first path is synchronous, but it's fast and only local inside the data plane of the CPU cluster.
00:11:49The other two loops are asynchronous loop, and they are to improve future routing decision.
00:11:59So that's pretty much of the architecture part, but that still leaves one question.
00:12:04How do we compute those routing weights?
00:12:10But before answering that question, let's answer another question first.
00:12:15Why not just send a request to the nearest engine?
00:12:20That's because the traffic demand and GPU capacity are not geographically balanced.
00:12:27For example, in Region 1, CPU cluster A send 90 RPS, and the nearby engine A can serve 100 RPS.
00:12:36So in this case, nearest-release only is fine.
00:12:40While in Region 2, CPU cluster B send 120 RPS, and the nearby engine B could only serve 100 RPS.
00:12:49So in this case, if we insist on keeping everything local, the extra 20 RPS need to wait on an overloaded
00:12:57engine B. While in Region 3, we are only using 40 RPS of an 80 RPS engine C.
00:13:05That still leaves 40 RPS spare.
00:13:08So if we send the extra 20 RPS from cluster B to Engine C, that will add network distance.
00:13:16But it could also avoid a probably much larger engine-side waiting time.
00:13:22So in this case, a further engine might be faster end-to-end.
00:13:27That's why we need something better than the nearest-only routing.
00:13:34Now let's open the black box of the optimizer.
00:13:39The optimizer accepts four types of input.
00:13:44The request from each CPU cluster, the network latency to each engine,
00:13:50the available engine capacity and health, and also the TTFT/TPOT latency profiles.
00:13:57That tells us how's the engine-side latency change as the load increases.
00:14:04And with those inputs, the optimizer turn the input to the output routing weights.
00:14:10The routing weights say for each CPU cluster, what fraction of its traffic should go to each GPU engine.
00:14:19And the optimization goal is straightforward.
00:14:22It's to minimize the expected end-to-end latency across all routing traffic.
00:14:28The important part is that the end-to-end latency includes both the network distance and the engine-side latency.
00:14:36That means a nearby engine might be attractive when it still has room to serve traffic.
00:14:43While a further engine might be better if all the nearby engines are close to full.
00:14:50And the optimizer also needs to respect several hard constraints.
00:14:56First, it needs to route all the traffic demand.
00:14:59Second, it needs to ensure all the engines stay within the effective capacity.
00:15:05Third, it needs to keep the routing weights non-negative.
00:15:09With this, the control plane get the routing weights from the optimizer and publish them.
00:15:17And the data plane pull them and use them to make a globally optimized routing decision.
00:15:24And that's pretty much of my part.
00:15:26And Lu will continue to talk about the protection mechanisms in the system.
00:15:33Thanks, Chenru.
00:15:34So as AI engineers, we all kind of know that production in many cases are not behaving in the most ideal case.
00:15:45So clusters can fill, GPUs or individual nodes can degrade, and networking can just get to all kind of mysterious issues.
00:15:55So how do we keep our production system healthy as much as possible under the heavy load?
00:16:02The first thing we have is penalties.
00:16:05Basically, when an engine is an outlier, we detect the abnormally and try to reduce the routing weight to that engine.
00:16:13In that way, we give it a chance to either recover by themselves if there's some transient issue,
00:16:22or we can have a human intervene to rotate it out or replace the faulty hardware.
00:16:28And secondly, the retries, which is a very common technique used to mitigate problems.
00:16:34However, during some cases, it actually could make them even worse.
00:16:39Like when the system is very close to like a tip over or very heavily utilized.
00:16:45Retries will send more load.
00:16:47And this more load will cause more failures and cause more retries, which is an infamous retries storm.
00:16:52So we implemented caps or budget to constantly retries into an acceptable region.
00:17:00And this is actually even need to be dynamic because in the happy time or in the normal time,
00:17:07we can tolerate a lot more retries than when the system are heavily utilized.
00:17:12And finally, we have the load shedding, which is our last result when the production capacity
00:17:20couldn't meet the increasing amount of inference demands.
00:17:24So we instead will try to have all the system fill.
00:17:27We basically proactively load shed a portion of the traffic to have the system degrade gracefully.
00:17:35So that pretty much concludes our talk today.
00:17:38And thanks for joining us.
00:17:40Both of us will be around in our booth area this afternoon.
00:17:45So if you have further questions, feel free to walk to the area and chat with us.
00:17:50Thank you.
00:17:51Thank you.

Key Takeaway

OpenAI scaled its inference routing architecture by transitioning from engine-driven PID feedback loops to an explicit control plane and data plane split that balances global network distance with real-time engine processing profiles.

Highlights

  • The Inference Load Balancer (ILB) routes user requests from CPU front-end clusters to engine-backed GPU clusters hosting model instances.

  • Early routing relied on a PID-like proportional feedback loop using engine signals, which caused traffic oscillations and disrupted KV cache utilization.

  • The redesigned architecture decouples into an asynchronous control plane for global weight calculations and a synchronous local data plane for low-latency engine selection.

  • A global optimizer minimizes total end-to-end latency by factoring both physical network distance and engine-side TTFT and TPOT latency profiles.

  • Dynamic retry budgets and proactive load shedding prevent cascading system failure and retry storms under heavy production stress.

Timeline

Role and Challenges of the Inference Load Balancer

  • The Inference Load Balancer sits between CPU front-end gateways and multi-engine GPU clusters across global regions.
  • Engine selection requires balancing real-time performance metrics like Time-to-First-Token (TTFT) and Time-Between-Output-Tokens (TPOT).
  • Routing decisions must preserve KV cache context across conversation turns to minimize redundant processing.

CPU front-end clusters accept incoming user requests and format them for processing before passing them to the Inference Load Balancer (ILB). Target models reside on GPU clusters that span multiple continents to ensure regional resilience. Effective selection algorithms weigh network distance against hardware utilization and cache state. Routing follow-up turns to the same engine reuses existing KV cache contexts, avoiding expensive recomputation and reducing latency.

Limitations of Feedback Loop Routing

  • Early routing applied weighted consistent hashing based on periodic, PID-like proportional feedback loops.
  • Combined metrics made individual engine weights difficult to reason about or adjust without unintended side effects.
  • Feedback delays caused destructive traffic oscillations between cooler and hotter GPU engines.

Initial routing filtered engines by geo-restrictions and capabilities before selecting destinations via weighted consistent hashing. A central controller periodically smoothed engine signals to calculate performance scores relative to fleet averages and adjust routing weights accordingly. While this self-balancing feedback mechanism required minimal manual intervention, it struggled with mixed GPU hardware generations. Furthermore, when traffic shifted away from an engine, its cooled performance metrics triggered sudden floods of new traffic, breaking KV cache efficiency.

Control Plane and Data Plane Architecture

  • The redesigned architecture splits routing responsibilities into an offline control plane and a fast local data plane.
  • Data plane routing decisions run synchronously without waiting for control plane network calls.
  • Three isolated execution loops process request forwarding, signal collection, and weight updates asynchronously.

Simple round-robin and isolated local cluster routing fail because GPU capacity and hardware configurations are heterogeneous across regions. The control plane aggregates global state—including ready replicas, engine health, and offline regressions for TTFT and TPOT—to compute an optimized snapshot of routing weights. The data plane pulls these weight snapshots asynchronously and executes synchronous engine selection within the local CPU cluster while applying real-time local guardrails.

Global Latency Optimization Model

  • Geographic imbalances between request demand and available GPU capacity require routing beyond local regional boundaries.
  • The global optimizer minimizes expected end-to-end latency by combining network travel time with engine queue times.
  • Routing weight calculations strictly satisfy overall demand without exceeding total effective engine capacity.

Routing strictly to the closest geographic engine creates severe bottlenecks when local request rates exceed local GPU throughput. Sending excess requests to a distant engine adds network transmission time but prevents significantly longer wait times on overloaded local queues. The optimizer processes regional request volume, network latencies, capacity bounds, and latency degradation profiles to output fractional routing weights per CPU cluster.

Production Stability and Fault Tolerance

  • Automated weight penalties temporarily isolate degraded engines to allow auto-recovery or hardware replacement.
  • Dynamic retry budgets cap maximum retry attempt rates to prevent system-wide retry storms.
  • Proactive load shedding drops a fraction of overall traffic during extreme capacity shortfalls to preserve baseline stability.

Real-world infrastructure faces hardware degradation, network failures, and sudden load spikes. When performance monitoring flags an outlier engine, the load balancer penalizes its routing weight to minimize user impact while hardware is serviced. To avoid cascading failures when capacity is strained, retry budgets adjust dynamically down from normal levels. Under extreme resource shortages, system-wide crashes are prevented by shedding non-critical traffic proactively.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video