Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio
AAI Engineer
Computing/SoftwareManagementInternet Technology
Transcript
00:00:00I'm Ganesh Manuja. I'm a principal engineer at Twilio. Let's start with a quick show of hands.
00:00:20Who here has seen the message, something went wrong, please try again?
00:00:27Well, we have a few lucky ones and a few that have had a good lunch.
00:00:33So, behind that simple message is actually a system that is very complex that serves you that message despite the model providers being down.
00:00:45And that's what we're going to productionize today or discuss productionizing today.
00:00:50So, what is an LLM gateway?
00:00:52An LLM gateway is an entry point or a middleware between your apps and the model providers behind them.
00:00:58It does a bunch of things, routing, authentication, fallback, rate limits, all kinds of governance that you can think of.
00:01:08And right at the heart of the gateway is a fight between four things.
00:01:13It's availability, latency, your guardrails, and costs.
00:01:18In case of a degradation, you cannot maximize all four.
00:01:22You need to pick what you want.
00:01:25So, with this talk, if you use an LLM gateway, I want to help you to make that trade-off for your use case.
00:01:35And if you design a gateway, I want you to design or provide those levers to your callers and customers so that your customers are happy.
00:01:46Let's start with availability.
00:01:50If you have a single model provider, their ceiling is your ceiling.
00:01:56Their outage is your outage.
00:02:03So, in typical software engineering, the way you tackle unreliable dependency is by retrying.
00:02:11Retrying with exponential backups, with jitters.
00:02:15And when all of that fails, you have a circuit breaker that trips after you've seen sufficient failures, and you stop calling the damn thing.
00:02:24This is not enough for LLMs.
00:02:26LLMs are very different compared to your fast, cheap APIs that you retry on.
00:02:32Retrying an LLM API eats into your latency budget really fast.
00:02:38And also, tripping over a circuit breaker when you have another perfectly fine model provider to route to doesn't make sense.
00:02:47You should use the second model provider.
00:02:48And third, as I said, the calls are slow and expensive.
00:02:53So, blind retries, just multiply your cost and your tail latencies.
00:02:58So, what is a better idea here?
00:03:02It is actually a per request fallback.
00:03:05What that means is you can actually try model provider A and then, in sequence, try model provider B if your request to model provider A fails.
00:03:14Another option to consider here is you can fire requests to both providers in parallel, but that's only if you're highly, highly obsessed with latencies, because that's just going to double your cost.
00:03:26Some of the similar circuit breaking patterns apply here to LLMs as well.
00:03:32If you know that your primary has been failing for some time, it doesn't make sense to try it again.
00:03:40You take it out of the load balancer or your request path and put it in a cool down and then, after a few minutes have passed, try putting that back again.
00:03:51One interesting choice that you have to make here is where your failure counts live.
00:03:59You can decide to have the failure counts live in memory on the instances that are serving your traffic, or you can have shared info where your failure counts are shared across the fleet.
00:04:12There are trade-offs.
00:04:14If you want quick failovers, then fleet-wide helps.
00:04:19And with instance, with local state counters, the issue that you run into is whenever you change your deployment size, your configuration and your expectations change.
00:04:30So something to consider.
00:04:34What that clean diagram did not really show you are some of the other gotchas that I'm going to discuss.
00:04:39So fallbacks are not transparent.
00:04:41While the industry is converging on an OpenAI API-compatible format, I would say there are still nuances.
00:04:49So you need to really test your fallbacks well.
00:04:52They can have differences in your tool calling schemas, token limits, stop reasons, and what have you.
00:04:58So with LLM gateways, you can have a normalization layer that can ensure that you can do cross-provider fallbacks as well.
00:05:08Another thing is streaming.
00:05:15So essentially, nobody wants to wait for 30 seconds to have a wall of text appear in front of them.
00:05:22So there are use cases where streaming is absolutely required.
00:05:26But it comes at a cost.
00:05:27You trade away your levers.
00:05:29You cannot -- once you've decided to go with Provider A, you have to continue going with Provider A.
00:05:36You cannot midstream change the providers.
00:05:39Whatever has been sent to the client, it's done.
00:05:42And that's where the something went wrong message.
00:05:46That's the one that you see.
00:05:48It's not because of laziness.
00:05:49It's by design that you see that.
00:05:52And it's one of the tradeoffs.
00:05:54I would like to call out one other thing where I've seen teams trip over and over again.
00:06:00They really provision and test their primary providers really well.
00:06:05But the second provider, the fallback provider, doesn't necessarily get the same level of love.
00:06:11And I would argue that your throughputs or your capacity or your headroom should be even higher for the second provider or the fallback provider.
00:06:21Because that's your last line of defense.
00:06:23If that goes down, your application goes down.
00:06:29Let's discuss latencies.
00:06:31Availability failures are right in your face.
00:06:34They fail.
00:06:36You get alarmed.
00:06:37You get paged.
00:06:38But high latencies can be the quiet ones.
00:06:42And they need to receive more love than, I would say, tuning your services for just availability.
00:06:49One thing to call out.
00:06:54A gateway may run mixed workloads.
00:06:58And you can have embedding requests that take just less than a second.
00:07:04You can have classification requests that take less than a second.
00:07:07You have chat requests taking three seconds.
00:07:10And reasoning requests taking a long time.
00:07:13Quick show of hands.
00:07:15Quick show of hands if you measure your aggregate latency for your entire service.
00:07:20Well, that was a trick question.
00:07:23Sorry.
00:07:24You shouldn't.
00:07:25It doesn't make sense.
00:07:26It's a lie.
00:07:27You should be tracking your P99 per model per route, not a gateway wide number.
00:07:32Gateway wide number doesn't make sense, especially if you're running mixed workloads.
00:07:36And I hope you're not, for those who've raised your hand.
00:07:40Another thing that can really -- I cannot emphasize this enough is for you to set timeouts on per model class per route.
00:07:49That's where -- that's the number one root cause of your silent outage.
00:07:54If you don't have a timeout, your gateway things, your request is being happily served, while it is not.
00:08:00And I'll leave you with this message for latencies, specifically.
00:08:05A reasoning model's normal is actually a chat model's outage.
00:08:09So you definitely need to track latency per route.
00:08:13Okay, this is the most painful or the slide that has given me the most scarce, which is reasoning and router models.
00:08:24So this is where, truly, the latency is unpredictable.
00:08:29And reasoning models, they do not give you -- they're highly undeterministic, more undeterministic than your normal models.
00:08:40You cannot set the temperature to zero in many cases.
00:08:43And the same prompt can take somewhere from two seconds to 60 seconds.
00:08:48And we've seen that in production, where P99 suddenly popped to 60 seconds for no good reason.
00:08:53So that's -- while there's no magical solution to it, I would recommend that you at least start with fixing the reasoning level per route.
00:09:03So with router models, they hide that abstraction behind you.
00:09:08Like, they pick which models to run.
00:09:11And I would highly recommend that you at least make as much -- you make requests as deterministic as possible with an undeterministic system.
00:09:23Another idea is hedging the tail.
00:09:26You can have a -- you can fire another request if your primary request actually consumed, let's say, P90 of your latency budget.
00:09:37This can hedge the -- this can really hedge the P99 tail for your services.
00:09:44All right, this is one of my favorite ones.
00:09:47To keep your model secure, you need to have guardrails.
00:09:53And with that, guardrails are necessary for preventing your services from prompt injection attacks, keeping PII filters in place, having toxicity filters, keeping the LMs to stop swearing at your customers.
00:10:08All those good things.
00:10:11But just like a model provider, there are tradeoffs too.
00:10:15Guardrails are just like another service.
00:10:18That can go down.
00:10:19That can be unreliable.
00:10:21And that's where you need to choose.
00:10:24Do you fail open or do you fail close?
00:10:27When I say fail open, you can still serve the request even if your guardrails are down.
00:10:32Fail close, you block the request and say, hey, I'm not available.
00:10:36That's the tradeoff between availability and security to a certain extent.
00:10:41While there's a no universal answer, it really depends on your use case.
00:10:45You can decide, like, for example, a toxicity filter, if it's not up and running, you can still serve that request.
00:10:53So the default choice should be the worst case that you can live with.
00:11:02There are a few things that you can actually do to improve the behavior of your systems in face of, you know, guardrails being down and managing just unreliability of the guardrails themselves.
00:11:17So the first is time budget.
00:11:20Your request should never be bound by your guardrail timing.
00:11:25It should always be the LLM that is the rate determining step.
00:11:29So make sure that you have timeouts in place and those guardrails run with a specific time budget.
00:11:38Another important thing is fallback.
00:11:40You've heard -- you probably know, and I've talked about it, we always discuss fallbacks with regards to model providers.
00:11:47But guardrails are critical services, too, where you can consider fallbacks, have secondary providers, secondary checks, cash decisions to keep your service available when a guardrail provider is down.
00:12:03Another interesting choice that pops up with regards to guardrails is the placement of the guardrails.
00:12:10Typically, you can place the guardrail in three ways.
00:12:15You can have a prehook that runs -- where the guardrail actually runs on the input.
00:12:19You can -- and that's probably the safest, but it does add serial latency to your requests.
00:12:26Another one is in parallel.
00:12:29This is one of my favorites, but just to call out, streaming wouldn't work well here with parallel.
00:12:35So if you're specially producing structured output, please don't stream them.
00:12:40Try to save your latencies and run these guardrails concurrently for your structured outputs.
00:12:46Another one is post hooks.
00:12:48These are best for output monitoring, auditing your outputs, and so forth.
00:12:58So, so far, we've all -- I've discussed all the things that can go wrong with regards to our dependencies.
00:13:06We haven't discussed that we are actually adding another dependency in the request path itself, which is the central -- or which is the LLM gateway itself.
00:13:15There are a few things where we have been bitten by, and we've learned some lessons that I want to share with you if you're working on an LLM gateway or using one.
00:13:25One is shared limits.
00:13:28Make sure that your API keys are segregated per route, per use case, to the most granular possible -- to the most granular thing that you can imagine.
00:13:40Having a noisy tenant can be one of the biggest problems here.
00:13:47Another thing is load shedding.
00:13:50This is a feature that you should, as part of your runbooks, game days, make sure that the gateway that you're using supports load shedding.
00:13:59Because when you have a retry storm, it becomes really hard to just scale out.
00:14:03You cannot simply scale out services that is under a retry storm.
00:14:07And all these web servers, they have an internal queue, and they're configurable.
00:14:13Make sure that they're bounded, and they cannot request -- they cannot accept requests that are unbounded.
00:14:19And if you want to have some custom logic, you can even have traffic prioritization here as well, to make sure under load your most important use cases get served well.
00:14:29Last thing that I want to discuss is the whole idea of a central gateway itself.
00:14:38It is a single point of failure.
00:14:40So if you're thinking of having a central gateway for your entire company for two LLMs, I would recommend rethink that and see what are the reasons that you want it.
00:14:52What I've noticed is that in most scenarios, it's not the central gateway that they want.
00:14:57They want centralized governance.
00:15:00And there is a path forward where you can actually decentralize the gateway and still centralize governance.
00:15:07So do not try to centralize your traffic, but you can have plugins, you can have custom code that can centralize your governance.
00:15:17Governance can be in the form of cost tracking, rate limit management, and there are other solutions possible.
00:15:24So explore those before you chart on having one central gateway for your entire company.
00:15:30It can be managed by a single team, but I wouldn't recommend deploying it as a single deployment for the entire company, even though it's distributed.
00:15:43With that said, I want to end this talk on a personal note.
00:15:47So it is my son's birthday today, and I'm here talking to strangers about circuit breaking.
00:15:54So the least you can do for me is please go and prevent one incident for me and for your customers.
00:16:02Thank you.
00:16:03If you have any questions, yeah.
00:16:06.