Every step you take, every call you make: the reliable agent stack — Giselle van Dongen, Restate
AAI Engineer
컴퓨터/소프트웨어AI/미래기술
스크립트
00:00:00Hi, everyone. This talk will be about how to run agents reliably in production. It will not be
00:00:18about the eval part, but it will be about all the other things you need to get going in order to run
00:00:23agents resiliently. So, the infrastructure layer, basically. I want to set the scene with this quote
00:00:30of Andrej Karpathy of last week. It describes that the way we interact with agents and LLMs has been
00:00:36evolving in three waves. The first wave was an LLM being something like a website where we go to,
00:00:43we ask it a question, it thinks for a few seconds, and then gives us a response. The second wave was
00:00:50going towards agents. It was an app that we downloaded to our computer. It has some tools
00:00:55at its disposal, and it can do some work with our interaction. Now, the third wave will be going
00:01:02more and more towards persistent and asynchronous entities. So, agents being long-running processes
00:01:08in our infrastructure with access to tools and other agents around the organization and context.
00:01:15And so, as our use cases are evolving more and more from single agents to agentic platforms that connect
00:01:23parts around the organization, our infrastructure layer should also evolve with that. So, when we look
00:01:30at the types of tools that are currently out there to implement agents, a lot of innovation has been done
00:01:36on sites such as agent seks and memory. And agent seks are really cool to implement POCs and get started
00:01:44quickly, but they don't necessarily help with, like, connecting the distributed bits around an
00:01:49organization. And if you want to implement more complex agentic systems, you actually need all of those
00:01:56things. So, that is the layer that you see below here where you have to deploy extra infrastructure. You need to
00:02:04write things like retry logic, recovery logic, and all of that is actually pretty complex to get right,
00:02:10but completely necessary to run long-running stateful and distributed processes in production.
00:02:17So, today, I want to talk about an open source framework called Restate, and you can see it a bit as a flexible, durable
00:02:24foundation that lets you build any back-end. So, it's not specific for agents, but as agents are also just
00:02:31a type of a back-end, it also works well for them. The ideas behind Restate come from Apache Flink,
00:02:39which is a popular distributed stream processing engine, and also from some of the ex-architects behind
00:02:45Metascore Event Infra. So, what are the ingredients in Restate? Basically, four parts.
00:02:53First of all, it makes sure that a single run of an agent is resilient. This is called durable execution
00:03:00in the industry. Think about things like when an agent runs for a week and then crashes, we want to be
00:03:06able to bring it back and let it continue exactly at the point where it failed. We don't want it to start
00:03:12over from the beginning. Another area here is running many concurrent sessions in parallel. Imagine running
00:03:20thousands of concurrent agent sessions at the same time and needing to make sure that state is always
00:03:26consistent and that different agents don't interfere with each other. And then going more towards
00:03:32things like communication between agents, between agents and MCP servers and other tools. And finally,
00:03:38also control. Making sure that when an agent, for example, is doing something you don't want it to
00:03:45continue or when it's stuck being able to actually cancel or kill the execution.
00:03:51So, the way that you can think of it is as follows: Restate is basically a server which runs in front of
00:03:57your agent service, so as a separate component. It sits there a bit like a message broker or a proxy.
00:04:04And when there's a request for your agent, Restate proxies the request to the service and pushes
00:04:10it to the service, basically. And from that moment, there's an open connection between Restate and the
00:04:16agent. And that connection will basically be a bit like a lifeline for the agent. So, as the agent is doing
00:04:22stuff, it sends events over to Restate. And Restate will use that journal of events to recover the process
00:04:30after a failure. So, from a slightly higher level explanation, you could say that it's turning a
00:04:37normal function in your application into something that is long-running, durable, and stateful, without
00:04:44having to do a lot of the complex things you otherwise need to do for this. So, my talk today will be
00:04:50mainly a demo. So, I'll be showing you a research agent that's connected to Slack. Imagine we are
00:04:57like working at some company and we want to make a Slack agent available to all of our employees.
00:05:03So, if I go here into Slack, then I can hear in this channel, for example, ask what is new in AI.
00:05:12Now, let's have a look at what it's doing under the hood. So, if I go back here, I have here the Restate
00:05:18UI. This is a bit like a cockpit for your agents. So, you can see a registry of all the agents that are
00:05:25currently registered. And you can also see, for example, which execution is currently happening.
00:05:31So, here is the deep research agent that I spinned up a few seconds ago. We can see what it's
00:05:37currently doing now. It called first an LLM and then it sent me an answer via Slack. This first LLM call was
00:05:44a planner agent. So, what it did is it planned the research and sent me a list of subtopics that it
00:05:51wants to research. Now, if I press here approve, then this will unblock the workflow and will spin up a
00:05:59set of parallel research agents. So, this is basically like the classical deep research workflow, right?
00:06:05You have a planner, then a set of sub-research agents, and then finally someone who writes a report on
00:06:12this, like a writer agent. And so, this journal you see here on the left, that is basically the events that
00:06:19get sent from the agent to the Restate server. And if this now crashes at some point, this journal is what will be used to
00:06:27recover the execution to the point where it failed. I don't know if there were some errors. I injected a bit
00:06:34of like tool errors in here. Yeah. Here you can, for example, see that the sub-agent first did an LLM call,
00:06:41then started doing some web searches, and eventually one of the web searches didn't go through because the
00:06:47API was down. And then you see here on the right how it got retried and eventually completed successfully.
00:06:53So, instead of starting over, it uses the journal to recover the progress. Let's now have a look at what
00:06:59this looks like in code. So, the basic unit of how you implement applications in Restate is by writing
00:07:07HTTP handlers. And those handlers become durable by using the Restate SDK. So, here in this case, we have
00:07:15here our deep research handler. And here, as a first argument, we have a Restate object context. And the
00:07:21way you can imagine that is basically as that connection to that Restate server. Whenever I do an action on this
00:07:28Restate object, it will lead to an event being sent to Restate. So, for example, when I did that planner LLM call,
00:07:37what actually happened under the hood was it executed here this Python function. This is just a simple
00:07:43light LLM-like LLM call. And the way I made it durable is by wrapping it in Restate.run.
00:07:54So, what happens is by doing these durable steps, if this fails somewhere here, two hours or two months
00:08:01later, it will recover to exactly that point. So, that's the idea of durable execution. You're always
00:08:07able to recover a process to where it was. You can also use that for other things, not necessarily
00:08:13for failure recovery. For example, imagine we want to ask a human to approve something and this approval
00:08:19might take weeks or a month. This process needs to be able to survive restarts and redeploys over those
00:08:28kind of long periods of time. And so, with durable execution, you can actually also suspend a function
00:08:36and bring it back when it's able to make progress. So, in the case of a human approval, what we do here
00:08:42is basically we create a durable promise, which lives in that journal, a bit like a suspension point.
00:08:49Then we ask a human to click that button in Slack, as I showed in the beginning. And while we are waiting,
00:08:56this process actually suspends. So, if it's running on serverless, this is not using execution time on
00:09:04our functions. Once the response comes in, this then gets unblocked and can continue where it left off.
00:09:11So, what we see here is a bit like a workflow. It's a set of steps that get executed durable.
00:09:17But when we think about agents and also the way that Karpathy described it in the tweet, it's more like a
00:09:22persistent stateful entity that lives for a longer period of time, that has some memory. So, a workflow
00:09:30is not the nicest way to model this kind of thing. So, the way that we can model this in Restate is by
00:09:37using something called a virtual object. So, imagine in the use case that I'm showing this Slack research
00:09:43agent, imagine that I don't want to wait for 10 minutes to give it some follow-up context or maybe
00:09:50I think about something else that I should have told it. I want to actually be able to interact with it,
00:09:55not wait till that research is finished before I can send a follow-up. And so, this is basically
00:10:02what a virtual object in Restate is. It's a bit like a stateful actor. It has a unique ID, for example,
00:10:08a session ID. It has some key value states that is isolated for that specific session that you can
00:10:15write to. Imagine, for example, your history of messages. And it also has like a set of handlers
00:10:22that can execute durable functions for this session. So, here, the way I implemented this use case that I
00:10:31mentioned of interacting with a running process is as follows. This is a session controller. Again,
00:10:40it has like this Restate object context at its disposal to do things in a recoverable way. It can
00:10:47write to this session store. Here, I'm retrieving the chat history. And one thing that's interesting
00:10:53there is that in order to run these kind of sessions in very high parallelized ways, so thousands of
00:11:00sessions at the same time, we need to make sure that agents do not interfere with each other. Imagine I'm
00:11:07sending two messages on Slack and now two agents are actually overwriting each other's session state.
00:11:13To prevent that, this will guarantee that only one execution is running at a time. So, a second
00:11:19execution will be queued behind the current one.
00:11:26Then, let's have a look at how we implement this, like interacting with another execution.
00:11:31So, an execution in Restate has a unique identifier. And you can use that identifier to connect to it from
00:11:38other processes. For example, to retrieve the output, but also to cancel it or maybe to signal it, being
00:11:46injecting a bit of state into an already running agent loop. And so, this is like a very flexible type of
00:11:56capabilities that you can do to implement things like, for example,
00:11:59signaling an already ongoing agent loop. So, what we do here is if there is a current execution ongoing,
00:12:06then we will ask an LLM, is this like something that is relevant for the current agent loop?
00:12:13If that is the case, inject this via a signal. If it's not really relevant for what we're currently
00:12:19doing, then cancel what you're currently doing and start over again with this new information.
00:12:26And so, this goes a little bit further than workflows. It goes a bit more towards like writing persistent,
00:12:31stateful entities that can interact with each other and have memory at their disposal. So, let me show
00:12:38you how this works. So, here, if I now ask again what is new in AI and I wait a few seconds, then it should
00:12:47respond again with a plan. And then I can say, for example, some extra info focus on frontier models, let's say.
00:12:58So, once I have the plan, I will inject that bit of extra state.
00:13:06Now, let's look at the UI of what this is now doing. So, here I have that controller which I just showed.
00:13:12It started calling an LLM to classify this new input.
00:13:19Once this comes back, it will probably decide that it should signal it because it's still relevant to the
00:13:24research it's currently doing. So, this injects that new message into the ongoing agent loop. So,
00:13:31let me show you in the deep research agent again. So, first it's called an LLM, then asked us,
00:13:37then we injected this new message of focus on frontier models, and then it took that into account and
00:13:45started over again. Here, I can now, for example, also say something like, forget about that.
00:13:57Research AI policy. And if I send this, then the coordinator will decide to cancel the ongoing run
00:14:06and start a new one that will research this new topic. And so, this cancellation is basically like a
00:14:13signal that gets sent down the stack of or the call chain. So, if my agent was already spinning up
00:14:22sub-agents first, so sub-agents would be cancelled, then the controller itself. And like that, it would
00:14:28basically rewind the stack and give agents also the ability to roll back. Okay. So, this went a bit more
00:14:35into the direction of like stateful persistent entities that we can interact with over longer periods of
00:14:41time. Now, the last part of the demo that I want to show is going more towards like being able to write
00:14:48highly customized applications. Imagine that we deploy this in production, but then a few months
00:14:54later, a new model provider brings out a new model, for example, Fabulous. And even though the model is
00:15:00very good, it's also very expensive. And we notice that this research agent is actually starting to cost a lot.
00:15:06These kind of things that pop up halfway through a project require you to then deploy a lot of new
00:15:14extra infra or like find a good way to solve this. This is the kind of things that Restate really excels at.
00:15:21It doesn't really peg you into a specific way of how you should write your application. It basically gives you
00:15:26like a durable programming model that lets you implement an application in the way that fits for you and also
00:15:33extend it if necessary. So, first, I showed this LLM call in the first example as an inline step. It was
00:15:42just a Python function that got persisted. But imagine this use case that we want to actually have a bit more
00:15:48control over those LLM calls. For example, what you can do is pull this out into its own handler.
00:15:56And this handler can now do things like, for example, a policy check and then do the LLM call.
00:16:03And the other agents, instead of doing this LLM call inline, can now use Restates like distributed
00:16:09communication primitives to actually just call this LLM gateway instead of doing it as an inline step.
00:16:17And this service fabric that lets you communicate between agents also gives you some things like
00:16:23flow control. So, we can, for example, say one department is only allowed to run 300 calls to
00:16:30this LLM gateway at the same time. So, the reason why I showed this was just to show you a bit like that
00:16:37it's basically just a resilient foundation. It makes sure that your process can recover from even more
00:16:44advanced types of infrastructure failures, things like network partitions and zombie failures.
00:16:49And it gives you, like, tooling to extend and customize as your use case grows.
00:16:57Let's go back to the slides to have a little more of an idea of how this thing is actually
00:17:03implemented on the inside, because it's actually a pretty interesting
00:17:09design or architecture. So, the way it's implemented is basically by having an event-driven
00:17:15distributed log implementation. So, inside the box, you basically on one side have the clients,
00:17:21on the other side the services, and inside the box is a log which persists all those journal
00:17:27events and an event loop. And that event loop basically gets the events from the service
00:17:33based on what the event is. It either persists some state in the embedded state store, or it sets a timer,
00:17:40or it sends a request to another agent. And by doing that, you basically have a durable
00:17:46foundation for whatever an application is doing. The design of this distributed log is heavily inspired by
00:17:55the way that the core event infralayer at Meta works. It's basically like an iteration on top of that.
00:18:03And some of those architects now have designed that for Restate as a more generic solution that is
00:18:09available in open source. There are two important things related to this architecture that make it
00:18:15interesting. The first one is that it works as a push model. So, whereas most workflow orchestrators
00:18:21actually pull for new tasks, pull from the workflow server, Restate actually pushes the invocations.
00:18:30And the benefit you get from that is that it has a much lower latency. So, you can use these kind of
00:18:36workflow guarantees in functions around your application and have like a latencies of, for example,
00:18:4345 milliseconds p99 for like a 10-step workflow. Pushing invocations also works very well for serverless
00:18:51because they require you to basically send the request and wake up the function. So, this design that I
00:19:00show here includes includes everything you need. It includes as well that state store where we were
00:19:06embedding the state as the UI. It's a single binary, so it's pretty easy to operate as well. To run it in
00:19:13like a highly available way, you just spin it up multiple times and let it snapshot to object storage.
00:19:21So, Restate has six different SDKs. We also have integrations for most of the popular agent frameworks out
00:19:29there and, of course, because it's just like a flexible layer, you can also just use any LLM SDK and
00:19:35implement custom agents by just wrapping some steps into these SDK constructs. So, it's open source. You
00:19:43can self-host it. We also have a BYOC offering where we deploy Restate in your cloud account and that
00:19:51gives you the benefit that data doesn't leave your cloud account. Otherwise, there's also a managed
00:19:56cloud offering. This was mainly what I wanted to show. If you want to explore the code a bit further,
00:20:03there is here the GitHub repo. It's publicly available. If you like the project, then have a look at the
00:20:09Restate repo itself. We are hiring across the board for all sorts of roles going from engineering to marketing,
00:20:16especially also here in the Bay Area. So, if you're interested in that, then definitely check out our
00:20:21careers page. And I will be outside in front of the conference hall here if you want to ask any
00:20:28questions or learn more about Restate. Thank you very much.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기