Agents Are Where Microservices Were in 2015 — Roberto Milev & Uday Kanagala, Navan
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Hello, everybody. Welcome to our talk. My name is Roberto Milev. I am the chief architect at
00:00:20Navan and I have Uday here who is also part of the architecture team. Navan is a travel and
00:00:27expense management company and we'll share with you some of our learnings
00:00:32around how we run an AI and what have we discovered. So if you've been long enough
00:00:41in this industry you remember that over time there are a few paradigm shifts and
00:00:46we all tend to jump on a bandwagon and try to kind of do things right. Last time
00:00:53was when we all jumped on the microservices bandwagon and out of that a
00:00:58lot of good things came out like container orchestration, Kubernetes, then we had
00:01:05service mesh, circuit breakers, all of those good things. But it didn't happen
00:01:11overnight. Like it took a long time. It took some time for us to learn how to do
00:01:16these things. So one of the quotes from there is, if you can build a well
00:01:21structured monolith, why even try to build microservices? It kind of translates
00:01:26today because if you can't build a single agentic loop, why go in and try to build a
00:01:32multi-agent orchestrated system? So over time, just like previously, a reference
00:01:41architecture is emerging. So we have learned a few things by doing in production. We have a
00:01:50lot of agents, a lot of tokens per day being used. And as I said, there are a few layers that have
00:01:58standardized, that have crystallized around what do we need to run agentic flows
00:02:04reliably in production. Runtime memory, context management, all around operational
00:02:12cross-cutting concerns and around orchestration as well. So today we'll go over some of these layers,
00:02:18all of these layers actually. And we will show kind of where the industry is, what we have done, what we have learned, and so on.
00:02:27So starting at the runtime layer, we've talked a lot and we've built a lot of
00:02:34services in order to scale them statelessly before. And now we're in a new world where, you know, agents
00:02:43are stateful by nature. They need to have persistent sessions. They need to have isolation. Their life
00:02:51life cycle is different than the life cycle of a traditional API service and so on. So the cloud
00:02:59providers have jumped in and tried to fill this gap. You know, AWS, GCP, Azure, they all have some
00:03:10incarnation of agentic runtime. If you scan the QR code for this slide and for the following slides,
00:03:16you will see a comparison of some of the features and how different cloud providers try to approach this.
00:03:25At Navan, we run everything on AWS. AWS has an agent core runtime. We heavily use that, but we have filled some gaps
00:03:33around that, like the session persistence and rehydration is something that we have built. And we
00:03:41also run a bunch of other, a bunch of SDKs for writing agents. And part of these runtimes is typically
00:03:50their framework agnostic although they all prefer their native framework in a way.
00:03:58The next layer in the stack is around memory.
00:04:04We started with RAG. RAG was kind of a big thing for a while. We were kind of driven to that out of
00:04:11necessity because you cannot fit an unlimited amount of context into an agent. And over time,
00:04:20all of these cloud providers and the industry has implemented a pipeline where memory is kind of
00:04:27automatically generated by following a workflow of ingestion, extraction, and then consolidation and
00:04:33retrieval. And there are parts of RAG that are built in things like a long-term memory that
00:04:40inherently has some semantic characteristics. But memory is built up over time from short-term
00:04:46conversational memory to long-term memory that you kind of manage yourself. Then episodic memories about
00:04:54kind of instances that worked well and didn't work well and so on. We at Navan, again, being an AWS
00:05:02shop, utilize their agent core memory. But we are also kind of doing it in a way that
00:05:09matches our use case. And then the next thing is context management. You know, it's a hot topic. It was
00:05:18a hot topic and it's still a hot topic. Context windows are growing bigger, but there's never enough
00:05:23context. Or if there is too much context, again, agents struggle with that because you lose focus
00:05:29and so on. What we found working is that focusing on skills as a unit of context. And I'll explain what I
00:05:39mean by that. We look at skills as both having context, meaning instructions and setup about a certain
00:05:48domain or a task. And there's also the second part of the skill, which is the tool execution and, you know,
00:05:54the agentic part. And we compose context dynamically out of skills that we use as units of works that are
00:06:06pluggable, that we can test independently, and that we can reuse. So, for example, when we have an agent,
00:06:15we have skills that are specific to a domain. And based on that, we compose them. And we rely on the
00:06:24progressive disclosure, which is a feature of the skills itself to start with the limited scope of context and then
00:06:32expand by included metadata further down the line. I'll hand it out to Uday now to kind of walk us through
00:06:42the rest of this. Thanks, Uday. All right. Can I have a quick show of hands here? Who had built an agent
00:06:52that failed halfway through a multi 20-step or 30-step process and be able to figure out quickly or reason
00:07:00about why the agent failed?
00:07:05So, again, logs, we've generally been traditionally with microservices. We all are familiar with logs.
00:07:11There's logs out there, and then we go check out the logs. But this changes everything the moment we
00:07:16switch to agents. Agents output a lot of thinking. There's too much to consume. So that's not the right
00:07:22way to do it, right? So traditionally, that was the way, but our thought has to be changed right now
00:07:28in the way that Claude has an example. When we take Claude as an example for an agent, there's hooks,
00:07:35and we can intercept everything that Claude as an agent does at that level. So what kind of tool it calls,
00:07:42right? What kind of decision it's making? So before pre-tool and post-tool call or a pre-session or a post-session. So all of that
00:07:49are a point in time for us to intercept and make a decision and either block
00:07:56to do a blocking operation or to log a metric or emit a metric, right? So this is a critical
00:08:04place where we can emit otel traces. At now on, we use one of our provider brain trusts to emit these otel
00:08:11traces. And through these traces, we should be able to figure out the spans, the traces, and at what
00:08:17point in time where the agent is stuck, which gives much more confidence into how we operate and build the
00:08:25agents. This is a day two operational challenge. Building agent these days, there's so many frameworks,
00:08:32but how do you navigate building and operating an agent later is a primary concern now.
00:08:39And moreover, the reasoning chain, the thought process, and critical signals that we emit here. As
00:08:45part of the trace captures, we emit a few primary signals here. What is the current goal the agent is
00:08:52going through? The reasons behind its operations and the belief status and the tool calls that it's
00:08:57making. So this kind of gives us a judgment pointers in the traces. And when the agent makes a decision,
00:09:07there is a confidence score, how confident it is when it makes this judgment, right? So whether there
00:09:12are multiple paths that it leads to this choice or whether this is an inferred answer. So basically,
00:09:18these are signals that gives us confidence later to review. If this is an inferred answer, there could be a
00:09:24human in the loop to guide through and treat the agent to perform a little better.
00:09:32Again, can I have a raise of hands again to see how confident are you like 100% confident in
00:09:40testing pipelines with your agents?
00:09:43Right? So this is one of the other critical aspect today.
00:09:51Because agents are non-deterministic. We've all been used to program and write much more
00:09:55deterministic flows. And we know how it works. When I ask an engineer, the engineer can come and tell me
00:10:02how the algorithm, the sequence of operations, everything is programmed in our mind, everything is
00:10:06expectations. But now the agents come into a non-deterministic way, and how do we test them,
00:10:11right? So that is very criticality here. And yeah, we are also struggling. We've started
00:10:19doing building agents. The data operations was challenging, and then we failed in a lot of
00:10:23steps. How do we course correct? The moment we change something, something else breaks, right?
00:10:29So how do we do that? One approach that we took, this is from research papers around the concept of
00:10:36trajectory. Like in a multi-step orchestration, when an agent makes 30 steps or decisions to make to reach
00:10:44to a goal, if that is a program or that's a different story. But this is not a program. This is
00:10:51non-deterministic way of, it makes up its own steps every time differently. So how can we
00:11:00chart a deterministic graph here? Is it possible? No. Can we have a trajectory of its starting from an end
00:11:09to a goal and then see how much, how far it went in the trajectory and how far it went from the source
00:11:15to the destination? Is what we can compute to evaluate the efficiency or the completeness of the agent
00:11:23evaluation? So we heavily rely on trajectory evals. And there are a few other signals, as I briefly spoke
00:11:35around in the previous slide, around the inferred signal. If the answer is from an inferred answer,
00:11:42how can we loop that into and make signals around how can we classify that this is a regression and make
00:11:52fixes towards the agent? So the next is the guardrails.
00:12:11Is this the one? Yeah. So guardrails and authorization. This is a critical,
00:12:21displays a critical role in enterprise AI. A lot of information is being piped to models.
00:12:28There could be sensitive information that goes into it without our knowledge. And we, as leaders,
00:12:35how can we put in this governance layer to stop this is very critical here. And the concept of
00:12:44authentication and authentication and authorization is taking up a different approach here. Traditionally,
00:12:51we've seen a user or a service account, but now what is an agent? Agent can be acting as on behalf of
00:13:00users. There is so much of things, so many of use cases there. Hey, book me a flight whenever it's cheaper
00:13:06than $200, right? So we just tell this assertion and then agent go figures out and does this action on behalf
00:13:12of me. So is it me making this purchase or is it agent me making on behalf of me? So there is
00:13:19Agent acts on behalf of user or agent user service account as well. So the line is being blurred here and we need to
00:13:27make fine-grained authorization decisions here. And the policy layer, that's where the guardrails
00:13:32and authentication authorization plays a critical role. And in Navan, what we employ here is
00:13:39Before every tool call, pre-tool and post-tool, we have these guardrails to check and block and make informed decisions.
00:13:51And this single agent versus multi-agent, again, this is kind of orchestration wars you can think of
00:14:00whether to build a single agent or a multi-agent. Again, as Roboto briefly hinted,
00:14:05if you can't perfect and build a single agent, why go towards multi-agent, right? So learn from our
00:14:14failures, experiences, and build towards that. At Navan, yeah, the approach that we have taken is
00:14:22single master and then we adopted sub-skills. There are sub-agents within it. So it's a single agent that can
00:14:32progressively load the skills and understand decisively what needs to be loaded into the context and then
00:14:39make this navigation through the use case. But there are other patterns that are also emerging.
00:14:48There are different class of use cases here. One is
00:14:51agent-to-agent communication. So there are, if you take a large-scale organization and there are so many of
00:14:56these teams that are acting as a boundaries and they don't talk to each other, let's say,
00:15:02how do we communicate? There are two agents on either of this side, right? How do we do it?
00:15:07So there is A2A protocol which can help us establish the contracts in terms of skills
00:15:14and we can use A2A as a protocol there, which kind of is a boundary between the teams.
00:15:22Yeah, A2A protocol.
00:15:29All right. So as we went through the stack, it's obvious that some components of the stack are in a
00:15:36more mature state and we already have good answers for them. As I said, the runtime, I think it's
00:15:42pretty much solved. We are so advanced in orchestration and we are running LLMs in kind of a very
00:15:50brute force way. So scaling is not a problem. Also memory. I think as the frontier LLMs get better and
00:15:59as our practices get better, we will find a way to cover the majority of the use cases and there is
00:16:06good maturity around the cloud providers. MCP has emerged as a de facto protocol and tool calling is
00:16:15now a feature that everybody supports. So we are seeing some industry convergence around that as well.
00:16:23And MCP as a standard is also evolving. Now it's becoming stateless. We are reaching a point where
00:16:29kind of we know how to invoke services and tools with agents. In some areas, things are happening,
00:16:39but you know, there's still a lot of unknown around observability. There is a push towards otel, but
00:16:45does otel really work for agentic calls? Yeah, you can make it work as Uday was saying.
00:16:53Also, we are getting more comfortable around the testing patterns. It's very hard to test,
00:17:00but we have found a way to give customers quality experiences, even with the unreliability of agentic
00:17:07system. And I think that's kind of getting in a state that is more or better defined. Orchestration is
00:17:15another one where we have patterns. We can build bigger agents, smaller agents. As we said previously,
00:17:29probably the right answer is to not over engineer. So we are learning there and a pattern is also
00:17:40emerging. Where we are all struggling with, and the previous talk was about this for the developer
00:17:47AI assisted development perspective, but also we are seeing these issues from our production agents. It's very
00:17:54hard to predict cost and it's very hard to manage cost and put guardrails and solve this in a way where
00:18:02there is a reliable maybe fallback or have agents be using cheaper models for certain tasks.
00:18:12This is all driven by kind of the big AI vendors who I think their interest is for us all to spend
00:18:20more tokens. Replay and debugging. Ude talked about that. That's also a big, big issue. It's very hard
00:18:28to understand, but I think this is also something that that is going to be solved because we can now use
00:18:36agents to get over the cognitive overload of trying to debug what they do. And then standards. Standards are
00:18:46emerging by the community. Hotel, as I mentioned, agent to agent is young. It's kind of pushed by
00:18:55certain vendors, but I think over time we will get there. With all of this said, we know what we need
00:19:04and it's up to us to try and build it. Thank you, everybody.