Transcript
00:00:00Hi, I'm Brandon Bloom. I lead AI products and AI product engineering at Brex. Today I'm going to
00:00:12tell you about our audit agent. We make products that include our corporate card for businesses,
00:00:19as you heard Tom tell you about earlier, and we have policy to enforce on corporate spend.
00:00:27Our audit agent finds waste, fraud, abuse, and other out-of-policy spend inside all these
00:00:34transactions in your business. We're going to tell you why we gave this audit agent a bash show,
00:00:39and what happens when we did. Many of you are building powerful agents that are taking on
00:00:46challenging tasks. Allow me to set out the challenge that lays ahead of our audit agent. The industry
00:00:55benchmark is that about 5% of all corporate spend is out of policy. The difficult question posed to us,
00:01:02and to our agent, is which 5%. To solve that problem, we needed to really understand the
00:01:09natural language policies that drive what it's okay to spend corporate money on.
00:01:16Expense policies are actually really strange. You might think you can have very simple rules about how
00:01:22much money you're allowed to spend, but it turns out everything I'm about to show you is a real policy
00:01:27from a real company. For example, you can only use $75 a day for food when traveling. Unless, of course,
00:01:34you're an executive, you can spend as much as you like. You're not allowed to book an Airbnb, unless you've got
00:01:40dietary restrictions and you need to cook a special meal. You can't upgrade for a flight, unless you're taller
00:01:49than six foot five and you really need the leg room. You also can't buy dinner for your friends, unless you didn't
00:01:56stay in a hotel and you're saving the company money and you want to thank them for being a good host. These
00:02:01have all been in real expense policies, and there's no rule engine on the planet that is going to be
00:02:06able to enforce in or out of policy for these and the many subtle and interacting rules that a company may
00:02:13define. So once we understand the policy, we need to flag those expenses that might be out of policy,
00:02:23cluster them together, and provide suggestions through our case management system so that you can take
00:02:29action. Either remind someone of the policy, request reimbursement, and the like. So here's some real
00:02:38cases in our agent system that show some of this being found here. So Sarah took a flight on Delta
00:02:45Airlines. She decided she wanted to extend her business trip to go over to Italy and spent a little
00:02:50extra too much money doing so, and that's not acceptable. Or here, Elena decided to treat her family to a meal
00:02:57during a business outing, and the kid's ramen on the receipt gave her away, right? And so you have to go and
00:03:03read all the line items and complicated receipts, and you have to dig in and understand, like, what is the
00:03:07purpose? Who are these meals for? There's a lot of reasoning the agent needs to do. Or in this case,
00:03:13John really enjoyed the luxury of Uber Black, but he kept upgrading. And over time, somebody said, you know, we've had a
00:03:22pattern of abuse, not just an individual transaction, but many, and are able to recommend, like, hey,
00:03:27please, you know, use UberX or a lower price card share. So this is all sorts of creative out-of-policy,
00:03:37creative abuse of the corporate spend policy. And so to catch this creative abuse, we needed a creative
00:03:42agent, which means we had to give it the tools and the skills to go connect the dots and understand what's
00:03:49going on. So we gather up all of our tools, and we take a first naive attempt at writing a tool calling
00:03:58agent. And you wind up with a trace that looks sort of like this. Each of these columns represents a turn of
00:04:05the agent. The pink output there is all the tool response, loads of JSON information, trip output,
00:04:13policy information. And each one of these is cumulative. So by the time you get to the end of
00:04:17this graph, you've bloated your context window, mostly with tool responses, most of the information you
00:04:23don't actually need. So when you're talking with an agent's very verbose in this mode, and when you're
00:04:32talking with somebody who's very verbose and they're slow to get the job done, we as programmers have
00:04:37long known what to tell those folks. We basically say, "Go away or I'll replace you with a very small
00:04:43shell script." And that's exactly what we can do with our agents. The truth is that shell scripts are
00:04:51a shockingly effective way to reduce the verbosity and complexity of getting a task done. And as all of
00:04:58us work with coding agents, we know that if we write little programs, we can save ourselves a lot of
00:05:03work in intermediary steps that can reduce all that complexity that takes place in that very many
00:05:09turns with chatty tool responses. Every agent is becoming a coding agent, right? All of these models
00:05:18are tuned for coding. That's the first and best use case. And everything else is just gravy. So we want to
00:05:23work with that. Cut with the grain, not against it. And so we can turn all of our agents into coding agents
00:05:30and give them the same creativity they use when you build your apps. So Just Bash is an open source
00:05:37library provided by Vercel that is perfect for this use case. It's a totally sandboxed WASM execution
00:05:45environment that provides Bash, custom standard tools like GRAP and CAT, and is programmable from TypeScript.
00:05:55Because it's fully sandboxed in this way, sailed right through security review. Didn't need a complex
00:06:01operational setup with containers or our Docker deployments. Basically just an NPM library.
00:06:08And when we gave Bash to our agent, the results were immediately dramatic. For example, consider we're
00:06:19auditing the spend of a typical mid-sized company that spends $2.5 million every month, right? That's
00:06:28over 9,000 expenses we need to audit individually and in clusters. And at that 5% out of policy rate,
00:06:36you'd expect around 475 cases to be captured. This represents $300,000 on average of exposure risk to
00:06:45a company out of policy spend in potential. With a traditional tool calling agent, this would take
00:06:53200,000 direct tool calls and consumed over 6 billion LLM tokens to conduct this audit. That's not an
00:07:02economically viable product for us at Brex at that scale. You're spending more money to audit it than
00:07:08you would have saved. So we needed a way to bring these numbers way down. And our very first attempt,
00:07:15just handing Bash to our agents, produced dramatic reduction. 60,000 Bash invocations instead of 200,000
00:07:24tool calls. A 75% reduction in tool calls and a commiserate reduction in token usage. This is wild,
00:07:31and there's so much more room to bring these numbers down, and we have done so since. So look at that
00:07:37first graph that I showed you earlier on top and the graph below. When you use Bash, Bash gives the agent
00:07:44control over all that tool output so it can bring down all those verbose responses and have a much less
00:07:51chatty interaction to do the same job. In this case, the cumulative number of tokens used far reduced.
00:07:59It also makes the agent faster. But the critical thing is Bash lets agents manage their own context
00:08:06instead of drowning in it. As a bonus, the tool design work that you need to do when you're building lots of
00:08:15different tools, for example, a simple MCP setup, you have to do X times Y work. You've got along the X axis
00:08:24here, various domain objects we need to interact with. And along the Y axis, we have various tasks we want to
00:08:30complete. Fetching, filtering, grouping, sorting. And we really don't want to do X times Y work. We actually
00:08:39want to do X plus Y work. And this also means that there's far fewer tools. You don't have the product
00:08:45of all these things in your tool context. And this is what the Unix philosophy gives us.
00:08:53The Unix philosophy says to write programs that do one thing well, work together, and handle
00:08:59text streams. It's the universal interface after all. And this was a great idea in the 70s. And it's
00:09:05still a great idea today. And text streams are an even more universal interface today than they ever
00:09:10were in the past, thanks to modern agents and LLMs. And so by working with this sort of same tools and
00:09:19leaning into this same philosophy, we can do less work, have less verbose interactions, and get better results.
00:09:27So here's some examples. We have a simple tool, get expenses, that just dumps JSON of all the
00:09:33expenses that are relevant to a particular investigation. You can see we pipe into JQ,
00:09:38which is built into the Just Bash package, standard off-the-shelf JSON querying tool.
00:09:44There's many, many, many fields, several hundred at this point, of information associated with expenses.
00:09:50But I only really care about the ID, the merchant, the amount, the type in this context.
00:09:55So none of that JSON winds up in my tool response, and therefore none of that JSON counts as input
00:10:01tokens to subsequent turns. Additionally, I can save the work of having to make a tool call multiple
00:10:10times. If I'm going to get the most recent expense or the most expensive meal, I can combine these tools
00:10:21through the file system. I pipe the JSON response to the file system, which of course, none of that
00:10:28becomes input tokens, and then I can run operations on that temporary working memory twice using the
00:10:35expenses JSON file, bypassing the context. Additionally, my system prompt gets smaller.
00:10:43These expense policies are complicated. They're big. They often come in PDF format,
00:10:48and they can be tens or sometimes as much as a hundred pages of full detail of every little
00:10:52aspect of a business. And loading that into the context window every time is prohibitive. So instead,
00:10:59we can seed the file system with those policies and many other documents. And the agent can use
00:11:05standard tools like grep to find out the first-class flight policy for an organization. This is a simple
00:11:13example, but just this basic level here is what got us those improvements before, and there's a lot of things
00:11:19you can do to index the information, add additional tools to explore it, provide advice on how to get just the
00:11:26context you need, not all of it all at once.
00:11:32So let's see how you define tools that you can add to extend like our get expenses tool there. It's a very simple
00:11:40process. You import just bash. You call it define command. And then you get args and a context object. The args are standard
00:11:46command line objects, just like you'd have on any command line tool. You return standard out, standard error, exit code.
00:11:52And then you invoke the bash environment and give it a set of tools.
00:11:56And you get all the standard ones in there for free. You can customize as well with the parameters.
00:12:02The same tool design that you would use to make great Unix tools apply here.
00:12:09They make great agent tools. There's 50 years of tradition that you can call upon to make a great
00:12:14developer experience that also leads to a great agent experience.
00:12:18And the system prompt design couldn't be more straightforward. You say, use the bash tool for
00:12:22all audit investigations. Call get expenses to analyze transactions. You can give it some information
00:12:28about some, but not necessarily all of the options your tools have. You know, here we have user and trip to
00:12:34do some pre-filtering, but you can pass dash dash help if you need more information. Online inline help is great for developers.
00:12:42And this enables your agents for the same reason to explore and uncover and find the way to operate these tools just in time.
00:12:52So here's an example. We call get trips. We notice that it passes dash this city because it wants
00:12:57to know which trips went to that city. And whoops, that flag's not provided. No big deal.
00:13:01The agent encounters that standard error output, says, I can deal with that. I can solve this problem on my own.
00:13:07I'll use JQ to do some filtering. It might be a little slower. It might send a little more data across the network,
00:13:12but it's not going to go into my context window and I can get the job done. And then we proceed to,
00:13:17you know, look at policy and expenses and so forth. This is a hallucinated request. This is a hallucinated
00:13:26command line flag. Failures and hallucinations are feature requests. It's your agent telling you how it
00:13:33wants to use your tools. You don't get this experience as often when you work with something like MCP
00:13:39because the agent isn't building on all of that ingrained knowledge of how to use a shell, how to
00:13:44write code. And so you can look at this failure that was in line in the interaction in which the agent
00:13:53ultimately succeeded. Your customer still had a good experience, but that recovery taught you something
00:13:58about how to make the agent more efficient, faster, and more successful in the future.
00:14:02It's just like mentoring a new software engineer. You sit over their shoulder and go, hey, did you know
00:14:07you can pass that flag? It's the exact same skill set applied to agents. So now we go and we teach
00:14:14the get trips command dash dash city and we do with even fewer requests to solve the next query,
00:14:19get the same job done. So yeah, so I think the thing to understand here is that just as you're putting
00:14:31together all the components to build your system, you can essentially give a subset of those components
00:14:36as a block to your agent and the agent will tell you what you need to do to set free the right design.
00:14:46It may be a hallucinated fact. I think it was Michelangelo, but there's a famous
00:14:51sculptor whose name eludes me, who said that like the sculpture was in the marble.
00:14:56And I think your agent's in the marble as well. You take that big block of stuff that is all your tools
00:15:01and whatever agent, you give it a task, you watch it explore, you watch it figure out how to solve
00:15:06the problem. It's your job as an agent designer to set that agent free from all the raw materials you
00:15:14already have. Luckily, software and LLMs are a little bit more malleable than stone. So if you make a mistake,
00:15:21you can correct. And I think the thought that I'd like to leave you with is that if your agent needs
00:15:30to solve challenging problems and exhibit creativity, just like our audit agent finding creative
00:15:39out-of-policy expenses has to do, then just maybe your agent should also be a bash agent. Thank you.