How We Solved Agent Building — Andrew Qu, Vercel
AAI Engineer
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Hey everyone, thanks for coming. I'm Andrew, I'm the Chief of Software at Vercel, and I'm here to
00:00:20talk to you about how we solved agent building at Vercel. I'm the Chief of Software, so I work on
00:00:26a mix of internal engineering, external experimentation, and generally being at the
00:00:30frontier and building new libraries, frameworks, and technologies. For those of you that don't
00:00:36know Vercel, Vercel builds agentic infrastructure so people can build what's next. We got started
00:00:42in the web world, helping people ship websites and web apps without having to worry about the
00:00:47infrastructure that doesn't make their app any better. It can scale to a million and scale down to
00:00:50zero effortlessly. But we're seeing a change in what people want to build. People started by
00:00:56building pages, but now we see them wanting to build agents, and we've been embarking on a similar
00:01:01journey to make it easy for people to build agents and agentic applications easier. We built this
00:01:08thing called the AISCK, so instead of needing to switch out 300, 400 lines of provider-specific code,
00:01:13you can switch out one line of code, and we have the same model interface underlyingly for all these
00:01:19different providers. We built a lot of other tools to make it easier to have model fallbacks, secure
00:01:25code execution, better pricing when it's inactive and waiting for responses, as well as for durability and
00:01:30resumability. And I'm here to talk to you about how I went on this crazy experiment roughly a year ago that led to a
00:01:38an agentic explosion at Vercel and led to a really cool thing that we built recently.
00:01:46About, this is 1980, Bill Gates, before my time, had this quote saying he imagined there would be a computer
00:01:53on every desk and in every home. You know, that was probably pretty contrarian then, and today it seems like very
00:01:59normal to have that happen. And me and the CTO had this thought, you know, instead of a computer on every desk, could we
00:02:07potentially have an agent on every desk? You know, today we only really use agents for coding and technical
00:02:15workloads, but we're starting to see expansion into things like design and product management and other
00:02:20verticals. And this was maybe about a year ago, so I would say I'm pretty early to this. But that was when it was like
00:02:27Sonnet 4 and things weren't as sophisticated as they were today. And I tried to actually explore this out, see what we could do
00:02:33about it. I went around to various job functions at Vercel. Marketing, sales, finance, legal. And I asked them, what do you hate
00:02:42most about your job? And the most compelling use case I heard was at the data team. They were growing, they were a very lean
00:02:50team, but Vercel was growing faster. You know, they had so much more data from customers, analytics, metrics, sales. They just had to keep on aggregating and keep on making available for themselves.
00:03:02to use. And at this time, if you think about what the data science people have to do, whenever someone from marketing or sales has a
00:03:11question about a customer or a product, the data science team has to drop everything they're doing, write the query, process it, do an analysis, and come back with some
00:03:19recommendation on what to do. And this was really killer to productivity. You know, the data team did not want to drop everything and just write queries all day. And so I worked with our VP of data to try to build a better way to do.
00:03:31try to build a better way for them to operate this way. And so if you think about the very first thing you would ever do if you want to try to use AI
00:03:40to solve a problem, you may just build like a huge mega prompt. You know, you just have a question, you pass into an LM, you have it respond, and that's it. You know, this was how the first version really looked, honestly.
00:03:52I asked them for a dump of the Snowflake schema. I pasted it into a system prompt with a question. And then when it generated SQL, I actually
00:04:01copy and pasted that in and just ran it myself. You know, I just want to see are the models good enough today in order to write valid SQL given some decent
00:04:09structure. And I would say this gave us a little bit of confidence that, you know, models today aren't that good, but maybe we can harness engineer or make the context around it a little better and give it some more guardrails to operate a little better.
00:04:21And so if you actually think about what a data scientist actually needs to do when they get a question, you know, they have to process the question.
00:04:28They may have to explore the semantic layer and actually figure out what the join patterns are. They will actually go and execute the SQL.
00:04:35They may go back and do that again if the SQL did not execute or if it was too expensive. And they'll eventually report on it, including visualize the
00:04:42data, maybe write some paragraphs, maybe do a retro, maybe do some other stuff. And so if you think about those different phases, me and the VP of data
00:04:50tried to sit down and map those out into specific agent workloads. And so the second version of this data science
00:04:57agent called D0, I'm going to reference D0 from now on, is you ask a question, we have a query agent that passes on a query to the
00:05:05planning agent that will then have an execution agent, et cetera. And if you chain all of these together, you actually get
00:05:11something that looks like this, where each agent has a very dedicated system prompt focused to what that does with tools scoped to exactly that function.
00:05:20So an example here, you can see that for the first one, the planning agent has a read entity YAML and a search schemas tool. And so it will only use those
00:05:31capabilities until it has an answer to pass on to the planning agent and then to the SQL agent and then to reporting. And this was getting
00:05:38better. You know, we were able to get away from having to copy and paste a SQL and have to come back and report
00:05:44on it. It was now actually doing like the end to end loop from question to answer. But we started hitting some walls with this
00:05:54architecture. And around this time we came to the conclusion that, you know, what you actually need is you need one
00:06:01agent with all the mega context within it and for it to sort of manage its own memory. You know, this was around the
00:06:07time when we realized that you want to actually have the agent be able to look back on what it's done, sort of reflect and
00:06:13figure out the steps that got to get here. And with the previous model, you may have noticed that the only thing that the
00:06:19next agent gets is a summary and a small snippet of the previous thing that was done. Now, this way you can
00:06:25imagine that you have one mega agent and internally it manages its own state. At some points it's planning,
00:06:30some points it's building, some points it's executing, and some points it's reporting. And this is sort of what it
00:06:36looked like. You know, you have one big AI call, maybe max steps 100, and you give it the ability to manage its own
00:06:43state based on where it's at inside of its execution journey. And so you can see it's similar to what
00:06:48tools, you can see a similar shape. But the best part about this is if it ever ran to an error when executing or
00:06:54joining, it could go back and explore more, or it could go and read more and figure out what it was doing wrong.
00:07:00And it was very good at this point. We were pretty confident in the actual system at hand, and we actually
00:07:06spread it to a few trusted members of ourself. You know, this was a very powerful tool and we didn't really want to put in the
00:07:11hands of the wrong people or people that were using very critical workloads. So we got into a few people's hands and the immediate
00:07:17response was it was awful. You know, we thought we were cooking. We thought this was, you know, nailing 30% of our evals.
00:07:23But we couldn't have anticipated some of the questions that were being asked. And for us to spend more time
00:07:28manually mapping out some of these scenarios, it didn't seem like a very scalable way to do this.
00:07:33And then Claude Code and Opus 4.5 came out, or more like Opus 4.5 came out, and it in tangent with Claude Code was just so powerful.
00:07:44You know, they sort of unlocked the concept of a file system agent. And we, on the side, were like, wow, Claude Code and Opus 4.5 is basically AGI compared to what we had before.
00:07:56You know, it would answer most of our questions without even missing a beat compared to the hand-grown agent we had.
00:08:04And when we tried to step back and wonder what we were doing wrong and why this was so much better, we realized that the big unlock was that it was just a file system.
00:08:13You know, we -- it had a very minimal set of tools, list file, read file, run bash, and we gave a few more here for our own data agent use case.
00:08:24But the biggest thing was it was able to use the tools that agents are well-trained on and was able to explore and write work where it needs to.
00:08:32You know, we weren't giving it -- Claude Code was not giving it a very prescriptive set of tools.
00:08:37It was sort of just letting it go wild and explore emergent behavior.
00:08:41And so, from this, we learned that you can really just use the file system.
00:08:44You know, we saw the learnings from Claude Code and how powerful it was given that it just executes locally.
00:08:50And we tried to rebuild it in a way that was very Claude Code-esque.
00:08:54You know, it was now going to run in a sandbox.
00:08:57That sandbox would dump the whole semantic layer into it.
00:09:00You could -- the agent would be able to grab, bash, read file, write file all around to figure out what it needs.
00:09:05And we would just sprinkle a few tools on top to make sure it could do everything that is Vercel specific.
00:09:10And this was actually the biggest unlock ever.
00:09:14You know, the leap from single agent to Claude Code SDK and then from Claude Code SDK to file system agent in general, fine-tuned or purpose-built for our use case, was an amazing leap.
00:09:27At this point, we were starting to get ready to give it away to more people at Vercel.
00:09:31And at this point, the eval score basically doubled.
00:09:36And I wrote this -- this is basically how it looks.
00:09:39It's very simple.
00:09:40You just give it a bash tool.
00:09:41We have a nice helper called bash tool on NPM.
00:09:44And you attach it to a sandbox.
00:09:45And you can attach files to the sandbox for it to read, write, and execute.
00:09:50And after this revelation, and after I saw that we were passing so many of the questions that we failed to do before, I wrote this banger blog post.
00:09:59It's actually up today.
00:10:00And the week that I wrote this, it was responsible for 70% of our Vercel.com traffic.
00:10:05So you know it's a banger.
00:10:07And after that, the next logical step was that we wanted to figure out the common use cases we had.
00:10:14So by then, we've already sort of let it leash on all of our cell.
00:10:18And we were getting thousands of queries a day from people wanting everything from customer metrics, sales metrics, number metrics, NPM downloads.
00:10:27And it turns out that a lot of these queries are actually the same in shape.
00:10:30You know, there's only so many ways you can do an aggregation.
00:10:33Only so many ways you can look up a product.
00:10:35Only so many ways you can do billing info.
00:10:37And so we actually have a recurring job that takes the most recent queries and tries to distill them into a skill.
00:10:43And right now, we have roughly 100 skills that do a mix of aggregation all the way through looking up specific data about certain people.
00:10:50And we found this very effective.
00:10:52Because if you think about every new agent run, it sort of just starts from nothing.
00:10:56You know, there's really no pre-established context besides, you know, the semantic layer and the system prompt.
00:11:01But with a skill, it already starts off with a lot of contextual knowledge that has otherwise already been done.
00:11:09And this is roughly how it looks.
00:11:11It's very similar to the previous one.
00:11:12But the inclusion of a skills folder is actually very powerful.
00:11:15We also built this tool at Vercel called Skills SH.
00:11:18It's the most popular way to find agent skills and run them yourself.
00:11:22And I'm saying all this because this journey is something that most of you may hit once in a while.
00:11:28Where you start from something simple and you gradually add complexity and you eventually hit a system in which you can ship to prod.
00:11:34And I'm telling you this because at every step along building this agent, someone at Vercel was agent curious.
00:11:42And they tried to fork off of my D0 agent and build their own.
00:11:46And at every step, we sort of had a better way to do something that was not previously known.
00:11:50And we were wondering like what if people today could start from the very last insight and not have to ever start from just a simple prompt or from reinventing best principles from first principles.
00:12:04And so we actually thought what if we built the Next.js for agents.
00:12:09For those that don't know, Next.js is a popular web framework that Vercel built that invented this thing of file system framework defined infrastructure.
00:12:18You don't have to worry about where things go.
00:12:20You just have to write files in the right conventions.
00:12:23And it automatically declares where they should go.
00:12:26Your pages go to the CDN.
00:12:27Your serverless functions go there.
00:12:29Your caching goes in the middle.
00:12:31And we thought, you know, building agents should be this simple.
00:12:34You should only have to create a skills folder, a tools folder, a channels folder.
00:12:38And you should be able to just declare these very easily.
00:12:40And the framework should know exactly how to make an agent out of it.
00:12:44And that's why two weeks ago we released Eve.
00:12:47Eve is an agent framework like the Next.js for agents where it's very easy from just starting with a sample template to having a fully agent ready and being able to add in your own custom knowledge, your own custom tools,
00:12:59and even integrate it into the channels that you are familiar with.
00:13:03This is roughly what we think an agent actually looks like.
00:13:06You know, an agent has a runtime and it has channels.
00:13:09And in that runtime, you're going to have durability.
00:13:11You're going to want to run things in an isolated environment.
00:13:13You're going to want to call into different models.
00:13:15And you're going to want to have connections.
00:13:17And we built this with open source in mind.
00:13:19You know, we built Eve so you can plug in your own open source adapters for Postgres, OpenAI's responses API, Docker, other connectors.
00:13:29But we also made it incredibly easy to deploy in Vercel.
00:13:31The only thing here you see different is that everything here is using a Vercel product that we've been building over the years in order to make it easy to build these experiences.
00:13:39Vercel workflows for durability, Sandbox for secure execution, and Vercel Connect, something we just released to make it easy to generate short-lived OADC tokens for connections.
00:13:51And we actually rewrote the whole D0 agent in Eve as we were building Eve.
00:13:55And from the convoluted structures behind the scenes that you did not see from the code, this is roughly how the file system looks.
00:14:01It's very simple.
00:14:02You have a bunch of system instructions, a couple skills, a couple tools, and it's very easy to compose this into a real agent.
00:14:09And it's very easy to iterate on.
00:14:11We actually gave this out to a few beta customers before we actually fully released it two weeks ago at our London event.
00:14:17And this one company that partners closely with us, Aura, they've rebuilt their agent that's sort of like a mini-claw to go and test people's services.
00:14:26It goes to websites, installs them, it tries to use them.
00:14:29And they've seen incredible success on building their own agent from the ground up using Eve compared to using an off-the-shelf cloud code.
00:14:38Fewer steps, better successes, as well as better insights.
00:14:42And when you deploy Eve to Vercel, you get observability out of the box.
00:14:49You can see here that you get all the agent runs, you see all the tool calls, you see each step it takes, as well as maybe some estimated costs and some optimizations you could potentially take.
00:15:00And you can get started today at Eve.dev.
00:15:02You can just clone it and you can just start a template, deploy easily, self-host if you need.
00:15:07And the reason why I bring this up is because I hope that there will be more and more business-specific use case agents.
00:15:14You know, before we built D-Zero, we actually battle-tested a lot of the industry, well-funded startups that were doing these vertical agents.
00:15:23That were dedicated to taking your Snowflake instance and making it so their agent could run Snowflake queries against it.
00:15:30But we found out that what really makes this agent good is it has a lot of very specific company knowledge.
00:15:38You know, the way that Vercel is a web-based company, we have a lot of customers that have websites and web properties.
00:15:45That goes a lot deeper into when you should query for what and what things link to what.
00:15:51And so a lot of these off-the-shelf agents, they're great, they're good to try, but I think if you really want to get the most juice out of a squeeze,
00:15:58you should really try to build your own agent and add in as much company-specific knowledge as you can.
00:16:03Today, you know, we've had 20 roughly decently PMF agents at Vercel that range from anything from marketing retros to figure out who to reach out to,
00:16:14to the first ever red line of a contract when legal sees a new negotiation, all the way to my data science agent helping with data queries.
00:16:24And that goes to show that we at Vercel have been very agent-filled.
00:16:28You know, all of this stuff is actually saving us a lot of time.
00:16:32The data team has never been more productive.
00:16:34They have more time to go and improve the performance of Snowflake, to add new data sources that were missing,
00:16:40to fill in the gaps that they previously did not have time to because they were so busy writing queries.
00:16:46And I think it's never been easier for you at your big, small, medium-sized company to sort of automate away some of the things that you do not want to do
00:16:55or some of the things that you're spending too much time doing.
00:16:58You know, I think a lot of HR, finance, sales can be somewhat automated with agents, and I think Eve is the best way to build said agents today.
00:17:09And these are my socials.
00:17:11Thank you all for coming and listening.
00:17:13I'm Andrew, and I'll be around if you want to chat outside.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video