Ship 26 NYC - 1,200 analytics requests a day: Clay and Vercel's Bet on AI-Native Analytics
VVercel
Computing/SoftwareSmall Business/StartupsManagement
Transcript
00:00:00Hello, New York. Thanks for being here. I thought I would mix things up a little bit and talk about
00:00:14agents for a change, if that's okay with everyone. So it's obviously a very hot topic for us at
00:00:19Vercel. And earlier today, you heard Tomo talk about EVE, our framework for building agents.
00:00:26From Gene, you heard about how we have 100 agents in production. And you also heard about the most
00:00:34popular of those agents, the one nearest and dearest to my heart, D0, our data science agent.
00:00:41Now, Vercel has about 800 employees here at Vercel. And D0 answers over 30,000 questions every month for
00:00:52those employees. And I love that stat. I say it all the time. But it's not my favorite stat.
00:00:59This is. 45% of those questions aren't from humans. They're from the enterprise apps and agents you heard
00:01:08Gene talk about earlier today. Now, D0 has been so successful for us over the past few months that
00:01:12we've taken the show on the road. We've gone to some of the most inspirational companies we know. We've taken
00:01:18Eve. We've taken our learnings from D0 and helped them add their context, their creativity, their tools
00:01:24to build data agents of their own. And now, we have a lot to learn from them.
00:01:32One of those companies is New York City's very own Clay. Clay built Monty, their data agent. Monty is in
00:01:40production today. And it saves the Clay data team 40 human hours every day. And they built it with one
00:01:47engineer in under a month.
00:01:51I have one goal for today. I want to see a thousand new data agents in the wild. We built one. Clay built one. You can too. And you should. I want every company in this room to build your own D0.
00:02:04And transparently, this is also a little selfish. I want D0 to be the best it can possibly be. And I'm convinced that the way that happens is by watching to see what you build.
00:02:12So, I'm going to do two things today. One, I'm going to show you a little bit high level about how we build these agents. What's common across everyone that we've built, everyone that we've seen, what's different.
00:02:22And second, why you should be building these on your own and not buying them off the shelf. So, let's get started.
00:02:29So, over the past few months, as we've helped build, as we've watched dozens of these data agents take flight, across all of them, we've seen a common shape. A common shape of what we think of as a minimum lovable data agent.
00:02:40In particular, we see these five things.
00:02:44First, first, minimum lovable data agents meet users where they are. That means they need to be available in messaging tools. For us, that's Slack. For other people, that's Microsoft Teams.
00:02:56But not just those tools. When users ask sensitive questions that can't be answered publicly, D0 can't just answer in Slack, it needs a web UI.
00:03:06When users want to create and interact with charts and dashboards, they likewise need a web UI.
00:03:11When someone wants to call a data agent from cloud code or codex like they do every day, they need an MCP server. When they want to call it from an application, they need an HTTP API. And when other agents, as you heard increasingly for us, are calling our data agents, they need an agent-to-agent interface.
00:03:27Truly lovable data agents need to support all of these services.
00:03:32Two, lovable data agents need to navigate rich context. This is the whole game. They need to understand your business. They need to understand what your metrics are, what your terms mean, how your company works. And they need to pull all that context in at the right time into the right workflows.
00:03:51Three, lovable data agents need to query data. This is the obvious one, but it's not that simple because to really be able to answer any question, data agents need to be able to connect to any of your data sources.
00:04:07Not just the biggest one, not just a singular data warehouse. And to be able to take any action, data agents need to be able to connect to all of your systems of action.
00:04:17And all of those connections have to be secure. The agent can never leak your credentials, not even to the LLM, and every connection has to respect permissions.
00:04:27Your agent should only be able to see and do what its human counterpart can see and do.
00:04:32Four, lovable data agents need to follow analytical playbooks. Across these 30,000 questions a month, we find that most questions are non-novel. They have a pattern, they repeat, they can be templatized, and the agent needs to know what those patterns are and follow them.
00:04:50Five, lovable agents need to execute recurring work. Again, across those 30,000 questions, we see that the most important questions are not one-offs, they are recurring.
00:05:01Every Monday, your agent needs to pull key metrics. Every day, your agent needs to check customer health and flag warnings. Every month, your agent needs to analyze marketing ROI.
00:05:11True, lovable agents have routines. Now, for both DZero and Monty, it was actually easy to express these capabilities because we built on Eve.
00:05:25We designed Eve for agents like this very explicitly. Eve treats agents as a file system, and the structure of that file system means that every capability, these five capabilities we talked about,
00:05:37has a natural home with all the functionality we need for that capability to be complete.
00:05:43When we needed DZero to support multiple surfaces, we just declared those surfaces as first-class Eve channels.
00:05:55Now, it's the exact same DZero in Slack, in a web UI, behind an HTTP API, and as an MCP server. It's one core with many surfaces.
00:06:03When we needed DZero to securely connect to systems of data and systems of action, we just drop new tools in the tools folder.
00:06:11Execute Snowflake query, execute ClickHouse query, update Salesforce.
00:06:15The model backing the agent only ever sees the tool, never the credentials, and because we wire those tools to OAuth with Vercel Connect, every query assumes the roles and permissions of its human user.
00:06:29When we needed DZero to follow playbooks, we just wrote them up as skills, and we dropped them in the skills folder.
00:06:34And because skills are just English, anyone can author playbooks, either as dynamic skills just for themselves, or global skills available to everyone in the company.
00:06:43And when we needed DZero to do recurring work, we just wrote another plain English prompt.
00:06:49We add a cron condition, we drop it in the schedules folder, and that prompt gets invoked on a schedule, and DZero responds as if it were invoked live.
00:06:58But what about context?
00:07:00Now, the simple answer is that context also has a natural Home and Eve, and it's right there in that sandbox folder.
00:07:07For us, context, once it's in the sandbox, becomes a part of DZero's navigable file system, so DZero can now use Bash like a coding agent, or any structured retrieval tools that we provide it.
00:07:19But the more interesting answer is how we think about context, because again, for these agents in particular, and agents writ large, context is ultimately all that matters.
00:07:29For us, we went the route of being highly structured.
00:07:34We have over a dozen types of very structured contexts, most of which are represented in YAML, and five of those are the most important.
00:07:41One, our semantic layer.
00:07:43This is essentially a catalog of every single table in our warehouse, the fields it contains, the business logic that can be derived, the valid aggregations that an agent can perform.
00:07:53And again, YAML.
00:07:55Two, our metrics layer.
00:07:57This layer takes the aggregations we define in the semantic layer and reifies actual business metrics.
00:08:03Three, our events layer.
00:08:05Every day, we scrape our change log, we scrape our sales plays, our marketing campaigns, our outages, any data pipeline alerts, and we create a structured event stream.
00:08:15Four, our terminology layer.
00:08:17These are our glossary terms.
00:08:18They express how Vercel talks about Vercel, include aliases and synonyms and related concepts.
00:08:24And five, our guides layer.
00:08:26These are plain English essays written by data experts and domain experts to teach both humans and agents how Vercel works.
00:08:34This is very rich context, and it's also very structured context, which means, in our case, because again, a lot of these are YAML, we can give our agent structured retrieval tools and not just bash.
00:08:49Bash takes you very far.
00:08:50It's not all you need.
00:08:52Now, this is a lot of work, but we think it's the key to D0 success.
00:08:57Now, next up, I want to give you a different perspective.
00:09:01I'm going to bring up our friends at Clay.
00:09:03Their data agent, Monty, was inspired by D0.
00:09:07It's built on Eve, and it works very similarly, but it's also genuinely different.
00:09:12Monty is Monty, not a D0 clone.
00:09:15So joining me now is the lead of Clay's data team, Josh Hansen.
00:09:20Thanks, Abhi.
00:09:29We've had a lot of fun building Monty, and I'm excited to share what we've learned.
00:09:33I'm going to pick up where Abhi left off and walk through how Monty works and the unique decisions we made in building it.
00:09:39I want to start where Abhi ended, specifically with navigating context.
00:09:44Abhi mentioned that D0 has a rich, structured context layer.
00:09:48We took a slightly different approach, and this was the biggest design bet we made.
00:09:53At the start of every session, Monty clones our entire data team repo into its file system.
00:09:59That repo is the brain.
00:10:01DBT models, the semantic layer, metric definitions, team skills, dashboards, notebook analyses.
00:10:07Years of how the Clay data team thinks about Clay's data.
00:10:11This is a different approach, and we picked it because we were trying to get away from the classic semantic layer pattern.
00:10:17Semantic layers promise to tell the LLM exactly which tables it can query, by which measures, across which segments.
00:10:24They promise governance, but the reality is that you've taken an incredibly powerful reasoning engine and reduced it to an expensive SQL generator.
00:10:32We bet the other way.
00:10:34Modern models can reason across almost any problem if you give them the right context.
00:10:39Instead of a fixed menu of allowed cuts, we give Monty all the raw context it could possibly need, and we trust it to pick the right tools for the question.
00:10:48Here's how that works in practice.
00:10:50Monty has a navigation map in its system prompt.
00:10:54Every folder in our repo has a tiny catalog, a one-line index of what's inside.
00:10:59Monty scans that catalog, finds the one or two entries that matter, and opens only those.
00:11:05The big three Monty leans on most are, first, skills, which are the team playbooks with triggers and recipes and gotchas.
00:11:13Two, plain English model documentation covering about 100 governed models.
00:11:19And three, our raw DBT SQL, which Monty only opens when it really needs to see how a column gets computed.
00:11:26Three levels of progressive disclosure applied across an entire repo.
00:11:31But here's the thing I really want to plan in your head, because I think this is the bigger story.
00:11:37Everything analytical at Clay lives as plain text in GitHub.
00:11:41Transformations are SQL, column docs are YAML, metrics, skills, team context, they're all marked down.
00:11:48Dashboards and notebooks, all Python.
00:11:50When an analytics engineer ships a model, Monty knows about it on the next session.
00:11:55When a data scientist pushes a new notebook, same thing.
00:11:58When someone codifies a new metric, same thing.
00:12:02Nobody on our team maintains an AI knowledge base.
00:12:05Our code base is the AI knowledge base.
00:12:08The work the data team is already doing is the work that's making Monty smarter.
00:12:13Okay.
00:12:14So, that's how Monty knows where to look.
00:12:17Let me show you what it does with what it finds.
00:12:21Monty's toolkit is intentionally small.
00:12:24Monty has read-only access to Snowflake, locked down, row-capped, every query tagged for audit.
00:12:31It also has a Python sandbox where it can run pandas, stats models, sky kit learn, fits models,
00:12:38plots things, push charts back to Slack.
00:12:41It also has access to the Clay API, which allows us to connect and manage our Clay instance.
00:12:46And it also has what we call an analytical constitution.
00:12:50Guardrails written into the prompt.
00:12:53Things like, correlation isn't causation.
00:12:56Always pair rates with its absolute count.
00:12:59Flag small samples.
00:13:01This is all the stuff that kind of keeps Monty honest.
00:13:04This toolkit plays off in two situations in particular.
00:13:08The first is a metric lookup that doesn't dead end.
00:13:12When someone in Slack asks, what's our X metric and how is it trending?
00:13:17And immediately wants to know why?
00:13:19Which segments?
00:13:20Which workspaces?
00:13:21Which users?
00:13:22Monty can answer the metric question from the metric layer.
00:13:25Then walk down our entire dbt lineage to the workspace grain facts and re-answer from there.
00:13:32The second is repeatable work.
00:13:34That looks ad hoc, but isn't.
00:13:37Think renewal prep deep dives, account health reviews, pipeline reviews.
00:13:42They all have a shape.
00:13:44They can all be templated.
00:13:46We codify that shape as a skill.
00:13:48And Monty runs that workflow at the data team's quality bar.
00:13:52I want to show you the second one because I think it's the most illustrative.
00:13:57This is a real Slack thread, lightly scrubbed.
00:14:00It hit our channel at 5:00 a.m. one morning.
00:14:03Someone said, "I believe customer X is coming up for a renewal.
00:14:07Can you do an account health deep dive?"
00:14:09And Monty's first words back were, "Let me start by checking for the right skill to follow."
00:14:15And then it goes, end to end.
00:14:1815 tool calls, six governed queries, one matplotlib chart, zero errors, two minutes, 12 seconds.
00:14:26What it actually did, it read our skill catalog, matched the EGS account health skill, read the skill's two reference docs, one for gotchas, one for our pricing cutover, pulled the account from Snowflake, worked through the playbook's models, built a burndown chart in matplotlib, and posted it all back to the thread.
00:14:45One skill rooted the entire thing.
00:14:48But here's the part I really want you to see.
00:14:51The skill we wrote bakes in a warning.
00:14:53It literally says, "Don't let healthy pacing fool you."
00:14:57And Monty caught it.
00:14:59The headline numbers on this account looked great.
00:15:01They burned 86% of their credits within 91% of their contract term elapsed, right on pace.
00:15:09That lands the credit pool right at renewal.
00:15:12But the skill told Monty to keep digging.
00:15:15So Monty pulled the rolling 30-day consumption and it had dropped.
00:15:20The March peak was 1.5 million credits spent a day.
00:15:24Today it was 220,000, nearly down 33%.
00:15:30Then Monty looked at who was actually using the account.
00:15:34Five logged in users in the last 30 days.
00:15:37Two builders driving most of the spend.
00:15:40And then it did one more check.
00:15:42The account numbers one historical user, the person who originally built the use case, hadn't logged in for 188 days.
00:15:49So here's what was actually happening on this account.
00:15:53A rosy number was hiding a softening account.
00:15:56The skill warned about it.
00:15:57Monty caught it.
00:15:58And the CSN had the right read at 5:00 a.m.
00:16:02All before anyone's first meeting.
00:16:04And this is exactly what we mean by giving everyone the analyst playbook.
00:16:08We write the playbook down once and Monty runs it the way an analyst would.
00:16:13That's a small peek at how Monty works.
00:16:16We can't wait for it to transform Clay.
00:16:19In a lot of ways, it already has.
00:16:22And back to you, Avi.
00:16:33Thanks, Josh.
00:16:34Okay.
00:16:35So two data agents, same framework, Eve.
00:16:38Two very different companies, two very different agents.
00:16:41And that gets us to the second thing I promised.
00:16:43Why you should build your own and not buy these agents off the shelf.
00:16:48Here's how I think about it.
00:16:50Both data agents you saw today connect to four kinds of systems.
00:16:53First, systems of data.
00:16:55This is where your data actually lives.
00:16:57Second, systems of context.
00:16:59These are the systems that organize the architecture of context around that data that give it meaning.
00:17:04Third, systems of action.
00:17:06These are other business systems where the agents go to do work.
00:17:09And fourth, systems of interaction where the agents go to meet people.
00:17:14The punchline is no vendor, I promise, can ship all four for you.
00:17:19And that's because every one of these systems is different at every company.
00:17:22Your data stack is yours.
00:17:24Your context layer is yours.
00:17:27Your action systems are yours.
00:17:29Your interaction systems are yours.
00:17:30And off-the-shelf agents can connect to some of these.
00:17:34But they will never, never ship the specific combination of these four systems that your company actually runs on.
00:17:42And they'll never be able to express your context the way your company needs and is deemed optimal.
00:17:49Agents as a service providers can ship an agent, but they can't ship your agent.
00:17:55Now, again, I started this talk by saying I wanted to see you all build the next 1,000 agents.
00:18:01And I meant it.
00:18:02D0 has changed how Vercel operates.
00:18:05Full stop.
00:18:06Monty is changing how Clay operates.
00:18:08And we're watching a dozen more being built right now.
00:18:11Every one of them is unique.
00:18:12Every one of them is theirs.
00:18:14And we hope you built yours.
00:18:16And we want D0 to learn from yours.
00:18:18If I can be helpful, please reach out.
00:18:20Thank you.