Ship 26 NYC - CTO panel: Internal apps and agents in production
VVercel
Computing/SoftwareManagementInternet Technology
Transcript
00:00:00I don't think I've ever had a panelist get a shout out during a walk-on.
00:00:09This is going to be lit! All right, thank you everybody for joining me on stage.
00:00:15We spend a lot of time talking about what AI can do for customers, and today I want to talk about
00:00:22what it's doing inside organizations. To the people building things, to the teams deploying them,
00:00:28and to the question of who exactly gets to build. So Ali and Melky just introduced all of you,
00:00:34but to reiterate, we got next to me Andy, CTO of MJHLA Sitances, Greg, who's the CTO of Serhant,
00:00:42and Victor, the VP of Product Management from Zoom Info. All right, so to kick us off, we've watched
00:00:49organizations move from experimenting with AI to actually building with it. Before we get into
00:00:54specifics, I'd love to start high level with what you all have learned. So what's something you're
00:00:58doing with AI today that you might have thought was unrealistic a year ago? Anyone can start.
00:01:06I can start. All right. This is near and dear to my heart, being in enterprise software for so long,
00:01:11which is we call it voice of the customer to pull requests. So something that I never thought was
00:01:18possible from last year was, Gene, you've been through this, where you have tough conversations with
00:01:23customers, they have feature requests, and the response is, we'll put it on our 12-month roadmap,
00:01:28and I feel like I'm letting them down. Now what we do is we take those recorded conversations, we have
00:01:34agents that analyze them, and generate code and pull requests for them, so that the product team's job is
00:01:40to decide when that gets shipped to production, not actually synthesizing all the feedback. So it's
00:01:46kind of an amazing time to be in enterprise software, I think. I love that. You just took
00:01:50Char's EDV demo and added like another major agent hop onto it to add that much more value. This is good
00:01:57to know. I've started addressing the AI agent on my customer service calls. It's like, if the AI agent is
00:02:02listening, here's the problem I'm having. So it's just going to encourage me to keep doing that. Nice.
00:02:08Yeah, I mean, I think for me, I think it's when I look at last year, the work you can give an AI agent,
00:02:13typically, I would say, will take them minutes to get done autonomously, right? Maybe hour, a little
00:02:20bit over that. Now you can actually tell an AI agent to do really complex tasks that will take them 10,
00:02:2715 hours, even overnight days, right? And so it really starts becoming where coders are becoming
00:02:34conductors, right? And you're now able to delegate work instead of having to sit there and kind of
00:02:39babysit the AI, and every few minutes, you have to guide it to do the next thing. You can truly delegate
00:02:45work, and I think that's just a multiplying force for any engineers where, you know, it's not replacing
00:02:50engineers, it's actually just empowering them to do a lot more. So it's really exciting to see.
00:02:55Awesome. I've been really surprised that I'm living without a BI tool for the first time in 15 years.
00:03:01Yeah. I hope my Salesforce team isn't here. But we turned pretty hard off of Tableau,
00:03:07and we replaced it with nothing. We replaced it with homegrown internal tooling. Somebody's going to
00:03:14write a great sub stack about this. I don't know who it is in the room, but it's like the rise of the
00:03:18custom control plane. I don't want to just see data on a screen. I want something that I can get the metric,
00:03:24take an action, and hand it off to an agent. And we've built internal control planes over some of
00:03:29our data that allow us to do that.
00:03:30Yeah. Awesome. We're the same way. V0 powers 100% of our BI. Most of it originates with like
00:03:37direct connectivity into Snowflake, like I mentioned, but we drop spreadsheets, you name it, into it.
00:03:43Same thing. I can't imagine having a BI tool. So one of the things I find interesting is who's actually
00:03:49doing the building you guys are describing. So Andy, you all have this AI accelerator program.
00:03:57Who's in it? And what are they building?
00:03:59Yeah. They say that the seat license is dead, but I just keep bringing more and more people
00:04:03on to Vercel and giving them monthly licenses. Our AI accelerator is designed for non-engineers.
00:04:09So 20% of the program is pure marketers, people that are early in their career. They have a marketing
00:04:16degree. Needless to say, they don't have a GitHub account. They've never thought about themselves as
00:04:20an engineer. And we give them like what we call the, you know, the Twitter meme about the starter pack.
00:04:25It's Cloud Code, Vercel account, GitHub, and SSO. So we give them a base template to work off of
00:04:33and an engineering buddy, and then we let them loose. It's babies with chainsaws building some of the wildest
00:04:39stuff that I've seen in my career. And it's been very, very like exciting to have the voice of the customer,
00:04:45the internal customer working with us. Awesome. And Greg, tell us how Serhan is building vertical AI
00:04:51for real estate. Yeah. You know, as my colleague Coy mentioned, we're building vertical AI for real
00:04:56estate. And what that means is, I think we try to obsess over how to create differentiated value for
00:05:04our customer who are real estate agents while leveraging the AI models, right? And so there's this
00:05:09balance you have to take, which is you want to leverage the model in such a way that as the
00:05:14model gets better, your system gets better. At the same time, you want to add value on top of it so
00:05:20that, you know, for our agents, our real estate agents, they will have a much better customer
00:05:25experience using Simple, for example, which is the application you saw earlier, as opposed to just
00:05:31going to Cloud, right? So like, how do you actually find that right balance is really important.
00:05:35And for us, it really comes down to, I think, like three key areas. One is we're building workflows that
00:05:42are tailored for real estate agents. So, and then an orchestration platform on top. So things like,
00:05:48how do you create a competitive market analysis report? How do you generate a listing presentation?
00:05:54These are all very real estate specific workflows. And I think the other part is the data, right? Like,
00:05:59having authoritative data that powers your AI, right? Whether that's all of the deal transactions
00:06:05that we're making in the brokerage or our sales training data that we give to our salespeople,
00:06:11those are really important as well. And then I would say just the last thing is we have real estate
00:06:15professionals that are continuing to provide feedback and fine tuning the output of the AI model so that
00:06:21it just continues to get smarter and smarter over time, right? So I think those are kind of the area that
00:06:25really make us more of a vertical AI versus just kind of like bolting on the chat bot, you know,
00:06:30into, into legacy software. Yeah. The data point is great transition to zoom info. So you all are
00:06:37building a product that embeds V zero while being able to expose the data, um, in order to build new
00:06:43applications for, for your customers. Um, what problems are you looking to solve that folks couldn't solve
00:06:49before? Yeah, I mean, V zero is amazing. Um, what we're able to do now is, as I think you mentioned
00:06:57this in one of your podcast appearances, which is the calculus of build versus buy. I'm actually more
00:07:02bullish on SAS now than ever before. But what I will say is a one size fits all SAS application is
00:07:10probably seeing its last days, but verticalized, customized to the user, to the use case. And so what
00:07:16we're doing with V zero is how can we take zoom info vertical data, whether it's real estate prospects,
00:07:23manufacturing prospects with the magic of V zeros coding agent, so that we can bring the builder
00:07:30mentality to people like who have a degree in marketing, who never thought they would build
00:07:34things, but put it on rails for them to make it easy to get value out of it. Yeah. Awesome. So let's
00:07:40talk about security and guardrails, uh, because I think it's where a lot of organizations get stuck
00:07:46when it comes to getting something off your local machine and actually deployed in production. Um,
00:07:51Victor, when, uh, you're embedding AI capabilities into a product used by thousands of enterprise
00:07:56customers, the blast radius of mistakes, uh, is obviously enormous. How does your team think about
00:08:02trust and safety at scale? I think first is, um, if we're able to put things on rails and make users,
00:08:10let's say whether you have a computer science degree or not, be able to trust that you've got the right
00:08:16data, you've got the right infrastructure is where they can be most creative about building.
00:08:20And the first thing I think of is we thought of who is the best in class at standing up deployments,
00:08:26sandboxes, shipping, and Vercel is that. And at the context graph layer on the zoom info side,
00:08:32we'll take care of the, um, privacy permissions. And so users only have access to the data that
00:08:39they're allowed to have access. So their agents aren't burning through hundreds of thousands of
00:08:44tokens, for example, but I think I'll put it all together, which is best in class infrastructure
00:08:49guardrails and then best in class context graph guardrails so that anyone from go to market engineers to
00:08:56marketing folks can feel free to build on their ideas. Nice. Um, and Andy, uh, so, so you've made
00:09:03this counterintuitive bet, uh, that constraining the stack actually accelerates innovation rather
00:09:09than slowing it down. You're even replacing vendor contracts because of what people built internally.
00:09:15Can you explain that thinking? Yeah. Uh, the shadow it challenge right now is real. Um,
00:09:21and the bet that we've made is by giving people a ton, a ton, ton, ton of access to build something
00:09:28on Vercel with preview links. Those, you know, you know, I talked to these, um, marketers and they
00:09:33don't really seem to understand how amazing ephemeral environments are. They just assume that that's how
00:09:37software works. And it's like, took a long time for us to get here as an industry where you ship your
00:09:42code and you get an environment where just your code is reflected. Um, those ephemeral environments are
00:09:48great from a security perspective. They're not indexed. They're not discoverable. They have a degree
00:09:52of entropy. That's really nice. So I let people run pretty loose inside of those, um, ephemeral environments.
00:09:58And then we have some, you know, review and guidance for actually getting something shipped to a production
00:10:04Vercel endpoint. Um, because we're really permissive on this access, I can be really restrictive on other access.
00:10:10So again, sorry to my lovable friends in the building, but like, we're not building on lovable.
00:10:14We're building on Vercel, like whatever you need to do on lovable, come do it over here.
00:10:18We have a better solution for you. And I just have a strong fundamental belief that if it becomes the
00:10:24place where you bring your problem and they show you how to do it securely, you will win converts.
00:10:29You will win adopters. If it becomes the department of no, you will have people working around you.
00:10:33So by being extremely, extremely aggressive about what we let people do inside of our guardrails,
00:10:39I think we're going to win the shadow it battle and have a better security posture like net net.
00:10:43Nice. Um, and Greg, you've thought carefully about what traditional engineering
00:10:47controls work and where they break down. How do you build guardrails when builders aren't engineers?
00:10:52Yeah. I mean, we definitely, it's a big focus for us, right? Like putting the controls and guardrails in
00:10:59place. I think a couple of things we, we integrate it into our CICD pipeline first off, right? So when
00:11:04we're doing software development, we have automation. We use code rabbit for automated PR reviews for code
00:11:12changes that are really complex. We have a human review it as well. Um, we do security, uh, code scan for
00:11:18security vulnerabilities, code standards, code quality, things of that nature, check for sensitive data,
00:11:24secrets, right? Like, so all of those things are part of the automated pipeline. Um, because we are an AI
00:11:31native product and we have inputs and outputs for AI models. We have to also put controls over that as
00:11:37well, right? So input into an LLM, we need to make sure we process that and make sure there's no like
00:11:44problem injection, for example, right? The outputs of the model, same thing. We have to validate that output
00:11:50before it actually goes to our end customer, right? Who is our real estate agents. And so, you know, we do a number of
00:11:56different things. One of them is, you know, we'll do eval, for example, right? Like to actually go
00:12:01and score the output of the model. So, you know, we check it for hallucination, relevancy. Um, compliance
00:12:09is actually a big one as well for us because in real estate we have, you know, regulatory policies
00:12:14that we've got to abide by. So, for example, with Fair Housing Act, we have to make sure that
00:12:19the output of our AI is not discriminatory for, um, consumers, right? So, and we actually build a custom
00:12:25model for that. And so, anyways, I think having that full end-to-end pipeline or deployment, all the
00:12:33controls in place is going to be really key for us. And honestly, it's what makes us comfortable,
00:12:37what made me comfortable to really lean on AI for the development, right? Um, otherwise, I think we
00:12:43would be a little bit more conservative in our approach, but because of the controls in place,
00:12:47I think we feel a lot better. Yeah, nice. Um, so, I'd like to talk about the methodology behind how
00:12:52agents actually work inside your organizations. And, uh, Greg, we'll start with you. Um, so, you have
00:12:58very specific framework for how you structure agent workflows, um, where each agent mirrors a human
00:13:04role and engineers effectively become leads managing multiple streams. Can you walk us through how that
00:13:10works in practice? Yeah, we definitely model after humans, right? And so, if you think about a
00:13:17traditional software development process, you have different roles, right? You have your DevOps,
00:13:22you have full stack engineer, a web developer, QA. Those are roles which we create specialized agents
00:13:29for. And then we have responsibilities, right? Each person has different types of skills and
00:13:34responsibilities. So, we've built a library of different AI skills. For example, how do you scan
00:13:41the code for it to check for vulnerabilities? Or how do you write unit tests? These are all different
00:13:46skills. So, each AI agent can have and use a different set, you know, multiple sets of skills,
00:13:53um, at their disposal. So, that's a big part. I think the other part is just like the process for
00:13:59software development is also something we emulate even though it's AI. So, you know, we start with the
00:14:05product manager, right? So, our product manager will go and write requirements. Now, they will now
00:14:10leverage AI to help them write the requirements. And we put them in, we use linear, so we put them in
00:14:15linear, right? That's our source of truth. And then we have, um, engineers will take that and generate
00:14:22technical requirements. Again, leveraging AI. Then they'll break it down into user stories that are put in
00:14:27linear. And then we'll have AI agents to take the story and actually go do the development on it, right?
00:14:33And so on and so forth, right? And then you have QA, you have deployment. So, it follows, um, a kind of more
00:14:39traditional process, but just turbocharged with AI, right? And then I think at the end of it, we do leverage
00:14:46something called compound engineering, which is, you know, open source framework for as we're doing agentic
00:14:52development. Um, it actually will remember the lessons learned so that the next time you do another
00:14:59task, you pick up another task, the lessons will be there, right? And so you're just continually getting
00:15:03better and better, uh, in your software development. Nice. Um, Andy, you made a deliberate choice not to
00:15:10hand people V0 code generation shortcuts. You wanted them learning the fundamentals. So, why and how does that
00:15:18change what they're able to build? Yeah. Uh, we obviously we are very impressed and our team is using
00:15:26agent frameworks and other parts of our software development, but for my citizen builders, I should
00:15:31have made the counterintuitive choice that I wanted them, um, not in a fully managed environment. I wanted
00:15:36something that was like just enough engineering that, uh, they knew how to operate it, but not so distinct
00:15:43from our regular processes that it would be hard to build these processes in. Um, you know, it's not
00:15:48really a secret in our leadership team that some of these projects are throwaway wear or demo wear. Uh,
00:15:54had a very eager young associate who, you know, absolutely admire, love his energy, kind of a
00:15:59monitoring the situation guy who put together a risk and safety, uh, tool that monitored all our real-time
00:16:07events. So, you know, we're doing 2000 events around the country with doctors. We have to like, you know,
00:16:11monitor like, well, I guess you don't really have to monitor anything because we don't, but he wanted
00:16:15us to monitor weather and security and safety. And he built this crazy dashboard that was like, we have
00:16:20an event in Phoenix and there's a heat weather advisory. It was like, this is, this is nuts. Needless to
00:16:25say, that's not something we're using every day in our software. That was a good idea. He learned how to use AI.
00:16:31He learned how to use tools. It was an interesting demo. We haven't scaled it, but by putting these
00:16:36tools close to the builder, when we do find something that hits and we found a bunch of stuff that has
00:16:40hit, it's easier for us to pull it into our actual engineering workflows. It, it doesn't look too
00:16:45dissimilar. So that's why I'm bullish on frameworks like Vercel and not on like walled gardens where
00:16:51you're going to have your non-engineers build something in a low-code environment. I actually think
00:16:56that like making the high code environment more friendly to newcomers is going to get more of
00:17:01these projects into production. Yeah. Anything that's like a meaningful app or agent where like,
00:17:06you know, 400 salespeople at Vercel are using it. Ultimately it's in our GTM monorepo or GTM
00:17:12eng team went through it. Um, so that totally aligns. Um, Victor, you're designing a progressive
00:17:18disclosure model, um, point and click, then chat based customization, then full app building for
00:17:2440,000 plus customers. How do you think about meeting users where they are on that spectrum?
00:17:30Yeah, I think, um, we like to look at our builder personas usually in a binary way,
00:17:36which is either you can build, you can write code from scratch or you need a vibe coding tool. But I
00:17:43actually think that spectrum is, there's many different steps of capabilities. On one hand are
00:17:48the people in this room who can build anything and your GTM engineering team. And on the other end,
00:17:53you have sellers, sales managers in manufacturing verticals or that have never built anything with
00:18:00code all their life, right? So what we want to do with this V0 and zoom info integration is address all
00:18:06those. If somebody has the knowledge and know-how and gone through your training, uh, to write code,
00:18:13we want to unleash their creativity and then they can get access to all the tools. If they want to get
00:18:19started and get value right away through point and click, they can do that as well. We're in this
00:18:24world where you don't have to pick and choose between one or the other anymore. So really exciting about
00:18:29it. Awesome. All right. To close this out, I'd love to make it super practical for people in the room.
00:18:35So if an AI agent could do one thing at your company today that it currently can't, what would it be?
00:18:43Andy, you want to kick us off? I think for me, it's not so much the AI agent, it's what's around it.
00:18:53So, you know, you mentioned shadow AI, shadow IT, you know, think of it as like shadow AI, right? It's,
00:18:58I do think like right now we have this huge unlock where you're empowering non-engineers to build,
00:19:04which is great. But to your point, Gene, earlier, right? Like the governance and the framework around
00:19:09it, so you can turn that prototype into a durable, reliable, secure software. You know, I think in having
00:19:17like the framework and all those controls and governance stuff we just talked about, having that all kind of
00:19:21built into the pipeline is going to be really big unlock. So yeah, I'm looking forward to, you know,
00:19:27trying out the new product. Enterprise apps and agents, you heard it here first. Awesome.
00:19:34I think for us, it's this evolution from third-party tools to homegrown control planes,
00:19:40and then once the control planes get product market fit, thinking about like what the agent can do to
00:19:44support the team. So, 2026 was the first year that I started getting a lot of
00:19:50messages at work from robots, right? Like sort of requests for action from robots. And I think that
00:19:58that trend is only going to continue. So, you know, we're taking some of our marketing and our
00:20:03promotion and our lead gen and our, you know, sort of, you know, say we need to run an event in
00:20:08Arizona and I need to figure out all the cancer doctors in Arizona. I want the agent to understand at a
00:20:14high level that when we have an event coming up and when we have doctors that haven't interacted with
00:20:19us, just go target them, like go put money into LinkedIn ads, go put money into meta campaigns.
00:20:25And we've called that like the full self-driving vision. It's something that we're working on now. So
00:20:30figure out what the use case is, build your own control plane, and then figure out where you can
00:20:34let the agents take the wheel and do tasks that are either going to be more, you know, reliable,
00:20:41more, you know, like kind of boring tasks that you wouldn't want to have a person do,
00:20:44and hand those off to the agents and let them drive. Awesome. All right, Victor, round us out.
00:20:49All right. So right now, what we've been talking about is that the agents are assistants to the builders.
00:20:55What I'd love to see in the next few months is that the agent is the builder themselves.
00:21:01I want the agent to be able to build the application, ship the application through the right guardrails,
00:21:07and the new eval, instead of the output of the agent being the eval, the eval should be
00:21:12was the user successful in what the application was trying to deliver to the user.
00:21:16The agent then gets that instrumentation, and then does the next version of the application
00:21:22delivers that again, right? One thing why I love working with Vercel is you all are all about
00:21:27shipping. And I think you learn more about shipping to your first user than it is just building the
00:21:33code. So if an agent can get through you through that loop over and over again, it really accelerates
00:21:38the customer getting the value. Awesome. Well, thank you all for joining me,
00:21:43really appreciate it, and look forward to see what else he will ship this year. Thanks,
00:21:48everybody. Thank you.