Rebuilding the web for agents — Liad Yosef, MCP Apps
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Hi, everyone. Hello, hello. So we're going to talk a little bit about the agentic web and more
00:00:19specifically what it means and how do we make the web ready for agents. One disclaimer, I built this
00:00:26talk yesterday, so it might be outdated because things are moving really, really fast in this space.
00:00:32And I think I need to introduce myself just to get some context. So I'm the co-creator and
00:00:38maintainer of a spec called MCP Apps. MCP Apps, that's the underlying spec behind ChatGPT Apps,
00:00:45CloudApps, Copilot, GitHub. Every chat-based app that you see is based on the spec MCP Apps
00:00:53in the MCP committee. I'm also the co-founder of a company called Aura, where we research the
00:00:58agentic human interactions. And I built and led the agentic storefronts in Shopify. So the topic of
00:01:06agentic web is really close to my heart. And a little bit of primer about MCP Apps, if you're not
00:01:12familiar. MCP Apps are actually, this is actually the spec that paved the way to agentic web. So MCP
00:01:18Apps were released as a spec and as a standard a few months ago with the support of Claude first as
00:01:25the first client, and then all clients followed. And if you play with any chat-based app and you pulled
00:01:32any visualization and interaction UI layer from an MCP server, you probably used MCP Apps.
00:01:39And the good thing about MCP Apps is that everybody wins. Because if the servers or the providers can
00:01:48send UI chunks into the chats, then the apps gain their brand and identity. They get to keep their UI and
00:01:58not being reduced to a database or just text-based information. The users gain their trust and
00:02:04familiarity. Because if you ask chat-gpt, hey, book some hotels for me, and you see booking.com,
00:02:09then you know it's booking.com, right? You know who you're interacting with. And the hosts or the chats
00:02:16gain access to the world of capabilities. You know, a few months ago, people asked, yeah, why won't
00:02:22OpenAI just build everything out of scratch. But OpenAI will not negotiate deals with hotels or support
00:02:29users that want to change seats in a venue. We need those services. And MCP Apps actually solve for
00:02:35this last mile of interaction.
00:02:41Yeah. So this is the benefit that we get. And MCP Apps solved our last mile of interaction, right? So what
00:02:51does it mean the last mile? The last mile is as agents become better, as models become better, they can do
00:02:57things more autonomously. But we are still in the end of the chain as humans. We still need to be able to
00:03:05choose the hotel, or to view a 3D model, or to choose a seat at the venue. All those last mile of
00:03:12interaction, that can change. And this is an example from... Oh, sorry. Yeah. So this is from like
00:03:20Claude, a PR on MCP Apps. This is how it looks like, right? So when you're embedding apps inside chats,
00:03:28you get this unified experience of how UI feels inside the same context of a chat. And as you can see,
00:03:36in the same conversation, you get data from Booking, you get data from Altrails, you can get data from
00:03:41everything you want to interact with, right? So this is already happening. MCP Apps is already widely supported
00:03:49across every chat agent, except Gemini, but that's coming. And this is already happening. And the
00:03:57interesting part is that it brings us to what we call the agentic web. So what is the agentic web? The
00:04:05agentic web is not a web of agents. It's not the web as we know it today. It's a shift. It's a change. It's a
00:04:11change in the way that we see websites and the way that we see browsers. Because up until now, we spent
00:04:2120 years, two decades, perfecting this experience. Meaning that if I want to somehow do a project or
00:04:29fulfill a task or plan an anniversary, I have to open tabs in my browser, and I have to go through these tabs,
00:04:36and I have to convey my intent differently to each of those services, right? Meaning that if I just want
00:04:43to plan an anniversary, then I need to learn Google's UI and Amazon's UI and Booking UI and the other
00:04:51Booking UI and the other Amazon UI. All that just convey the same intent to those interfaces. And every
00:04:58company perfected this user flow. But we don't need to do it anymore, right? We can just take these
00:05:04interfaces and break them into atoms. Because I don't need 99% of Booking's dashboard. I don't need
00:05:1299% of Airbnb's dashboard. I definitely don't need Jira's dashboard. But I do want to convey my intent to
00:05:19those services. So why can't I let my personal assistant use that? So if I have a personal assistant,
00:05:27I can just take these atoms. And it composes these atoms. And it says, yeah, I see that you have an
00:05:31anniversary coming. This is the view from Google that says that you have an anniversary coming.
00:05:36So I can book, I can buy things for you. I can book your hotel. Now, Claude knows me. So he knows that
00:05:43I prefer to book hotels in the nature, right? So he knows to pull the map from Booking. I don't need to
00:05:49think about it. And what Booking and Amazon and Google benefits from this is that they have this integration
00:05:56layer that they don't need to develop. Because Claude has the context for me, right? So Booking doesn't
00:06:01need to develop integration with my calendar. Amazon doesn't need to develop integration with Booking.
00:06:06So it's a win-win-win. This is the view that we're aiming for. And this is very different
00:06:13from what the things are going, which is this, which is basically browsing into different services just
00:06:21to stare at different text boxes that ask what I want to do there. But no one is going to do that.
00:06:27In a year, no one is going to browse to Amazon's agents and Etsy's agents and Expedia's agent. I don't
00:06:34want to do that. I have my own personal assistant. I don't want to use your agent. I want to use my agent.
00:06:41Okay? Now, if we think about it, it means that websites as the source of truth are something that's
00:06:49going to go away. Because why would they need the website? Why would they need to open the tab? Why
00:06:55would they need the information that the website is working really hard to convey to me?
00:06:59And the immediate response to that is that, yeah, we have browser agents. We have computer use agents.
00:07:04We have very smart agents or assistants that can browse the web for us. But that doesn't make sense.
00:07:10Because that's a faster horse solution. Why would they let an agent that doesn't know anything about
00:07:17the human perception limitations work with filters and paginations and sorting and all of those UIs that we
00:07:24spent decades perfecting for us just for it to fulfill its task? So Google is very bullish on
00:07:33the other end of the spectrum, which is co-browsing and websites. And they have a protocol called or
00:07:37standard called WebMCP. WebMCP says, okay, instead of the agent taking screenshots of the websites and
00:07:44trying to figure out what's happening, let's have the website expose some tools in the website, JavaScript
00:07:49tools, and the agent will work with it. And you can see here an example of Gemini in Chrome that buys
00:07:55stuff for me or finds deals for me as I browse the web, which is very nice, but it's very simple.
00:08:02What happens if we push Salesforce dashboard to Gemini on Chrome? Why would I want that? Why would I want
00:08:09to browse to Salesforce dashboard with Gemini to click buttons for me in a dashboard that I don't even want
00:08:15to use? I mean, this is definitely not what we want. And the shift is that assistants are becoming our
00:08:24entry point to the web, right? We see it everywhere. Every major lab wants its app to be the everything
00:08:31app. And the shift is happening. And friends of mine that connect Jira's MCP to their IDs, they don't go to
00:08:39Jira's website anymore, right? And my mother uses ChatGPT for everything. If she could book a doctor's
00:08:44appointment using ChatGPT, she won't go to that clinic's website anymore. And you have a long
00:08:49deal of websites that no one is going to see because it's much easier for me to connect them using my
00:08:54personal assistant. So websites will become obsolete, browsers will become obsolete, and personal
00:08:59assistant will become our only gate to the web. And I know what it sounds like. It sounds like old men
00:09:04yelling at a literal cloud, right? I mean, yeah, they're not going to go completely obsolete,
00:09:11but we see it with the younger generation today. My nine-year-old goes to ChatGPT. He doesn't go to
00:09:17Google when he wants to search for stuff. I was traveling to Georgia, the country in Eastern Europe,
00:09:24and there was an older lady there that asked me to take a picture of her, like 70-year-old. And on her
00:09:28phone, she had three apps: WhatsApp, Camera, and ChatGPT. That's it. The shift is happening. And very
00:09:36smart people, like David Kremer from Sentry, said a year ago, I will take this bet against anyone who
00:09:44thinks websites are going to be obsolete in 25 years. That was one year ago. A few days ago, David
00:09:49Kremer published Designing for Agents, which says, we have to recognize that interaction with products at
00:09:54Sentry will no longer be exclusively through our web application. And we have to design for API first
00:10:00as a surface. So everything is going headless. Salesforce recently went headless, which is very
00:10:06important because Salesforce's main differentiator from its competitors is its UX. It's the way that it
00:10:12conveys everything to the user. But it went headless. Cloudflow went headless. Sentry went headless.
00:10:17This is a tweet by a Cloudflow CEO saying that agents' traffic surpassed human traffic in the web. And we
00:10:25have this spectrum of interaction because if my agent is autonomous, I don't need this last mile of UI,
00:10:30right? If I have an open claw, I just send and I just say, yeah, organize my emails. It comes back,
00:10:35said, done, good. But if I want to book a hotel for my honeymoon, I probably need this last mile of
00:10:40interaction. So we call it the nearly headless web, right? It's not completely headless. The agents will
00:10:47interact with your website headlessly, but then it will bring back these UI resources for you.
00:10:55So user experience as we know it is being split. Now it's the agent experience of your website plus how
00:11:02a user experiences the agent that experiences your website. But 90% is going to rely on agent experiences,
00:11:09which means your website needs to be agent ready for everything. It needs to be agent ready for
00:11:16your agent. If I'm booking, I need to make bookings that are ready for bookings agent. I need to make
00:11:21it ready for my customer's chat GPT or my customer's open claw. And I need to be ready for my human
00:11:28customer who doesn't have an agent but still wants to browse to booking.com, right? So I need to be
00:11:33agent ready. And what we found out is that agent ready means a lot of things. So we heard a lot about
00:11:39AEO, SEO, GEO, how to get discovered. But discovery is only the first step. Because once an agent knows
00:11:47about you, it still needs to know what you are and how to interact with you and how to authenticate to
00:11:51you and how to headlessly pay you, right? And actually, when we built -- it's an anecdote -- but when we
00:11:56built analytics for our product, we asked CloudCode what's the best analytics service. And it
00:12:02recommended PostHog, which is an analytics service. And we said, no, we prefer Mixpanel. Because we know
00:12:07Mixpanel. We worked with Mixpanel for a decade. We know how to work with it. And CloudCode insisted on
00:12:11PostHog. Because it said, PostHog has better MCP and API. And I can integrate to it better. So we
00:12:17don't have brand loyalty, right? We went with PostHog, if that's what CloudCode recommend. But it made
00:12:21us think about Mixpanel. Mixpanel spent a decade perfecting their UX and developer experience. We
00:12:26just left it just because CloudCode prefers PostHog. And it will always happen. Hermes, for example,
00:12:31if it uses your product and it needs to spin up a browser to use your product, it will remember that. And
00:12:35next time, it won't go to your product anymore. Right? So at Aura, which is an agentic web research lab,
00:12:43we started researching. We raised some capital. And we started researching what does it mean for the
00:12:49web to be ready for agents. And we have these readiness benchmarks, which is interesting. So you
00:12:57can run -- it's free. You can go to Aura.ai. You can run any website. You get this score and benchmark
00:13:02according to a lot of protocols and best practices. And you have agentic feedback. So the agent actually
00:13:09returns the feedback about your website. And we have this leaderboard of how companies and
00:13:16products rank according to this benchmark. And we started mapping the web and everything was nice.
00:13:24But we hit one insight, or one unexpected result. We found out -- anyone here heard about LMS.txt?
00:13:36LMS.txt, that's like the de facto standard to be agent ready. You say, yeah, if your website publishes an LMS.txt,
00:13:42agents know how to interact with you. And you have auth.md and pricing.md and X402 and a lot of standards. And we found out that almost 50% of the website that we tested,
00:13:52that we ran, published LMS.txt. But none of the agents that we ran on this website actually used LMS.txt.
00:14:02Actually, almost all of the agents went straight to the docs page and then the home page. And the 40% that did use LMS.txt used it only because the docs
00:14:12pointed out that there's a file called LMS.txt that they need to use. So then it hit us. We said, it doesn't
00:14:19make a lot of sense for us as humans to define to agents what they need. No one is doing it anymore.
00:14:26Even OpenAI, they don't publish best practices for tools anymore. Because they say that every time they publish
00:14:31best practices, the models become better and these practices become obsolete. Six months ago, the best
00:14:36practice for an MCP server was, yeah, have three paragraph descriptions so agents will know how to
00:14:41interact with you. And now three lines are enough. So best practices become solid. We need the agents to
00:14:47define what the agents need, right? We need agents' feedback on this website. So we build Aura's Journey.
00:14:54And Aura's Journey, that's really cool. That's also free. You can go to journey.aura.ai and you can run on
00:15:00any website, any intent with any agent and see the path of the agent as it tries to interact with the
00:15:08website. So for example, atia.com, choose Cloud Code and we run it on the website. And you can see in real
00:15:17time how the agent goes and what it tries to look for in the website. And we did it tens of thousands
00:15:25of times just to understand what agents really look for when they look, when they try to interact with
00:15:30websites. And the cool thing is not running just one harness. It's running multiple harnesses. You see
00:15:39here Cloud Code and Eve, which is Vercel's harness, and ChatGPT on the same website with the same intent.
00:15:45Okay, so this is Cloud Code. This is Haiku. This is Eve. See how different the agent journey looks like.
00:15:52And this is ChatGPT. So ChatGPT could find results better. You can just see the journey across the
00:16:01website. And we need to understand why. We need to understand why does it happen. Why websites publish
00:16:08auth.md files, but agents don't look for auth.md files. And what do they look for?
00:16:13So in Aura, you can actually go to any business question. So for any domain, you have like business
00:16:21goals or questions. And you can see the paths that the agents are taking. And the interesting
00:16:28part of it is that, okay, now we know how agents interact with websites. What's the next big milestone
00:16:33big milestone for the agentic web? And for those of you who heard the previous talk,
00:16:40it's discovery. Right? But it's not the discovery that we think of. It's not a CO. It's not GEO. It's
00:16:45not AEO. Because we have to remember that every revolution came with its discovery layer. Right?
00:16:52The web revolution came with search. Mobile came with app stores. Social came with feeds.
00:16:59What is the discovery layer for agentic resources? What is the discovery layer for MCPs? What is the
00:17:03discovery layer for OpenAPI.JSON? What do we even look for? I mean, maybe Airbnb has a better MCP
00:17:09than Booking.com. But Booking.com has better API than Airbnb. So how do we do it?
00:17:16Web search. Classic web search. It's not enough. Because it's based on human
00:17:22SEO and human page rank. And it's not flexible enough. Custom registries, like per agent or per
00:17:29chats, they're not enough. Because they're closed. And they require every app to submit itself to these
00:17:36registries. And a central registry, like an MCP registry, that's not enough. Because who will do
00:17:42the curation? What's the governance model? How do we decide which resource comes to that registry?
00:17:48There are emerging standards around it. There's AI Catalog.JSON, which is a standard by
00:17:53Anthropic OpenAI, Google, MCP, and A2A, which standardizes how a website exposes itself to agents.
00:18:00And there's the agentic resource discovery standard, which is by all these companies and more,
00:18:05which is basically standardized how a discovery layer or a directory exposes itself to agents. So we built
00:18:14our other directory, right? Because we're a research lab for the authentic web. So we build this. And in
00:18:18our other directory, we take all the domains that we scanned or scanned ourselves, and we put it in a
00:18:24directory that agents can actually query, right? We expose the AI Catalog.JSON files. So you can see
00:18:31here, for example, for monday.com, you can see that we generate this JSON file that basically tells
00:18:38agents, yeah, this is the MCP server for monday. This is the API server for monday.com. And you can
00:18:42just, the agent can just query that, right? So for every domain, you can just go to
00:18:47go to AI Catalog.JSON. And we expose the registry, the directory itself. So for every entry on the
00:18:55directory, we can tell the agent, yeah, for example, Vercel, these are the agentic resources,
00:19:02and this is how you access them. And obviously, this directory is fully ARD or agentic resource
00:19:08discovery compliance, so any agent can just query that directory with any query, right?
00:19:13So this is Aura.directory, if you want to take a look. All of it is part of the Aura multiverse,
00:19:21journey, directory, and the ranker. And we found out some very other interesting insights. For example,
00:19:26being agent-ready and being human-accessible is very similar. Because LLMs, when they come to your
00:19:33website, they're like users or people with vision disabilities. They don't see your website. They
00:19:39need other signals to understand how to work with your website. And making your website human-accessible
00:19:44helps agent accessibility and vice versa. So to wrap up, these are MCP apps. This is what I've been
00:19:51working on in the past few months. And this was the last piece for the agentic web. This actually brought
00:19:57the agentic web. And now agents start to roam the web. And agents need different things. They need
00:20:03different roads. They need different infras. But we don't want to rebuild the web for agents. We want to
00:20:09make the web agents accessible. So let's just make sure the web is prepared for when agents are coming.
00:20:18Thank you very much.
00:20:39you
Community Posts
No posts yet. Be the first to write about this video!
Write about this video