The Universal Remote Control for AI — Alex Hancock, Block
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Alex Hancock: Hey, everybody. My name is Alex Hancock. Today, I'm going to talk about universal
00:00:17remote control for AI. And before I start, I just want to say the previous speaker said
00:00:21that MCP client maintainers haven't implemented support for tasks because they're smart. I'm
00:00:26an MCP client maintainer. I can tell you just because I'm lazy. I haven't done it. Okay,
00:00:33so a little bit about me before we start. I am a software engineer at Block, which is the
00:00:37parent company of Cash App and Square and Tidal. We have a few different things going on now.
00:00:43And I've worked there for a long time. I worked on Square product stuff and Cash App stuff.
00:00:47But I've been doing open source AI for the last couple years. Specifically, I work on this
00:00:51open source harness project called Goose, which started as an internal project at Block.
00:00:56Yeah, some Goose fans out there. And then, yeah, we open sourced it and we donated it to the
00:01:02Linux Foundation. So now the IP is there. But we still, lots of us from Block still work on it.
00:01:07I'm also a maintainer of MCP, the model context protocol. I work on the rust SDK for that project.
00:01:14And more recently, I've also started some work on ACP, the agent client protocol, which is what I'm
00:01:20going to talk about today. So I think we have an issue with harnesses that I want to try to put to
00:01:27you all today, propose to you all today as a problem and then recommend a solution. So what I've been
00:01:35noticing recently is that we've got lots of great harnesses out there, right? There are ones from the
00:01:40labs. There are ones from different companies. There's lots of open standards-based ones.
00:01:46But I noticed that the interface to them is often custom or bespoke. And in the worst case, it's like,
00:01:52you might have some harnesses where there's literally only one client application you can use to control
00:01:57that harness, right? And I think this has a couple issues with it. But the analogy that I'll make with
00:02:03the web is it would be like if you had to use one browser or one given protocol to connect to every
00:02:11website, right? That just wouldn't work. You wouldn't have something like the open web if that were the
00:02:16reality with browsers. And so I think we can do better. And the thing about standards, by finding a
00:02:22standard. And the thing about standards is that they create ecosystems and markets. And I would argue
00:02:28that in the agentic AI space, we have a good standard for the agent going out and doing things,
00:02:35right? Calling tools, taking actions in other systems, reading resources, reading data. We've all
00:02:41benefited as a community from having MCP, right? And the most powerful thing about MCP
00:02:47is not anything about MCP itself, but it's that everyone uses MCP. And that's why we have, you know,
00:02:53thousands or tens of thousands of servers around the world, and all the agents can connect to them
00:02:58and go and do things in those other systems. I would say that we don't yet have a good solution
00:03:05or a standard for client software to tell agents what to do, giving it tasks, telling it what to work on,
00:03:13and getting updates. And so I'm going to put forward an option today that I think is a good option that
00:03:20that we on our team have been working on and we think is a good a good solution in the open standard
00:03:26space. And this is ACP. So agent client protocol is the name of this project. And it came from the
00:03:32editor companies. It came from like, if you've used the Zed text editor, or you've used any of JetBrains
00:03:39products, the Zed folks and the JetBrains folks teamed up and proposed a standard for
00:03:45clients to be able to control harnesses. And it makes sense if you put yourself in their shoes,
00:03:49right? What they wanted to be able to do is write a single high quality client implementation
00:03:54in an editor, maybe in Zed or in IntelliJ or something like that, and be able to control any harness
00:04:00with that single client implementation, sending tasks, getting results back,
00:04:06seeing what files are being edited, etc. It makes a ton of sense if you put yourself in their shoes,
00:04:10right? But we saw this on the Goose team. And we think that there is a much broader utility than just
00:04:17editors, right? So it's relatively neutral and it doesn't have many editor-specific features. And so
00:04:22we think that this can be spread to a wider range of client software. To go into a little bit more depth
00:04:30about ACP's design and what you can do with it, it lets you establish connections between clients and
00:04:37agent harnesses that have a given set of capabilities associated with the connection. And then you can make
00:04:45sessions. Within sessions, you can send user messages, the things that a user is maybe typing into the app or
00:04:51that the client software wants to send. The agent can then respond to those with text, more images or audio,
00:04:59text, etc., or updates about what's going on. So like if a tool is called, it can send a tool call
00:05:04notification and explain what tool was called and what the metadata was. And it can also send things like
00:05:10permission requests so that if the client software needs to show the user, you know, should I do this
00:05:16tool call, yes or no, it can go over this protocol. And it's pretty simple in its design. It uses JSON-RPC
00:05:24messages. And the thing we like about it most is that it's extensible as well. So you're not limited to
00:05:29just what's in the vanilla protocol. You can add custom methods. The convention is you put an underscore
00:05:37and then you start to put your custom methods. And the thing I like about this is that if enough
00:05:41harness projects or client projects adopt this, we can start to see what we're all doing that's the
00:05:46same, right? Like if the codex team has some custom methods, the goose team has some custom methods,
00:05:52the client team has some custom methods, whoever, we can see what emerges in the in the ecosystem and
00:05:58what makes sense to get on a standards track and bring into the protocol itself so that this is sort of
00:06:03shaped by usage and shaped by the community. I'm going to do a demo of a standard I/O version of this.
00:06:12So I'm going to open Zed and I just have a really simple project here where I'll say tell me about
00:06:18this project. And so this is a single HTML file. So you can see I was able to type my query into Zed
00:06:25and this is the agent in play here is Goose. So it's using Goose's ACP interface. And it's, you can see,
00:06:31it's like sending text back. It's sending tool call information back about what it read and what it
00:06:36did. And then it found, you know, that it's a single HTML file and explained it. And I'll do another,
00:06:42I'll do another one. This is one from a company called Poolside AI. I'll say, tell me about this
00:06:50project in the same project. And so this is a terminal based client getting exactly the same experience from
00:06:55the same agent, one implementation on the harness side, and you can now use any client, right? And so
00:07:01you can see it did the same thing. It showed me some text results back, it showed a tool call, and then
00:07:06it showed a, it's streaming in a summary. So that's a basic demo showing two clients talking to the same
00:07:13agent over standard I/O locally in this case. But local obviously isn't enough, right? If you want this to
00:07:21take off, you have to be able to do remote as well. Agents are going to be running in the cloud. And so
00:07:25when we came to this project, we saw that it did not have remote support yet. So we specified an HTTP
00:07:31transport, there's an HTTP version and there's a WebSocket upgrade. And so now the messages are the same,
00:07:37the protocol semantics are the same, but there's a new transport that is just landing now that enables
00:07:42remote. And the way we think about this on the Goose team, the agentic stack is there's sort of these
00:07:49four important components, right? You have the client, which is like the app that the user is using,
00:07:54or a headless app running somewhere on the machine. There's the harness, which is the program that
00:07:59implements the tool calling loop. There are the tools themselves, as often MCP. And then there's the
00:08:04model, right? And if you do a remote transport for the agent client protocol, and MCP has remote
00:08:13transport for tool calling, and the models have kind of all had remote endpoints, like responses,
00:08:18APIs for a long time, now you have the flexibility to move all of these four components around. They
00:08:24could all be on the same machine. The harness could be on a different machine than the client.
00:08:29The model could be the only thing that's remote. The tools could be the only thing that's remote.
00:08:34Aligning on standards and making sure that they have good transport stories is what's going to let us
00:08:39move all the pieces of this agentic stack around. And I can show a quick demo of this as well.
00:08:46So this is a client just to show how easy it is to create clients for this. I just vibe coded this,
00:08:53last night, and I'll say, write a poem. So this is again connecting to that same process on my machine.
00:09:01In this case, I'm running it over the network, but it's on my machine.
00:09:03It's connecting and sending goose instructions for what to do remotely. So this could be in a container,
00:09:09it could be up in the cloud, but the messages are the same and the library you use is the same.
00:09:15So you can just switch between local and remote very, very easily.
00:09:21So if you want to get plugged into this ecosystem, start experimenting with support,
00:09:25either making your own clients or adding stuff to harnesses.
00:09:29This will link you to the agent client protocol site for how to get started. There's a number of clients
00:09:34and agent servers already out there. This ranges from editors, desktop applications, mobile applications,
00:09:41terminal-based things. There's a proliferation. And I think the use cases are potentially huge,
00:09:49right? If we, if we get some interoperability going here, because you can have people can make personal
00:09:54clients. That's exactly how you want it orchestrating your agents. You could have sort of clients created
00:10:00for certain business domains or an individual company or a set of clients from a company. You could customize
00:10:06like a white label client and have it work with all the harnesses. And I also think if we make a new
00:10:12category here, we're going to see quality of the clients go up, right? Because any, anytime you get
00:10:17an ecosystem or a marketplace going and there's many options, users can vote with their feet if
00:10:22clients aren't meeting their needs. And so people will start to compete on the quality of the user
00:10:26experience. And, and like overall, I think this should drive up, uh, the user experience of using AI.
00:10:33That's what I've got today. Thank you very much. And if you want to chat with me, find me
00:10:37after or send me an email. I'm happy to get you plugged into this work. Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video