The Spatial Harness: Bringing Agents to the Canvas — Max Drake, tldraw
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00-
00:00:13- Thank you for coming here to my talk
00:00:16to watch me talk about agents on the canvas.
00:00:20The first thing I'm going to do though is before,
00:00:23I have to record my screen.
00:00:25The first thing I'm going to do is I'm going to ask
00:00:26my agent to do something on the canvas.
00:00:30And what I'm going to do is say hey,
00:00:33my colleague Spencer just emailed me a link
00:00:35to a Notion document for a really cool demo
00:00:37we could build with the TLDRAW desktop app.
00:00:40Can you like find that document
00:00:41and then can you build it on the desktop app?
00:00:45Thank you.
00:00:50Okay, so that's going to build
00:00:51and then we're going to come back to it later
00:00:53and hopefully it'll work.
00:00:55Hi everyone, my name is Max Drake.
00:00:56Thanks so much for coming.
00:00:58I work on agents on the canvas at TLDRAW.
00:01:02I'm a product engineer there.
00:01:05So first things first, am I qualified to be giving this talk?
00:01:07I like to think so.
00:01:08I've been doing like agents on the canvas stuff
00:01:10since before ChatGPT came out.
00:01:13I think it's really cool.
00:01:14I think there's like so much UX stuff you can do
00:01:16with when you get LLMs, you have them working in space.
00:01:20And I think it's really interesting.
00:01:21I've been doing it for about as long as you can
00:01:22have been doing it.
00:01:24More recently, I've been talking about this a lot.
00:01:27Here's some proof.
00:01:29And yeah, so I work at this company called TLDRAW.
00:01:32Can I get a quick show of hands?
00:01:33Has anybody ever heard of or used TLDRAW before?
00:01:36Yeah, okay.
00:01:38Awesome, so yeah.
00:01:39The thing that you probably use if you use TLDRAW
00:01:42is this app right here.
00:01:44So this is all TLDRAW is a free infinite canvas
00:01:48whiteboarding app.
00:01:49You know, we have selections and arrows and resizing
00:01:53and all the things that you need in a whiteboard.
00:01:57TLDRAW is also the company that makes this app.
00:02:00It's based in London.
00:02:01It's where I work.
00:02:03But the last thing that TLDRAW is,
00:02:04which is I think in my opinion the most important,
00:02:07is it's the infinite canvas SDKs that powers this app.
00:02:11And so what that means is that, you know,
00:02:13this is kind of the TLDRAW, the SDKs, the engine
00:02:16that powers a lot of infinite canvas experiences,
00:02:19because it turns out it's really hard
00:02:20to get that kind of stuff right.
00:02:23And so if you ever want to build a Miro competitor
00:02:26or a slide designer, or if you're like Replit,
00:02:30Replit has their whole new agent canvas stuff
00:02:32built on top of TLDRAW.
00:02:35And so the reason we built TLDRAW in the first place
00:02:37was that we were running into this issue,
00:02:40or people were running into this issue
00:02:41or they had this idea
00:02:42for this like really great killer canvas app.
00:02:44And they went to go build it,
00:02:45and everybody would run into the same problem
00:02:47where they would run into,
00:02:49they would have trouble making
00:02:50the actual canvas part of the app.
00:02:53And they would, you know,
00:02:54try to deal with resizing and selection
00:02:56and, you know, all the matrix math.
00:02:58And the issue is that they wouldn't be able
00:02:59to build their actual app itself.
00:03:01They would get stuck on the canvas.
00:03:02And so we built TLDRAW to kind of be the engine
00:03:06that could power, that could be the canvas,
00:03:07so that they could focus on the actual app.
00:03:10And so when LLMs came out, we, like a lot of other people,
00:03:12saw that this was going to be this weird new type of software.
00:03:16I don't know if anyone, you know,
00:03:17I'm sure a lot of you were building in 2022,
00:03:19and it was really exciting.
00:03:21And a lot of people, it was the exact same thing.
00:03:23People had the idea, had an idea for this cool app
00:03:26that would, you know, involve LLMs on the canvas,
00:03:29having them manipulating things in space.
00:03:31But then they would try to build it and they'd get stuck.
00:03:33There were no best practices.
00:03:34People didn't really know how to do it.
00:03:36And so at TLDRAW, we realized that we need to make it easy
00:03:40for people to build with agents, with LLMs on the canvas.
00:03:44And also, so TLDRAW, the SDK, as well as the app,
00:03:47has multiplayer built-in with, like, live sync.
00:03:49It's really nice.
00:03:50There's cursors.
00:03:51There's, you can see your collaborators' cursors,
00:03:53and selections, and viewports.
00:03:54And I think all of the things that make just the canvas,
00:03:57in general, a really great place for interacting with
00:04:00and collaborating with your colleagues,
00:04:02also make it a really great place for interacting
00:04:05and collaborating with agents.
00:04:06And I hope I'm going to be able to show you guys
00:04:08some of that in the demos that come up.
00:04:12So before we talk about agents on the canvas,
00:04:15really quickly, I want to talk about agents not on the canvas.
00:04:17I'm sure you guys have all used an app
00:04:19that looks like this, you know, Claude Code.
00:04:21And I'm gonna really oversimplify here,
00:04:23but basically, part of the reason why these apps are so good
00:04:26and why they work is because they're,
00:04:28the medium in which they're working, writing code,
00:04:31is essentially the medium in which they were trained.
00:04:32You know, it's text in, text out.
00:04:34That's how they were trained.
00:04:36And when we work with them, we give them a prompt,
00:04:38and they write code.
00:04:39It's text in, text out.
00:04:41You know, again, oversimplifying,
00:04:42but that's essentially how they work.
00:04:44I don't know if you guys have ever, you know,
00:04:46tried to get your agents to do, like, UI stuff
00:04:48and tried to get them to align something.
00:04:50Found that they could not do that whatsoever.
00:04:53'Cause it turns out, agents are really, really bad at working
00:04:56in 2D space and understanding 2D space.
00:04:58And it actually requires, like, a lot of engineering work
00:05:00to get them to do it.
00:05:02And that's kind of the project that we've been embarking on
00:05:05at Teal Draw recently.
00:05:06And so the first thing we had to do,
00:05:09this is an older project,
00:05:10but the first thing we had to do is get them to teach them,
00:05:12teach the agents, or at this point, not agents, LLMs,
00:05:15to understand the canvas and understand, kind of,
00:05:17what they're even looking at.
00:05:19So we had this project called Teach,
00:05:21where we taught, so I'm gonna prompt this really quick.
00:05:26I'm gonna say, hey, make the mouse blow out the candle.
00:05:29Yeah, so that's gonna take a second.
00:05:33This is an older project.
00:05:35But basically, what we had to do is we had to kind of, like,
00:05:39teach the LLMs how to take the screenshot that we give it,
00:05:42and the JSON, and all of the other information about the canvas,
00:05:45and, oh, yeah, there, okay, so yeah, that's some wind.
00:05:50It's, is it, sometimes it gives us smoke as well.
00:05:53Yeah, and we got a little smoke as well.
00:05:55So we basically had to take it, how to, like,
00:05:58and I wanna be very clear, this is not,
00:06:00this is not, like, a special mouse shape.
00:06:02These are just, like, you know,
00:06:04these are just shapes on the canvas.
00:06:07This is, and so the work behind this,
00:06:09it's a single-shot prompt, but we basically,
00:06:12we tell the agent how to interpret,
00:06:14both via screenshots and via the data,
00:06:16what is actually on the canvas, like, what it's looking at,
00:06:19which is actually, you know, it's not a trivial problem,
00:06:21and then also how to, we teach it how to actually act
00:06:25on the canvas and to understand how the actions
00:06:28that it produces will affect the canvas.
00:06:29So, you know, it got, it, you know,
00:06:31it made the, it made the smoke, it made the,
00:06:33it made the wind, it got the positions right,
00:06:35and it understood what it was doing.
00:06:38So we, we got this, we kind of figured out how the,
00:06:41like, kind of, we got, we taught it what the canvas is,
00:06:44but this was like a single-shot, single-prompt
00:06:46kind of thing, and so the next thing we built
00:06:49is the TealDraw Agent Starter Kit,
00:06:51which basically turns that and wraps in a harness
00:06:54that lets an agent work agentically on the canvas.
00:06:57The code is also MIT licensed.
00:06:58You can find it on, you can find it on the website.
00:07:01So here's a little, here's a little cat.
00:07:03I'm gonna make this a little bigger.
00:07:05What I'm gonna say is, hey, so somewhere else on the canvas,
00:07:08there are some friends for the cat.
00:07:12Can you please bring one of them over to the cat?
00:07:15Her favorite color is red.
00:07:18And so I'm gonna zoom out and I'm gonna show you guys
00:07:20what's actually going on.
00:07:21So you can see the view of the agent.
00:07:23There's some, there's some potential friends over here,
00:07:25and you know, if you read the, all right, so,
00:07:29and basically what's going on is that the agent
00:07:31has kind of like, we've given it a prompt,
00:07:34and using the information it has about the canvas,
00:07:37it's going to kind of like make some goals for itself.
00:07:40You can see there's some to-dos in the corner here.
00:07:42It changed its view in order to see what was, you know,
00:07:48the other stuff that was on the canvas,
00:07:49the same way that if you ask a coding agent,
00:07:52you know, you ask it, you know,
00:07:54where do we define this thing in the code base,
00:07:56it can go, it can search, it can find it.
00:07:58So this is kind of like turning that single shot
00:08:01prompting experience into this kind of like agentic thing
00:08:04that you can have, you know,
00:08:06it can autonomously set goals and work towards them.
00:08:11The next thing we did, we did this project called Fairies,
00:08:16and so we basically, we had this agentic experience,
00:08:19but we realized that, you know, TLDraw,
00:08:21and you know, the canvas in general is so collaborative,
00:08:23it's so multi-player, and we wanted to basically,
00:08:26we wanted people to be able to work together with their agents,
00:08:29and we also wanted the agents to be able to work together.
00:08:31So this is, this is a ferry.
00:08:34There's also, if you guys want to scan this QR code,
00:08:37you can actually, this is multi-player,
00:08:39you can join if you want.
00:08:40It requires a Gmail sign-up,
00:08:41but you don't need to pay for tokens.
00:08:43This is what the link is.
00:08:44So basically, this is a, this is a ferry.
00:08:47This ferry's name is Joan.
00:08:49They don't like being, they don't like being grabbed.
00:08:51You can, you can throw them around.
00:08:53You know, we added a lot of really important stuff.
00:08:55You can, you know, you can, you can change its hat.
00:08:58You can change the, the color.
00:09:01And, and this seems silly, but it's actually really important,
00:09:03and I'll talk about this a little bit more later,
00:09:05but actually understanding when you get a high level view of,
00:09:08when you see your agents working on the canvas,
00:09:10it's important to know which one is which.
00:09:11And so differentiating them is actually important,
00:09:13which is why, of course, we added the leg slider.
00:09:17But so, you know, I can say like, you know,
00:09:20I can, I can, I can say hey to it.
00:09:24And I can say, you know, it's something like draw a cat,
00:09:28and I can have it work.
00:09:29But the most important thing here
00:09:30is that fairies have friends, right?
00:09:32And they can, here we go.
00:09:36And they can, fairies can work together.
00:09:37And so we kind of designed this like multi-agent
00:09:39collaboration system that works on the canvas.
00:09:43And I'm gonna actually, I'm gonna go to,
00:09:46I think one of my, yeah, I think so much.
00:09:49My colleagues agents are here working,
00:09:51making this, this really great scene.
00:09:55I'm gonna bring mine over, summon.
00:10:00And I'm gonna give them a slightly different prompt.
00:10:01So I'm gonna select them all,
00:10:02and now I have a group chat of, of the agents, right?
00:10:04And I'm gonna say, hey, I have a board meeting coming up
00:10:08in like 10 minutes, and I don't have any of my, my figures.
00:10:10Can you draw up like a little memo
00:10:12for all of my financial data for fiscal year 2025?
00:10:16Thank you.
00:10:18Okay, so what's gonna happen there, basically,
00:10:21is this kind of like creates this multi-agent,
00:10:23you know, coordination thing.
00:10:25We have, one of the fairies is writing,
00:10:28writing out a plan.
00:10:30You can see it.
00:10:32And again, the animations are kind of cute and funny,
00:10:34but it's actually really important.
00:10:35I don't have to read a chat or go through,
00:10:37you know, imagine if I have 10 agents working.
00:10:39I don't have to read a chat in order to know
00:10:41what's actually going on.
00:10:42I can look at the state of the agents,
00:10:44and I can actually, you know,
00:10:46I can see what's happening.
00:10:47So we have a task here that's been defined.
00:10:54It seems like, you know, the fairy is waiting
00:10:57for that to finish.
00:11:00Yeah, so that one's bored, that one's waiting.
00:11:03So this is the orchestrator fairy.
00:11:04What it's done is it's assigned the task,
00:11:07and now it's waiting for the other ones
00:11:09to start and finish it.
00:11:10And it's going to get notified.
00:11:12It's gonna get prompted in order to review.
00:11:15It seems like for whatever reason
00:11:16my internet's not working, but thanks.
00:11:18Oh, never mind.
00:11:20So yeah, we have one, we have this one,
00:11:23so yeah, we have one fairy who made the task,
00:11:25one fairy who's working on it.
00:11:26And so this is this kind of, you know,
00:11:29multi-agent coordination system on the canvas.
00:11:31I have, you know, you can see my colleague
00:11:33has his agents over here.
00:11:34They're working as well.
00:11:37And so you can kind of collaborate with people
00:11:38and with agents in this environment.
00:11:41And I don't know, I think that's really cool.
00:11:44The,
00:11:46so the next thing,
00:11:48so the problem with fairies is that
00:11:51they're kind of trapped in the canvas.
00:11:52And all the stuff you've seen before,
00:11:54this requires, if you want to build something like this,
00:11:56this requires like the, you to like opt in
00:11:58and have your entire harness be a like canvas harness.
00:12:02And the downside of that is that it makes it really hard
00:12:04to have any of this work with stuff like outside
00:12:06in the real world.
00:12:07The fairies are, the fairies are trapped in the canvas.
00:12:11And so I, I, I built this experiment.
00:12:15We had a little hackathon internally,
00:12:17but first as a quick motivation for that,
00:12:20at Teal Draw, whenever we have a,
00:12:21whenever we're getting closer to a launch,
00:12:23we like abandon all of our task tracking software
00:12:26and we make just one massive dependency graph of how,
00:12:30like, so this is what an actual,
00:12:31this is a real thing from when we launched fairies actually.
00:12:33And so this is what it looks like when we're like,
00:12:37really like when shit is hitting the fan at,
00:12:39at Teal Draw when we're launching something.
00:12:41And I really like this interface because it kind of lets you,
00:12:45this is not like a special app.
00:12:46This is still just TealDraw.com.
00:12:48You can, you know, move your shapes around
00:12:50and things like that.
00:12:52But I really like this because it both,
00:12:54it lets you see like what depends on what,
00:12:56it lets you know what's coming next,
00:12:57it lets you get a high level overview.
00:12:59You know, these are all green 'cause we finished them,
00:13:00but you know, you can imagine during the project,
00:13:02some of them are in process.
00:13:04And I really want, I really wanted something like this,
00:13:07but something that I could actually,
00:13:09that could actually do the work itself.
00:13:11And so I prototype this thing.
00:13:15It's called the Tech Tree app.
00:13:16And basically it's similar to this.
00:13:17It's a dependency graph.
00:13:19But each of these tasks is a coding agent
00:13:23that you can kick off.
00:13:24And you can have your agent kind of like be running
00:13:27and doing them autonomously.
00:13:30The project itself that's working on,
00:13:31it's this little, this is just kind of like a demo app.
00:13:35But this is, can I, yeah.
00:13:38So this is a little fun, you know, multimodal input thing.
00:13:43I haven't written any of the code for this.
00:13:45This is all written by agents.
00:13:47But I can manage all of the work is being done
00:13:49in this desktop app or in this app here.
00:13:52And so I can do something like,
00:13:53I can see this one has finished building
00:13:56some gesture controls for the canvas.
00:13:58So I can open the PR.
00:14:00And unfortunately, sorry Jeffrey,
00:14:02I am just going to merge this.
00:14:03I am not going to have it be explained to me.
00:14:06But so this is, and so yeah, great, awesome.
00:14:10It looks good.
00:14:10And then, you know, eventually this is gonna get marked
00:14:13as complete.
00:14:15And this is also a multiplayer, which is really cool.
00:14:18And you can have people working together.
00:14:20Yeah, so that's finished.
00:14:21And you can also prompt from like inside the app.
00:14:27You can draw and have a prompt.
00:14:28So I can basically, I can just take all of this
00:14:30and I can draw a little, like, so this is my prompt.
00:14:32And I can wrap it in a task.
00:14:36And I can, you know, call it facial animation canvas control.
00:14:40And then I can assign that to Claude.
00:14:42And I can just hit run.
00:14:43And so now that's working as well.
00:14:44And so this is kind of like, you know,
00:14:46this is kind of something similar to Conductor
00:14:47or OpenAI Symphony, where you're using a kind of like
00:14:50one abstracted interface above what the actual,
00:14:54in order to like manage your multi-agent coordination
00:14:56and things like that.
00:14:57And the thing I like about this also is that because this is
00:15:00multiplayer, one of my colleagues can come and join
00:15:02and add tasks and edit things and see the work that's been going on.
00:15:06So it's much more collaborative than like your own instance of something.
00:15:09Here's the moment of truth.
00:15:12Let's see if that demo that I had it build in the beginning worked.
00:15:20All right, it's, it hasn't built the fluid simulation yet.
00:15:23It's been working for 13 minutes.
00:15:28That's actually fine.
00:15:29So basically this is the TLDraw desktop app.
00:15:31Something that's really cool here is that we have,
00:15:34so this is running locally.
00:15:35It's working on files.
00:15:36We'll see if it finishes.
00:15:37We'll, we'll let this run.
00:15:39But basically what this does is this,
00:15:42this basically exposes the editor instance of the TLDraw app
00:15:44that's running here.
00:15:45And it has a server that lets any agent,
00:15:48for example, my cloud code,
00:15:50write just plain JavaScript against the, against the editor.
00:15:54And, and it basically, it, you know, it's, it's code mode,
00:15:59if you've ever used code mode.
00:16:01But you can basically turn your TLDraw desktop app
00:16:04into a like scripting environment.
00:16:07And the, one of my colleagues actually is, I'm gonna,
00:16:12this is, this is the kind of off the rails bit of the canvas here.
00:16:16Um, or of the, of the, of the talk.
00:16:18So, here's something my colleague made, uh, using the same thing.
00:16:22So this is, he has a TLDraw desktop app in the corner here.
00:16:25And he's using it as his, uh, window manager.
00:16:28Um, and what he did, the way he did this was he just told cloud code to,
00:16:32because cloud code has access to your actual computer,
00:16:34it's not locked into the canvas, uh, it basically, you know,
00:16:37it, it made some rectangles and it, uh, probably wrote some Apple script
00:16:41or something to actually re, uh, you know, move the things around.
00:16:44And so you can kind of make all of these, uh, like ephemeral UIs
00:16:48and have them actually be doing things in the real world.
00:16:51Um, another really cool one that he did was, um, if this loads,
00:16:56it's Pong on the desktop.
00:16:59Uh, let's hit it with a little refresh there and see if it works.
00:17:04Yeah, so this is, he's got in the corner here, you know, you, you have, uh,
00:17:09you have TLDraw running.
00:17:10This is the desktop app and it's using the windows in order to, uh, play Pong.
00:17:14And so again, like kind of crazy, but there's, uh, you know,
00:17:18maybe it seems a little silly, but let's see if this, this worked.
00:17:24Oh, it's still working, man.
00:17:25It's usually much faster, um, but I, I think this stuff is so cool
00:17:29because this lets you kind of, you know, do all of the weird kind of,
00:17:32like, spatial interfaces that you can do, uh, on the canvas.
00:17:35You get all of, like, the primitives of the canvas.
00:17:38Um, but you can, like, you can have your agents working kind of, like,
00:17:41in the real world, uh, it has access to real data.
00:17:45If I scroll up, I'll show that, um, if, uh, you know, this is my cloud code
00:17:52and it found, it, it got the Gmail, it got the Notion doc, it found the spec,
00:17:56and it's going to implement it.
00:17:58Um, but yeah, so to sum up, I think that agents working on the canvas is so cool,
00:18:05and I think that there's, like, so much we can do if we use, like, the agent,
00:18:08the canvas as a, a place to work with agents, and I think the place part of it is really
00:18:13important because, you know, when we do, you know, with remote work, collaboration,
00:18:18we do a lot of stuff online with each other, and we collaborate with people on the canvas,
00:18:22and I think that the, yeah, the canvas can be a place where we, uh, collaborate with agents,
00:18:27and I'm, I'm, I'm vamping because I'm trying to see if this is finished,
00:18:30but I don't think it's going to finish, um, but thank you so much.
00:18:32Thank you so much.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video