Gauntlet Loop Has A Huge Flaw... This Claude Skill Just Fixed That
AAI LABS
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00There's a type of loop that's been hyped up a lot lately and it's called the gauntlet loop.
00:00:04It's the loop that actually does let people build huge apps and proper high quality games and all of it one shot.
00:00:10For example, we gave it a huge product to build and while it took over an hour, it was able to complete the job without any bugs.
00:00:16And the prompt behind all of that is nowhere near as complex as you'd expect.
00:00:20But that's the part everyone is telling you about.
00:00:22What they're not telling you is that this loop has two major issues and those issues are exactly why it's fine for building games
00:00:28and why you can't use it on a real project.
00:00:31So you'd think fixing that takes some whole new technique, but it doesn't because the fix has already been built and shipped in a way you wouldn't expect.
00:00:38If this is your first time, then welcome.
00:00:40We're a software company and this is our channel AI Labs where we show you how to optimize your business with AI.
00:00:45And if you don't have your own, you can use these skills to get paid by optimizing it for someone else.
00:00:50And in this video, we're going over the loop, what those two problems are and how they've already been solved.
00:00:55Now if you already know what a gauntlet loop is, skip to the next section because this part is just the background.
00:01:00But before going to gauntlet loop, you need to know what a loop even is.
00:01:04Before loops, when you build something with Claude, you're the one checking it.
00:01:07You send a prompt, it gives you something back, you look at it, you tell it what's wrong and you keep doing that until it's right.
00:01:13A loop is when you're not in that back and forth anymore.
00:01:16You give the agent the goal and the standard it has to reach and it keeps checking its own work until it gets there.
00:01:21You only see the result at the end.
00:01:23The gauntlet loop is a new kind of loop that's been going around a lot lately.
00:01:27People have used it to build games, 3D worlds and full websites all from a single prompt.
00:01:32But what's behind it is a lot simpler than the results make it look because the whole method is a prompt that's only three lines long and that's what sets the loop off.
00:01:40Now this whole thing started when Matt Schumer posted a demo on X of a first person shooter game and said Claude built it from a single prompt without using any existing assets.
00:01:50That's why the post got really popular because a model producing a complete playable game in one prompt is not what people thought these models could do yet.
00:01:57A few days later, he published how he actually built the game and also gave the method the name gauntlet loop.
00:02:03He could have told the agent to make the game good, but good is something it decides for itself and an agent that decides its own standard passes its own work.
00:02:11So he gave it a real game to measure the work against instead.
00:02:14And once the method was out, other people started running it themselves.
00:02:17Andre Karpathy, who was one of the founding members of OpenAI, said results like these matter because work this high quality was never worth producing before, when the time it took was never worth what you got.
00:02:28A model doesn't have that limit, so people went ahead and experimented with different ideas that pushed the model's boundaries to the max.
00:02:34But before we explore more of it, it would be great if you subscribed to the channel and hit the hype button.
00:02:39This small gesture of support goes a long way for us.
00:02:43Now let's actually understand how this loop works underneath.
00:02:46Schumer published the exact prompt he used and it was just three lines that built the entire game.
00:02:51Each line handles a different part of the build and it's simple enough that you can write the same three lines for whatever you're making.
00:02:57The first line defines what you're building.
00:03:00Schumer's was to build a first-person shooter.
00:03:02And the second half of that line defines what quality it needed to hit, which was the level of the most recent Call of Duty games.
00:03:09He asked for AAA quality, which is what the industry calls its biggest budget games, and he spelled out that he needed it perfect in literally everything from the textures to the physics.
00:03:18The second line is how it should build, and this is the line that decides how the work gets split up.
00:03:23You tell the main agent to break the goal into smaller parts itself and hand each part to its own sub-agent.
00:03:28Now if you don't know what sub-agents are, they're separate agents that the main agent kicks off and each one works on its own small task in its own memory without seeing what the others are doing.
00:03:38And the prompt is very specific about this.
00:03:40The agent has to split the work up by itself so that this hassle doesn't come on you.
00:03:44But the agent still needs to know it's going in the right direction, and that's what the critic is for.
00:03:49The prompt puts a loop on each agent that the main one spawns, and to review the work of each one, it attaches a separate sub-agent.
00:03:56That review sub-agent is called the critic.
00:03:58The critic's only job is to check the work, and if it isn't good enough, it sends it back to the builder agent to correct until it is.
00:04:04The critic never builds anything itself, and it starts with no memory of what came before, so it doesn't know who made the work or how many times this part has already been sent back.
00:04:13It just has to be brutal and critically honest in grading the thing.
00:04:16The third line is the actual quality level that the final output is supposed to match, and that's what tells a critic when it's allowed to stop.
00:04:23Schumer sets Call of Duty as the standard for how the game being built by the agent should look.
00:04:28His instruction was that the critic should literally compare the two blindly and say which one looks better.
00:04:33Blind here just means it isn't told which one Claude made, so it doesn't know which one is the product and which one is the game Claude designed, out of which it just picked the game which was better of the two.
00:04:43And there's one more word at the end of that prompt worth knowing about, which is ultracode.
00:04:47That's a keyword in Claude code, and putting it in is what turns this into a dynamic workflow,
00:04:52which is a feature that runs a whole fleet of those sub-agents at once instead of dividing the work across just a few.
00:04:58And when you run the work like that, that's what we actually call a graph.
00:05:01Now if you don't know what graphs are, they're basically an advanced form of loops,
00:05:05where a lot of different sub-agents are looping rather than one agent looping on its own,
00:05:09which is exactly what the gauntlet loop is doing.
00:05:12Now if you've been watching our previous videos, you'll know we already covered something like this long before it got popular as the gauntlet loop.
00:05:18And what the gauntlet loop draws is a diamond graph, which is one task at the top, splitting out into several sub-agents running side-by-side,
00:05:26and then narrowing back down into a single agent that pulls everything they found into one answer.
00:05:31So the whole concept of gauntlet loop might not be a new thing for you, but the thing you don't know is that this loop's got some major problems.
00:05:38But before we talk about what problems, let's have a word by our sponsor.
00:05:42If you're learning to code, what really makes a portfolio project stand out is real data.
00:05:47Most beginner projects run on fake sample data.
00:05:49The catch is that getting real data usually means scraping, and that's a headache nobody wants.
00:05:54That's where SERP API comes in.
00:05:55It gives you clean structured search results from Google, YouTube, and more in a single API call,
00:06:00and it handles the scraping headaches for you, captures, proxies, all of it.
00:06:04You get millions of real data points as clean JSON and sharpen your data skills while you build.
00:06:09Instead of another to-do list clone, you could build a live price tracker with their Google Shopping API
00:06:14or a trends dashboard powered by their Google Trends API to show what's rising in real time.
00:06:19That's the difference between a project that looks like a tutorial and one that gets you hired.
00:06:24You can start with 250 free credits, just click the link in the description or scan the QR code on screen.
00:06:30And a big thanks to SERP API for sponsoring this video.
00:06:33Despite how many people on X and YouTube are one-shotting entire projects with gauntlet loop,
00:06:38many aren't realizing the underlying problems.
00:06:40There are two major problems.
00:06:42The first is that the main agent is completely responsible for checking.
00:06:45It manages on its own how it spins up the critics and writes their instructions.
00:06:49So basically, you don't have control on the agents nor how the judgment prompt is being passed to the critics.
00:06:55All you gave it was a game to compare it to, and past that, the prompt just says,
00:06:59be a really harsh critic and check it visually.
00:07:01And we've said this before on the channel, checking is something you should set more concretely rather than letting the agent verify on its own.
00:07:08So what it means is that the agent on its own is figuring out everything, and given the scale it is working,
00:07:14you won't be able to figure out what caused the issue in the long run.
00:07:17The second problem is the one that decides whether this method works for you at all.
00:07:21The quality bar in that prompt is an existing product.
00:07:24Schumer set it as Call of Duty, a game that already shipped, so the critic had a real thing to measure the work against instead of deciding on its own what good meant.
00:07:32That's the entire reason this worked.
00:07:34And that's fine when you're building a game, because there's always something you can compare it to.
00:07:38Same goes with a landing page that you're building, or a 3D world.
00:07:42You name what you want to build, the critic goes and looks at it, and it picks which of the two is better.
00:07:46But when you're building something new, there's no existing app to set that bar by.
00:07:51The critic makes up a standard and starts passing work by it.
00:07:54And when you realize that it's a zoom direction isn't what you want, you've wasted a lot of time and tokens.
00:07:59So you come back to a pile of features that all got built in one go, against a standard the agent made up itself.
00:08:05And having nothing to compare the work to is the normal case, not the exception.
00:08:09If you're building the billing rules your own business runs on, there's nothing finished you can hand the critic to check it by.
00:08:15So the gauntlet loop works when there's something close enough to copy, and it breaks the moment there isn't.
00:08:21There's a fix for this, and it comes in two parts.
00:08:23The first is verification, where you plan the checks properly instead of letting the agent invent them, and we covered that in the graph engineering video.
00:08:30The second is giving the loop concrete requirements so that it doesn't drift from what you want.
00:08:35And that's where a skill called Wayfinder comes in.
00:08:37Wayfinder is made by Matt Pocock.
00:08:39He's a software developer who's put out a whole set of skills you can install and use, and each one is built for a different job.
00:08:46Wayfinder is his intensive planning one, and in his own words, it's built on the fundamentals of planning work that he learned back when he was a developer before AI existed.
00:08:54And it isn't tied to coding at all.
00:08:56You can use it for anything you need to plan out.
00:08:59And the reason this skill exists at all is a problem you've probably already run into.
00:09:03When you plan something big with an agent, the destination is clear, but the way to get there isn't.
00:09:08And that stretch in between is what Pocock calls the fog.
00:09:11The agent never tells you when it's in the fog.
00:09:13It fills the gap with its own assumption and carries on planning as though the thing was settled.
00:09:18So what you get back is a plan that looks finished with invented parts in the middle of it.
00:09:22Out of that, it builds a map.
00:09:23Every decision that still needs making goes onto that map as its own question, and those questions get split into two groups.
00:09:30There are the ones you can settle right now, because everything they depend on has already been decided,
00:09:35and the ones still in the fog, because something you haven't looked into yet is holding them up.
00:09:39And that's the part that matters for us.
00:09:41When Wayfinder hits fog, it doesn't guess.
00:09:43It sends the agent off to go and clear the fog, and that can be done through either researching,
00:09:47or building something rough to look at and react to, or a real-world job like signing up for a service so you can judge it.
00:09:54Each one of those is handled on its own, and once it's settled, the answer goes back onto the map and opens up whatever was waiting on it.
00:10:01You work through that map question by question until there's no fog left.
00:10:04What you're left with is every decision written down with the reasoning attached to it,
00:10:09and Wayfinder turns all of that into a single spec, which is basically the one document that lays out what you're building and why.
00:10:15And that's what makes Wayfinder the fix for the gauntlet loop, because just like how Call of Duty worked as a bar for the game,
00:10:21a spec become the same bar and becomes a way to verify what marks the thing you want to build as done.
00:10:27This spec is the exact thing that clears up the planning fog problem we talked about.
00:10:32Every decision in there was actually settled instead of assumed,
00:10:35so when you're building the billing rules for your own business, or anything else with nothing to copy from,
00:10:40the critic still has something solid to work from instead of inventing a standard of its own.
00:10:44So to actually show you how this works, let's take this HR system as an example.
00:10:49People can use this to apply for leaves, file complaints, and more.
00:10:52Now if you want to install the original Wayfinder, you can install it from the GitHub link below.
00:10:56The GitHub repo has an install command that you can run inside Claude Code.
00:11:01But there is a problem with the original skill.
00:11:03Wayfinder is really extensive in how it plans, because it's built to carry you all the way through to a finished spec.
00:11:09That wasn't what we needed. We only wanted the part that gets you to clarity,
00:11:13so that we'd come out of it with a complete plan and nothing else.
00:11:16When you run the original Wayfinder skill, it takes every decision you made during the questioning session
00:11:21and turns each one into a separate requirement file the agent can refer back to.
00:11:26It defined what to build, but what we wanted was a key the agent could check itself against.
00:11:31So we asked Claude to change the skill, so that instead of writing a separate file for every decision,
00:11:36it turns those decisions into a single answer key the agent can keep checking itself against.
00:11:41It rewrote the skill entirely, kept the main parts of the original, and shaped the rest around the different goal.
00:11:47What we ended up with is a much simpler version, one that only writes two files,
00:11:51the map and the answer key, instead of the separate tickets the original produces.
00:11:55But here is another interesting thing. The Wayfinder skill doesn't work by itself.
00:11:59It calls other skills from Matt Pocock as well, but in our version, we've also reduced that.
00:12:03You just need these three skills so you can get these from the GitHub link below as well.
00:12:07For example, the Wayfinder skill uses Grill.me for the questioning part of the planning process.
00:12:12To use it, we opened Claude code and ran the Wayfinder command,
00:12:15because this is one of those skills the agent won't start on its own.
00:12:18Then we told it what we wanted, which was that HR system.
00:12:21And it started questioning us.
00:12:23It asked who the app was actually for, what had to be in it and what didn't,
00:12:27then what done was supposed to look like, and what could go wrong once people were using it.
00:12:31It was a long back and forth that ended at 34 questions.
00:12:34Once we'd answered all of them, it closed the interview and wrote two files into a .wayfinder folder.
00:12:39The first is the map, which holds every decision we made,
00:12:42the reason behind each one, and what the finished thing is supposed to look like.
00:12:46The second is the answer key, and that one is nothing but checks,
00:12:49where every line comes back as either a pass or a fail.
00:12:52So to actually start the gauntlet loop, we needed a prompt.
00:12:55We copied the exact one Matt posted and gave it to Claude.
00:12:58Then you ask it to rewrite that prompt for the app you're building,
00:13:01which in our case was the HR system.
00:13:03The one thing you change is what it measures against.
00:13:06Instead of a game, you tell it the source of truth is the .wayfinder folder
00:13:10that came out of the planning, and everything gets checked by what's in there,
00:13:13exactly the way Call of Duty worked in the original.
00:13:16And Claude hands the whole thing back in the same gauntlet loop format just written for your app.
00:13:20From there, it ran the same way the original does.
00:13:23It planned out the sub-agents, except this time they had the answer key to check everything by.
00:13:27We told it there was no close enough and no shortcuts.
00:13:30It planned what it was going to build first, then the tools it needed to install,
00:13:34then the foundation work so the agents had something solid to build on.
00:13:37After that, it launched a lot of agents at once, each one on a different part of the system.
00:13:42The build took 1 hour and 33 minutes, and it went through about 40% of our session limit.
00:13:47And if we'd been running it on the API instead of our max plan,
00:13:50that same build would have cost around $116, which is a lot.
00:13:54But what came out the other side was the app just like how we'd planned it.
00:13:58Every feature we wanted was working correctly in the app,
00:14:00though it did have some issues, but overall it was the closest to what kind of implementation we needed.
00:14:05The skill we created for this one, along with all the other resources,
00:14:09are available in AI Labs Pro, which is our community.
00:14:12So if you found value in what we do and want to support the channel,
00:14:15this is the best way to do it.
00:14:17The link's in description.
00:14:18That brings us to the end of this video.
00:14:20If you'd like to support the channel and help us keep making videos like this,
00:14:24you can do so by using the super thanks button below.
00:14:26As always, thank you for watching, and I'll see you in the next one.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video