스크립트
00:00:00Hello everybody. How's it going? Okay, I think we still have some people
00:00:20trickling in, but I'm super excited to be here. Okay, first off, let me reload these slides
00:00:31because my cloud code was still building something on it. Okay, so the contrast is a little bit
00:00:41low, so please bear with me. I'm going to try to cover what you cannot read as much as possible.
00:00:49I also have heard that the Wi-Fi is not the strongest. So while it is a workshop, hopefully
00:00:55you just take away a lot of the lessons and then can apply it whenever you want to. But
00:01:01I do have a sample repo if you want to follow along. Okay, first of all, hi, my name is Paul.
00:01:09I'm really glad you found your way into this room. I'm the author of Impeccable. Who here
00:01:15has used Impeccable by any chance? Can I see some hands? Okay, a few people, nice.
00:01:21So for those of you who have not used Impeccable, Impeccable is a skill that I've built for myself
00:01:28plainly. I've built this large enterprise app over the last year and it has lots of different
00:01:34views, states, whatever. And I wanted to design really quickly with my agents, with codecs,
00:01:40with Cloud. But I noticed that even though it gets me quickly to something that I can look
00:01:46like normalizing it back to the design system was really, really challenging. So the first
00:01:51skill that I've built for myself was called Normalize. And it kind of like brought it, brought
00:01:57whatever Cloud design back to the design system. That's how I started. And I also used Anthropics
00:02:01front and design skill, like maybe many of you, when I first got going. And from there on, it
00:02:07kind of expanded into more and more design skills that allowed me to turn Cloud code and then
00:02:13codecs and other harnesses more into a design harness. At some point I decided maybe other
00:02:19people might find this useful as well. So I open sourced it as OpenSource skill, released it, and it
00:02:27turns out a lot of you liked it. So if you haven't checked it out yet, give it a go. It's on
00:02:34Impeccable.style. But today won't be a talk about Impeccable per se. It will be about what I
00:02:41learned from making these skills because it kind of escalated. It started with like a simple prompt and
00:02:46then went all the way to what it is today. A lot of people have looked at the code of Impeccable and
00:02:53they see like a whole bunch of scripts in the script folder and they're like, what is all this stuff? And
00:02:57so I wanted to share some of my knowledge that I've gained with you. So let's get into it. Let's talk about
00:03:05the dark arts of skill engineering.
00:03:10Whoa!
00:03:16Okay.
00:03:19Yeah, the production value of this talk is out of this roof.
00:03:23Okay, so first of all, you've all seen this kind of design.
00:03:26This is actually a real design built with the front end design skill and clawed code.
00:03:34And you probably all have seen a design like this. This is for like a fake
00:03:37kids reader, iPad reader app. You have italic serif, you have like some capitalized
00:03:45hero, and you have like a eyebrow text, like a kicker, whatever you want to call it. The top of it, like a weird label.
00:03:53You have beige, I call it clawed beige, clawed beige backgrounds.
00:04:00And now it's not necessarily a bad design, right? But I think you all can point this out and say
00:04:06like, well, this is clearly AI generated. It's clearly slop. And it turns out slop is a moving target.
00:04:12You've all seen, you might have thinking, might have been thinking of slop as purple gradients,
00:04:17but we've kind of moved on since that, into clawed beige.
00:04:22Okay, so this is where I started. I started with a system prompt and a prayer, basically.
00:04:28So I started using the front end design skill, like many of you, and it's like 55 lines of named bands.
00:04:37No scripts, no routing, pure prose, right? And you just hope for the best.
00:04:43That sometimes worked, and most of the time it didn't. So for example, I'm just going to read some
00:04:49of this. So if, I mean, maybe it's really readable, I don't know. But for example, in the front end design
00:04:55skill, you have sentences like, never use generic AI aesthetics, overused fonts like inter, Roboto,
00:05:01Arial system fonts, or cliched color schemes, particularly purple gradients and white backgrounds.
00:05:08Never converge on common choices like space grotesque, for example. Now, there are two problems with this
00:05:14approach. The first one, it over applies, and then a band just relocates the model to the next cluster.
00:05:21And I'll show you why this is a problem, a little bit further down the road. But really, you tell it
00:05:25not to use inter, it just uses the next best font it finds in a latent space. And so it doesn't actually
00:05:31make it more creative. It just, again, this is why I said the slop is a moving target, it kind of picks
00:05:36the next best thing. I have learned my lesson here, the hard way, because I don't know if you noticed,
00:05:44but the reason why we got purple gradients in the first place is because of Tailwind. Tailwind's default,
00:05:50sample pages, theme, whatever, it was purple. Well, it turns out I've turned the web orange before,
00:05:56many years before that. I created a framework called jQuery UI, and the first default theme of jQuery UI
00:06:02was orange. So overnight, I call it the web orange. I thought people would modify the theme,
00:06:07but now they didn't. So I learned my lesson. Okay, the median is the model's gravity. Even 250 lines of
00:06:17like artisanal, crafted, beautiful skill pros cannot change this. It's just not enough. It doesn't help
00:06:24enough, right? It's nowhere near enough. And I learned this the hard way, and hopefully you don't have to.
00:06:31My overall thesis for this talk is that prompting is sort of like the starter level,
00:06:39but harness engineering is where you should end up. Like you should reframe when you're building
00:06:44skills, you should think about, okay, skills the same way as MCP is an extension to the coding harness,
00:06:50or whatever harness you're in. It's not just a prompt that you package. It's something more than that,
00:06:57or you should at least conceptually think about it more than that. It is extending the harness
00:07:01of whoever is using that thing. And it also has more capabilities as just prompting. And when I
00:07:07thought about that way, it sort of like clicked for me. Prompting is a spell harnessing is magic.
00:07:12And we'll talk about nine different dark arts today that I learned in the process of building
00:07:19Impeccable. We'll make sub-agents argue with each other. We'll talk about how to force divergence as
00:07:27opposed to convergence. Routing like a model. And basically, you've seen this probably. Most modern
00:07:35models are mixed of expert architectures. And Impeccable is built like a mixture of expert skill.
00:07:43We give the memory. We'll create scripts that talk back. And I promise this will make sense. Hooks that
00:07:50fight back. Live wire the browser and use more of the harness. Compile to every harness and design for the
00:07:56weakest model. Let's get into it. Number one, make it argue. So, if you're building something like a
00:08:04critique skill or code review skill, here's one huge issue. If you are, I mean, you've probably noticed
00:08:11this when you're working with Claude Code or Codex, that doesn't matter. But if you ask Codex or Claude
00:08:17Code to review its own work, it will usually rate it as very high. Like, I mean, like, I've built this. I've done a great
00:08:25job, right? Like, it's like creating your own homework. It doesn't make any sense, right? It anchors on what it
00:08:32already created. Now, that's not great. What can you do in order to solve this? Well, you can make
00:08:41a model argue with another model, right? Adversarial prompting is also called. So, you have two sub-agents
00:08:47components, and they never see each other's work. And here's why this matters. So, in impeccable, there's a
00:08:53critique command that actually critiques your design. And you can point it to your landing page. You can
00:08:58point it to anything. And there are two particular failure scenarios. The first one, and this is almost
00:09:05impossible to read, so I'll explain it. The first one is a strong page, right? So, it's a really good
00:09:11looking page. But there's a whole bunch of, maybe, like, deterministic errors. And impeccable actually
00:09:18has a deterministic engine, like a design linter, that can detect things like bad contrast. It can
00:09:26detect things like, you know, too many fonts. You know, maybe things that are too close to the edge of
00:09:32an element. So, it detects some of, I would say, polish issues. But you could have this really beautiful
00:09:39website and then the detector runs. And it's doing that as part of the same skill and the same model
00:09:45thread. And then the model just sees the detector output and says, well, I guess there's 500 issues.
00:09:52Therefore, this design must be bad. Now, that's one. The other one is the opposite. The other one is,
00:09:58it's actually a really terrible page, or maybe an empty page. But there are no detected issues by the
00:10:04deterministic detector. So, the model is like, hmm, we didn't find any issues. So, this must be
00:10:09great design. So, both of those are not amazing. What you want is, and this is what
00:10:15Impeccable's critique skill does, it kind of combines two things. It spawns two sub-agents
00:10:23and they are blind to each other. And that's how you get to a balanced critique. So, the first sub-agent
00:10:30does, actually, like a design director. So, it's an LLM that acts like a design director. And so, it looks
00:10:38for hierarchy, it looks for slop, it looks for heuristics. And so, it does a critique the
00:10:43way human would with the browser tools that are available to it. The second sub-agent runs the
00:10:50deterministic detector and also collects browser evidence. And then, once both of those results come
00:10:57in, the main thread synthesizes both into one critique. And that produces a much more balanced result.
00:11:06And before I go on, I realized I actually have not shown you where the sample repo of this lives. So,
00:11:15let me bring this up real quick. If you want to clone this and you have a decent enough internet
00:11:20connection, go ahead if you like. So, this is P-B-A-K-A-U-S slash impeccable minus talks.
00:11:31The talk lives here, but also in the dark arts folder, there is a starter folder and a demos folder.
00:11:40Demos has a pretty average median page that you can manipulate. And then, in the starter kit,
00:11:50you have enough to build sort of a mini impeccable if you want to, if you want to follow along or try it
00:11:55out yourself. And so, as part of this, this tutorial here too, you'll follow along sort of the checkpoints,
00:12:03the dark arts, and build something yourself. I would suggest, you know, if you like to multitask, great.
00:12:10You can apply this, by the way, to anything. You can do like a code review thing. You can do,
00:12:15it doesn't have to be the design skill, but I wanted to have something for you to play.
00:12:21Back to the deck. So, two blind opinions beat one confident guess. You can use this, again,
00:12:30I already said code review design review, but also security audits ones is a good example.
00:12:35Or a plan, creating a really good plan, RFC critique, where you have multiple LLM judges argue with each
00:12:41other before it gets good. Or ranking outputs is a good example. Now, here's a problem though.
00:12:50Codex, why? Why you don't let me do this? It's bad. It turns out a codex never created these
00:12:58sub-agents when I first tried this. And I banged my head against the wall. I'm like, why is this?
00:13:04It turns out codex has a different permission model than Claude Code and other harnesses. In codex,
00:13:11you have to explicitly, as a user, request the use of sub-agents for anything in the harness to use sub-agents.
00:13:19So, if you're distributing a skill, you're out of luck. The only way to make this work, as far as I know
00:13:25today, is to actually tell the model, okay, if you have sub-agents capabilities, but you do not have
00:13:33permission, please stop right here and ask the user. And so, that's pretty much the only way you can get
00:13:40codex to comply. So, in impeccable, if you see something that makes you go, huh, it's probably because of
00:13:47that through lots and lots of issues that people filed and a lot of testing on my end. Also,
00:13:56a lot of this obscure knowledge got into the skills so that it works truly across harnesses.
00:14:01For instance, here you see kind of pseudocode of how this would work. And then also, this is another
00:14:08really important thing. Very often, if codex realizes it can get away with something, it will do it. So,
00:14:19if there is no punishment for not spawning sub-agents, it will simply not spawn them. It's like, well,
00:14:24this is the easier route. I will take this easier route. So, what you have to say is, actually, if you
00:14:30cannot use sub-agents, you must say that you are giving the user a degraded experience. And codex hates that.
00:14:39So, use that to your advantage. So, you can watch them argue. Now, I did not pre-record an
00:14:49actual example here because I'm like, let's do it. So, we're going to go into cursor and
00:14:54I'm going to do like critique and let's hopefully it'll work. I don't know if Composer sponsors
00:15:11sub-agents well enough. But let's see. Composer, by the way, if you haven't used it, is a really fast,
00:15:26well-balanced model. So, it's kind of neat for work that you want to show on stage in particular.
00:15:35Okay, so now it's doing something here. Okay, this repository, I think, has an old version
00:15:45of the back of it. It doesn't spawn sub-agents. I see this is the unfortunate. Well, maybe it does.
00:16:01I'm not sure if it did or not, but I at least want to show you what the type of critique looks like.
00:16:05All right, now it's asking a bunch of questions of what I actually want to create. I'm going to
00:16:24skip this. And now I get a design critique on what's working, what the priorities issues are,
00:16:33persona red flags. Now, the actual thing that I wanted to show you, unfortunately,
00:16:38couldn't be seen in this particular set, but we can come back to it.
00:16:41If you run this in cloud code or codex on the most recent version, you should very clearly see,
00:16:47I mean, cloud code is very easy to see the sub-agents running and doing its work. So,
00:16:52it will spin up two sub-agents and you see it at the bottom of the cloud code thread doing its thing.
00:16:59Okay, level number two. A ban just moves the problem. We talked about this already.
00:17:05You ban into the model graphs, the graph space grotesque. How do you solve that? How do you force
00:17:10divergence? Well, a ban only moves the model around inside its own cluster. And what I've built for
00:17:17Impeccable and for a bunch of other skills that I've released is what I call an anti-attractor.
00:17:23And the anti-attractor works by creating a random seed of sorts. And that can come from user input,
00:17:30or it can come from a script that it can run that produces something that is completely unexpected to
00:17:36the model. Because that's what you want. And so, in this case, for instance, it would be font selection.
00:17:42And instead of selecting this save next step prediction font, it went into a completely different
00:17:48space through a different seed. There are three techniques that, you know, are easy to hard and
00:17:56work differently. The first one is something you can do right now. It's the most simple one. And it's to
00:18:04shave the save picks. So basically, tell the model, okay, name your top three fonts. And then the model
00:18:11is like, okay, I got the top three fonts. And then you're like, now throw them away. And you're like,
00:18:16but why? But then you basically shaved off the next token that is predicted. And you do that three times.
00:18:27And then now you get to a different space, like a further away in the latent space, right? Now that's
00:18:33doable. But now at some point, you still get convergence. So this is just, you know, a limited
00:18:38technique. The second technique is to generate a lot of different things and then have a sub-agent rank.
00:18:46I've done this for a shader library that I've created called Radiant Shaders. And the goal here was to
00:18:52create around 100 different shaders. And the problem is every time I would say, you know, create a new
00:18:57shader or create ideas for 10 new shaders, I would get the same repeating ideas. I solved this in two
00:19:05different ways. The first one is I created something unexpected, a random seed, a creative seed. In this
00:19:10case, I used celebrities. I said like, well, what would Rihanna look like as a shader? Or what would
00:19:17Beyonce look like as a shader? And then the model was like, hmm, let me think about that. So that's
00:19:24the first thing. And then I said, well, generate 100 of these ideas and then spawn a sub-agent
00:19:29that ranks all of those ideas. And that's important. It has to be a sub-agent because the sub-agent
00:19:36doesn't know anything from the prior context of the session and can then completely change the order.
00:19:43The third one is to create a random seed from a script. So in Impeccable, for example, when you
00:19:49first start a project, it calls a script called color.js. And color.js has over 100 hand-selected,
00:19:59they're not complete color palettes, but they are primary colors and they're kind of a starting point
00:20:05of a palette. And it reads that and then the model uses that as a creative spark to build a palette
00:20:12around it for you. You can still say, oh, I don't like this. I mean, I don't like what it proposed,
00:20:17but it turns it into a different direction. So those are all ways to create divergence.
00:20:24And when you design something with Impeccable, the same brief, depending on the user's input and,
00:20:31you know, color script that runs, et cetera, can produce vastly different results because of that.
00:20:36Because I didn't want to have the whole internet look like everything else.
00:20:41Number three. Here's the problem. If you cram everything into one skill, it kind of blurs them.
00:20:49The instruction following becomes not very good enough anymore. So if you're building some general
00:20:54purpose skill and you expand it and expand it and expand it, at some point, it becomes really,
00:21:00really blurry to the model. Here's a concrete example of this. The Anthropic front-end design skill,
00:21:08the former version of it, they just shipped a new version three weeks ago. But the former version
00:21:13had a line that I read earlier that says avoid system fonts. That's okay for landing page design.
00:21:21But for product UI, oftentimes you want it to feel as native as possible. So system fonts are actually
00:21:26the thing that you want. So how do you solve this? You can say, I mean, you can have this giant if/else
00:21:32block in a skill. Say like, well, if the user wants a landing page, do this. If the user wants a product,
00:21:37do this. But that becomes really convoluted, waste a lot of tokens and honestly doesn't work very well.
00:21:44So Impeccable started as a lot of different subskills and now has this mixture of experts model
00:21:51that routes internally. Both in terms of capabilities, so you can call Impeccable critique or Impeccable
00:21:59polish. And you get a different MD file loaded behind the scenes for a particular job. So it's not just one
00:22:05giant skill MD. But also, and this is something not a lot of people know, behind the scenes,
00:22:12Impeccable decides based on your brief and what you input, whether you're trying to design something
00:22:19brandy, so like a landing page or something that wants to attract attention, or whether it's the
00:22:24actual product that you're designing. So it switches registers and then loads completely different rules
00:22:31for those two registers. Because product design and brand design are very, very different. So that's also
00:22:36something that I would recommend you doing if you're building a larger skill. This works for big multi-tool
00:22:42skills, works for context on demand type of skills, per audience behavior, agent toolkits, that kind of thing.
00:22:51Number four, everyone starts from zero. Skills by default don't have long-term memory. They don't
00:22:58really compound over time, but you can make it so. So you have a skill folder and you can save things in
00:23:04that skill folder. In fact, in Cloud, you even have an environment variable that resolves to the actual
00:23:11directory that you can save things in, which is nice. No other harness supports this right now, I believe.
00:23:18But you can hack around that. Impeccable uses a .impeccable folder in the current repository route.
00:23:25But you can also save things directly in the skill folder. And maybe ask the user to ignore them.
00:23:32How could this work? So for example, if you're running a critique in Impeccable, that critique is
00:23:38saved as a file in that folder. And by default, it's getting ignored. But then, if you then later on say,
00:23:46well, okay, I just ran a critique. I'd like to polish my page. Even if you do it in another session,
00:23:51it actually uses that prior critique as a signal to understand what have we found out about this page.
00:23:58And it can look at all prior critiques and see sort of the progression of the page. For example,
00:24:03you could have said in one of the critiques, you know, I don't agree with this critique.
00:24:07I don't think you're right. And I think I really like my instrument zero fonts.
00:24:13And then the model would be like, okay, no problem. I'm going to mark this for later.
00:24:16And the skill is now smart enough. The skill has built context to realize, okay, well,
00:24:23that's the user preference. So I'm going to respect it going forward. So compound engineering
00:24:29really is an interesting theme for skills as well. You can make skills aware of prior sessions with that
00:24:35technique. So make the runs compound. This works really well for resumable gradients, resumable agents,
00:24:43progress tracking, multi-session refactors, migrations, that kind of thing.
00:24:49For instance, one of the things that I do all the time is refactor my code. And how do I do that? By
00:24:55having a sort of like a skill that spawns itself across multiple sessions and tackles one file at a
00:25:03time. So I basically tell it, okay, here's your TSX file or whatever for today's session. And now refactor
00:25:12everything around this file and linking into that file. And then it sort of builds up context over time
00:25:18until it's completely finished with the whole code base. Okay, number five. Buried rules get skimmed.
00:25:27We talked a bit about this before, but this is a little bit of a different point I'm trying to make.
00:25:34Now, especially with weaker models, and now if you're building a skill for yourself and you're only
00:25:39running Opus or you're only running Codex, this isn't that big of an issue, right? You know which model you
00:25:46run? You know, if it works with GPT-55, for example, I'm good because that's the only model I use. Now,
00:25:54if you want to distribute your skill to lots of users, this is where things get kind of hairy.
00:26:00Because some of those users might be running Sonnet. Some of them might be running Haiku. Some of them
00:26:05might be running Grog. I don't know. Some of them might be running Gemini. You never know. Sometimes I
00:26:13meet somebody who does. But really that's where it gets complicated, right? Because you need to build for
00:26:23the lowest common denominator. And ideally for the one model that is the weakest at instruction following.
00:26:31For example, GPT-5 mini is not a very good rule follower. There are things, even in Impeccable,
00:26:38that don't work with GPT-5 mini. It consistently doesn't load certain MD files that are sorted.
00:26:45It consistently doesn't spin up the live mode. So there are boundaries to instruction following
00:26:51across these models. And it gets especially bad with longer skills that have lots of rules.
00:26:57So how do you work around this? Well, in Impeccable, Impeccable really is kind of bionic of sorts.
00:27:03It's really not just prose. It is a combination of scripts that run inline within the skill at certain
00:27:10times and then prose around it. For example, every time you call Impeccable, it runs a file called
00:27:17context.mjs. And the context.mjs does a couple of things. The first thing is if there's a product MD,
00:27:26which is Impeccable, it's almost like design MD, but it is for product strategy. So it wants to
00:27:32understand who's the target audience or what do you want to achieve with this thing, which is oftentimes
00:27:37more important in a design interview than, you know, how round do you want your borders to be?
00:27:45But it supports both. It supports product MD and design MD. And by default, context.mjs brings these
00:27:51files together and then spits them into the session. Now that's not exciting. But when those files are
00:27:57not available, it will actually give the skill structured JSON and say, by the way, there is no
00:28:04product.md. And here's exactly what you should do about it. Or here's another thing that context.mjs does.
00:28:12It actually makes Impeccable self update if there's a new version of Impeccable. Now with your permission,
00:28:17so we will ask you. But it will say, hey, by the way, there's an update available for the Impeccable
00:28:22skill. And here's what you should do now to ask the user whether they want to update Impeccable.
00:28:28So it's overloaded in many ways. And it will always tell the model the exact instructions on what to do
00:28:35next. And the really interesting thing about this is that I found that that works significantly better
00:28:41than some random rule in the pros of the main skill. When you put something out from the exit value,
00:28:50from the standard out of a script, somehow the model will follow it a lot more than before.
00:28:58So that could be environment-aware setup, dynamic onboarding, repo-state gating, adaptive flows,
00:29:04anything really. Actually, before I end this session, one of the shortcomings of this technique,
00:29:11and this is something to be aware of, is prompt caching. So this works super, super well to keep a
00:29:18skill. Sort of like, you know, flowing in the right direction, instruction following. But it does so at the expense of
00:29:26prompt caching. If you need prompt caching, if you run this skill many, many times, and you want the whole
00:29:33thing to be cached, this is not a good technique to use. But I found it to be very useful in really interactive scenarios.
00:29:41All right. Number six. Hooks that fight back. It's something I shipped quite recently, and I really like it.
00:29:50I want to show you what I mean by that. So a lot of people have impeccable systems,
00:29:59but sometimes they forget to run it. Sometimes they're like, you know, I don't know. I mean, I wish,
00:30:04Codex is actually pretty good. Some of the harnesses are pretty good, consistently looping in the right
00:30:09skill. But because it now bundles as one skill, oftentimes the harnesses forget to simply call
00:30:18impeccable when you don't explicitly mention it. So now you're building some front-end code and maybe
00:30:23doesn't follow your design system or whatever. Now that can be solved with hooks. Who has used hooks
00:30:31before in Cloud Code or Codex? A few people. Okay, nice. So this skill that I've built here,
00:30:41Impeccable, ships design hooks. So I've basically built a design linter that runs under the hood and
00:30:48ships with the skill. When you install Impeccable, these hooks install into Cloud Code, Cursor, Codex,
00:30:56and GitHub Copilot. And they will keep the model exactly where it needs to be. So the hooks come to
00:31:05you. It's a guardrail that fires on every edit. And there are some differences between the different
00:31:14providers here. So the hook syntax for Codex and Cloud Code is not the same. And also the behavior is not
00:31:21the same. So for instance, we found out that with weaker models, slightly weaker models like Composer
00:31:26and Cursor, you kind of want to use a pre-tool use hook that prevents writing of code as opposed to a
00:31:34post-tool use hook. Post-tool use basically happens right after the agent has written a file, for example,
00:31:42and then it tells you, hey, by the way, like the contrast of these colors is bad. Or, you know,
00:31:48you have a purple gradient in here. And then ideally the model is smart enough to actually fix it.
00:31:54Some models don't follow those instructions very well. And so if you do a pre-tool use hook,
00:32:00you are actively preventing the writing of this file in the first place. So it's a much more heavy-handed
00:32:06approach. But we needed to do that for certain models and certain hardnesses. But this is nice.
00:32:12And what's even nicer about it is that you can personalize it to your design system and your use
00:32:17case. Or whether, let's say you use it for code reviews, you can personalize it with your own ESLint rules,
00:32:23with your own syntax guidelines, et cetera, and then expand it from there.
00:32:31So passive guardrails beat a command no one remembers to run. So these are passive guardrails that always
00:32:39keep you in the right lane on track. Again, that works for linting, for formatting. Of course,
00:32:46if you're using cloud code or codex, it already uses some of the linters for things like syntax formatting.
00:32:53But design linting is a whole different game. But I would really encourage you to
00:32:59try out hooks in combination with a skill and think about, okay, well, my skill does this.
00:33:05How can I create a feedback loop, a validation loop that uses hooks to actually keep me on the right lane?
00:33:16Okay, so here's, it's hard to show hooks in action. But, you know, if you can see this,
00:33:22this is roughly how it would happen in an agent. So for instance, in this case, I would use, let's say,
00:33:30Gemini does this all the time. Gemini creates animations on images like crazy. It will animate any
00:33:38image and it will usually do a hover zoom in effect. It loves that. And that's something that impeccable
00:33:45flags. And in this case, the hook would fire silently, usually. That's why I built this fake demo because
00:33:52you can't usually see it. And then it will tell the model, hey, by the way, here was a violation.
00:33:58And the experience of this is that oftentimes you don't have to do anything. The model just
00:34:01course corrects and fixes itself. Now, one important thing, if you do this and you ship it to users,
00:34:08very important to add a way to create ignore rules or something like that. Because oftentimes these hooks
00:34:16have false positives as well. And you want a way to configure those hooks. Otherwise, it gets really
00:34:21annoying very quickly. Impeccable ships with these design hooks that allow you to create ignore rules
00:34:28at a file basis within a CSS rule. So like many granular levels to exclude certain files, for example.
00:34:37Okay, level seven. Now, you can't really tune pixels to a chat box. Now, this might not be relevant if
00:34:45you're not building a design skill. But I think the general point is relevant. So if you think about
00:34:50a skill as harness engineering versus prompting, then you think about the harness as a whole, right?
00:34:58You're living in Cloud Code, for example, or you're living in Codex, or you're living in GitHub Copilot. Now,
00:35:05what are the capabilities of that harness that you can exploit to make the best user experience for your
00:35:10use case? That's the question you should ask yourself. For example, Codex on desktop now has an in-app
00:35:20browser built into the actual app. Can you use this in-app browser in some interesting ways? Can you use
00:35:28the browser screenshot tool in some interesting ways? And in my case, I could. I realized, hey,
00:35:35there's probably a way to connect the in-app browser and spin up the development server and just load the
00:35:42page there and then kind of connect it to the main thread in some ways so I can allow the user to
00:35:49visually iterate on that page instead of in the chat. And so, in Impeccable, what this looks like is it's not
00:36:01using MCP. It's simply spinning up a live polar, a little server that looks for input and inserts a
00:36:09snippet into your development server. It then, on the page, when you do something on the page, it sends an
00:36:16event back to that actual polar using server-side events. And then, and this is, I think, the clever
00:36:25bit maybe or the bit that makes it all work. The polar then stops. So, the polar ends itself. There's a
00:36:34standard out message. We talked about standard out before, right, the exit value of this thing. And
00:36:40the model reads that message and realizes, oh, something happened. I better do something. So,
00:36:47in this case, in the skill itself, I give it instructions on how to handle this event. I say,
00:36:52like, well, if this event comes in, you should probably build some design for this particular
00:36:57section of the page. And then, you should send it back to this polar so that it arrives on the user
00:37:04site. And so, this is a direct connection between one harness capability and another harness capability.
00:37:11So, the chat thread and the in-app browser. And, yeah, this is kind of how it looks like on a diagram.
00:37:19But I think the best way to experience with it is to see it. So, let me bring this up.
00:37:34Okay, cursor. I think I'm already in live mode here. Okay, so, I booted up live mode already. I'm
00:37:44now in picker mode. I get this little bar here at the bottom. And as you can see, I can pick anything
00:37:49on this page. I now get this little overlay bar. And I can select all sorts of sub-commands within the
00:37:57skill. So, these are basically translating to MD files that live within the skill. I can select the
00:38:05amount of variance I want. And then, I can hit go. And now, here in the thread, you can see that it
00:38:14picked up the actual signal in a main thread because the polar stopped. And it now knows, hopefully,
00:38:22exactly what it needs to do to first wrap this element in some special tag. Then, it knows how to
00:38:29create variants that are marked up in a special way with CSS. And now, it did that. So, now, as you can see,
00:38:38the thing updated immediately, I now get these three variants. And I can click through. And then, if I
00:38:46like one of them, I can click accept and accept it. If I don't like one of them, I hit escape and I'm
00:38:52back in this normal mode. So, this shows sort of like how to exploit a harness capability in an effective
00:39:02way for one problem space, in this case, design. You can also insert elements with this thing.
00:39:09And so, click into anything here. You can draw on top of this and leave comments. You can leave
00:39:15annotations if you want. You can dictate. You can steer the whole page by simply writing into this. And then,
00:39:25again, this goes back to the main agent. And it becomes a steering signal for the whole page.
00:39:31And you can also visualize lots of things this way. I mean, you might have read Tharik's blog post about
00:39:37this, about how HTML is a really cool way to communicate as opposed to Markdown. I agree. And I
00:39:45think also, like, DesignMD is much better visualized as HTML. In this case, you see the DesignMD of this,
00:39:52you know, not great website for demonstration purposes. But you can use this to advantage as well. If you
00:39:59hijack the in-app browser and use it to your advantage. So, this is how I make use of it.
00:40:06Okay. Number eight. It worked on my machine.
00:40:14Well, I mean, everybody who is a developer here knows this problem. This hits really hard when you
00:40:20ship a skill. There are so many times I kind of saw this argument on X. It was like, hey, bro, just
00:40:29symlink. Just, you know, symlink.clawd and all your problems will be gone. Well, that's great if you're
00:40:36building a simple skill and if you're doing it for yourself. By all means, go for it, right?
00:40:41Symlink your CloudMD to agents.md. Amazing. Like, symlink the shit out of everything.
00:40:47But it's not great if you're trying to ship a skill to lots of users. Because, again, we just talked
00:40:53about a whole lot of differences these harnesses have. And I'm going to talk about more differences.
00:40:57And I know it's annoying because it would be great to symlink those things. But unfortunately,
00:41:02we don't live in that world. And unfortunately, Anthropic has still not adopted agents.md.
00:41:08So what are the actual differences? For example, we talked about subagents already. We talked about
00:41:15how, well, on the bright side, they're widely supported now. But who can spawn one is very,
00:41:21very different. So with Cloud, you can programmatically do it very easily. Codex needs the user okay.
00:41:29In Cursor, it's agent-choosen most of the time. So there are clear differences. Also, if you want
00:41:35to pre-define these agents, Codex has a different syntax for that than Cloud and Cursor, et cetera.
00:41:44Another one is the Ask User tool. So one of the coolest tools in the Cloud Code Harness is the Ask User
00:41:52Question tool. It's a really nice tool that you can use to ask the user a question, right? It brings up this
00:41:57menu. So like, hey, what would you like to do? And then you pick some option. Well, turns out, Codex has
00:42:04a tool like this. That's the good news. The bad news is that tool is only available in plan mode. So again,
00:42:11big differences between how these things work. And what does that mean? That means that if you're not
00:42:17running Codex in plan mode, but your skill wants to ask questions, most of the time it simply doesn't. It will
00:42:24simply infer from the current context and not ask any questions to the user, which is not great. So
00:42:31there's a lot of sentences in the impeccable skill that specifically say, if you're Codex, you have to
00:42:37stop and ask questions. No, you're not smart enough to infer the context. So if you see lines like this,
00:42:44that's why. Another one is background jobs. And it's also something you learn through the hard way by
00:42:51doing this. For example, this live mode that I just showed you, it's spawning a background task. So it's
00:42:57running a shell in a background task. And that's cool because you can keep using the session.
00:43:05And then when the background task finishes, the model is automatically waken up, gets the message back,
00:43:10and then can do something and react to it. Where Codex cannot. Codex and other harnesses do not react
00:43:17when a background task finishes. You actually have to manually say, hey, by the way, this background task,
00:43:22can you take a look at what it did? And that's not great, right? If you were doing an automation like
00:43:27this. So there are differences in how these tasks are spawned and how they work. So that's why if you're
00:43:34using the live mode in cursor or in Codex, it creates a foreground task and it keeps the actual chat thread
00:43:42blocked. Not ideal, but it makes it actually work. So there are subtle differences on how these tasks
00:43:48are spawned. Watchers is another example. Tayden watch exists now. That's really cool. I mean,
00:43:53most of the harnesses have a way to watch, for instance, a log file. But those are throttled
00:43:59away harder than simply spawning a background task. Edit hooks, we talked about this already. They are
00:44:04different. And so lots and lots of behavioral differences. But there's also model differences.
00:44:11So for example, in my case, they all have different tells in the ways they're overfitted.
00:44:18For example, Gemini, again, I mentioned this, loves to animate pictures. It just loves it.
00:44:24You have to tell it not to animate pictures if you don't want a hover effect on every picture.
00:44:29It doesn't matter where it is. It loves it. Codex loves bad letter spacing. I don't know why,
00:44:36but it does. Codex also loves extremely rounded borders. It will round anything you thought it.
00:44:44It loves it. It doesn't matter if it's a hospital website or a kid's website. It also loves hairline
00:44:49borders. And so there are specific tells that are unique to every model. And that's not just for
00:44:55design. It's for architecture. It's for code architecture. It's for preferred NPM packages.
00:45:02Every model is overfitted in different ways. Finding out how to overfit it usually happens by accident.
00:45:09In my case, I have a pretty extensive evals harness that I run behind the scenes. In fact, every line
00:45:14of Impeccable is ablation tested. So I test every single line and see what it does across all models.
00:45:19I don't expect you to do that, but it is very good to know that the models are different and are
00:45:26following instructions differently and the behavior, the harness behavior is different as well.
00:45:32And so what Impeccable does, it creates harness specific and model specific builds for every single
00:45:41model. You might not have to go all this way for your own purposes, but I just wanted to show you
00:45:47how far you can go with this. For example, it actually has a substitute variable that picks the right
00:45:55user question tool, depending on the harness. Or it has these XML blocks for Gemini, for codecs, etc.,
00:46:03that will actually insert specific overfitting avoidance rules for the given models. Because
00:46:09it turns out, if you tell Claude not to let us space too much, it will let us space in the opposite
00:46:16direction. So you can't just include it all in the same skill. And that's why, you know, you can,
00:46:24you know, if you instrument this way enough, you can actually get to this write once, ship to all of
00:46:29them skill that actually works everywhere. It's a lot of work, but it does pay off and allows you to
00:46:34create beautiful pictures like this.
00:46:39Now, the only other problem is that typical install methods, like for MPX skills, for instance, if you've
00:46:46been using MPX skills, do not honor different directories for different harnesses. So they actually
00:46:53just take the first directory and then copy it or symlink it into all sorts of folders. That's why,
00:46:59if you go to the impeccable website, I've built my own CLI to solve this problem. That's why it doesn't
00:47:05use MPX skills. So I think the community hasn't quite yet gotten to the point where this is an accepted
00:47:12idea. And it's annoying. I get it. It's annoying to compile for different harnesses, but I found it
00:47:17worthwhile. Finally, again, built for the lowest common denominator. Our weaker model has opinions just
00:47:25fine, but what it loses is the discipline to follow yours. So Codex, for example, and GPT,
00:47:35specifically loves the word gate. If you've built a skill in Codex before, it loves gates.
00:47:43Whenever you say, hey, why didn't you follow these instructions? You're like, well, I think we need
00:47:46a gate. So I gave it what it loves the most, gates. But I only do that for Codex. So there's a CodexMD
00:47:55that gets loaded on the fly for Codex and GPT. And then it actually follows like, you know, okay, here are your
00:48:05eight gates. You have to pass every single gate. And you are not allowed to compress those gates.
00:48:12That's really important because it loves compressing these instructions as well. Just skim over it and
00:48:16say like, well, I guess I do one and two and five and good. And so the way you solve this is by actually
00:48:23having it log every single result of every gate and say like, well, I just passed gate one. Great success.
00:48:31And the most important lesson from this is if the gate can be skipped, it will be. I mentioned this
00:48:36before, right? If the model can wiggle itself out of a difficult situation, it will absolutely do that.
00:48:43It will not do all the things it needs to do to complete the end result. So be careful. Make it unskippable.
00:48:53So we just built a harness extension. We went from prompting all the way to building a monster.
00:49:02But I think it turned out to be pretty powerful in my case. And I wanted to share what I've learned on the way.
00:49:09I don't expect you to use all of those techniques. I think some of them are pretty exotic and maybe not
00:49:14applicable to every use case. But I hope that you find value in some of the advice that I've given today.
00:49:21So we've done a whole bunch of things today, nine things a prompt can't do. We made it much more
00:49:26deterministic and made Impeccable better for that reason.
00:49:33If you'd like to try it out yourself, again, you can clone the repository for this talk. You can clone
00:49:41Impeccable minus talks. But of course, it also is useful to just take a look at the actual skill and
00:49:48see how it's built. The project is completely open source, licensed under Apache 2. You can install
00:49:55Impeccable MPX Impeccable Skills install and check out the source code on GitHub. With that,
00:50:05I'm at the end of it. Thank you.
00:50:12And now I think we have about 10 minutes for any questions that you have.
00:50:18Does anybody have questions? Yes?
00:50:25Oh, sorry. What was that? A link to the repository? Yeah.
00:50:33This is hard to see, but let me put it up here. This is the repository for the talks. Yeah.
00:50:46The question is I mentioned that it breaks prompt caching. The actual sort of trick, the technique to
00:51:11actually get something back from a script within a skill. And the reason is because the result is
00:51:18dynamic. It could be anything. So unless the result is always the same, it's a dynamic shell execution.
00:51:26So it gets inserted into the thread. Now to be fair, the skill will still be cached. So the skill will
00:51:33still be cached. But I guess I'm differentiating between the skill with inline, you know, static content versus
00:51:43the skill with sort of like a dynamic instruction to call out. So this part will not get cached.
00:51:48Yeah, that was that was my main point. Yeah. Yeah.
00:51:52What is the process on how I evaluate and iterate on this skill? So I, the process is pretty involved.
00:52:10Let me see. Let me see. Let me see if I can bring this up on screen.
00:52:23Okay, here we go. So here's a glimpse. Oh, no. Okay. I just shut down the server. That's fine.
00:52:31Okay, I'll just voice over. So yeah, I mentioned I built an evals harness. And so I've created myself a harness
00:52:40that closely recreates the conditions and the tools of every harness that I care about. So for instance,
00:52:50it uses the cloud code SDK.
00:52:55Yes. Sorry, guys, can you can you lower your volume a little bit? Because people are still trying to
00:53:02hear the questions. Thank you. So how do I how do I test this? How do I build it? So this is a
00:53:09combination. So first of all, impeccable has a ton of end to end tests in the repository. That's both
00:53:15LLM driven tests, as well as end to end playwright tests. So that's one. And that's useful for things
00:53:22like testing the live mode scripts, for example. But then beyond that, how do I test that it actually
00:53:28works? Well, I've built an evals harness. That one is not open to us yet. But I built an evals harness
00:53:34that closely replicates every model harness that I care about right now, specifically right now,
00:53:41cloud code codecs in Gemini. And I'm trying to expand it to more. And it also recreates the tools,
00:53:48like for instance, a browser screenshot tools, or something along those lines. And then it also
00:53:55recreates the because some parts of impeccable are interactive. In the initialization of impeccable,
00:54:03oftentimes, the user gets asked, so what would you what would you like your page not to feel like?
00:54:08And so you get these interactive back and forth. And so I've built this LLM that acts as the user
00:54:14against the other LLM. And so it does like an interactive, you know, back and forth turn.
00:54:20So I've built that harness. And then I've built a mixture of expert design judge that runs on top of
00:54:26it. So basically, give it eyes to evaluate each result. And then I can run across 20 different niches,
00:54:35like for instance, like for instance, Italian restaurant. I run across all models that I care about GPT-55,
00:54:41Opus, Sonnet, and do like five to 10 tests for each of those, for each skill release to see, you know,
00:54:50how it changed. I also run against competitors. For instance, I run against the front end design skill
00:54:56to see, does it make a difference? And how does it make it worse or better? And then beyond that,
00:55:04I'm doing ablation testing. That's harder and more expensive, I would say. So I don't recommend it
00:55:10for everyone. But this, the ablation testing, so every, you'll see this in the source card of
00:55:15Impeccable. Every rule has sort of an XML tag that says like, you know, a unique identifier of that
00:55:20particular line. And that will be used by the harness to then do a test where it removes that line,
00:55:30runs the evals against all models, and then adds the line back in, and then uses the detection engine
00:55:38of Impeccable, the deterministic one to see, did it actually change, right? So if there's a line that
00:55:42says, hey, don't, don't do like gray on colorful backgrounds for, for contrast purposes. There's a,
00:55:52there's an ablation test and then a deterministic check or feedback loop that tests against it. So
00:55:58in short, quite involved. But, but I really, it started, you know, vibes based,
00:56:05and now it's really, uh, truly, um, well tested. Yeah. Yeah, go ahead.
00:56:15Sorry?
00:56:18Yes.
00:56:22Yes. Um, I do have, I do have, um, evals for evaluating taste, but I don't think they work particularly
00:56:29well. Um, I just talked to, uh, Ben from Contra about this. Um, I don't think, I mean, I know,
00:56:37I know, um, some of my colleagues might disagree, um, but I don't think taste can be solved at a model
00:56:44level. Um, I actually think it's a, it's a fundamentally human thing, um, because taste is scarce
00:56:50and unique and once everybody uses the same taste, uh, it becomes ubiquitous and then we don't think
00:56:56it's tasteful anymore. So it's, it, I think it's, uh, um, I think it's hot. Um, and I also think the
00:57:03models are particularly bad at evaluating taste. So for example, um, there are certain things that
00:57:09the models can evaluate well, like, hey, is this, is the correct thing in the first viewport, right?
00:57:14So functional stuff that works, but what doesn't work, and here's one example, I've built, again,
00:57:20this mixture of judges and one judge rates whether the first viewport looks great, right? And is
00:57:26effective. Um, and one of the tells is that, uh, Gemini, for example, the more stuff there is in the
00:57:37first viewport, the higher it rates it, right? This is just a general rule. Like if you just cramp the
00:57:43viewport full, it gives it a higher ranking. And so there's an interesting example of like, you know,
00:57:49the models are often maximalists, right? They're like, well, more is more, I guess. Um, and so
00:57:55oftentimes I build judges that actually invert the response of the model, uh, which is really strange,
00:58:01but it works, um, where it sort of judges something very high. I'm like, okay, that's definitely not
00:58:06a good design. Um, so anyway, I don't think it's solved and I don't think it's solvable,
00:58:12but, um, I do have, I would say a tool that gives you the design director eyes that works marginally
00:58:20better than random and that's good enough for me for like a first pass and then I use my own human
00:58:24eyes to evaluate results and annotate them. Any other questions? Yeah, over here.
00:58:31The future for skills. So I would say, hmm, that's a, that's a broad question. Um,
00:58:42Yeah, so I'll first answer for impeccable and for me. So in the, in the case of impeccable,
00:58:57I think we're, we're, we're definitely outgrowing, uh, the skill platform kind of what's possible with
00:59:03skills. I think the, the live mode is a good example of that. The live mode, um, was sort of like a Jurassic Park
00:59:11experiment to see, like, can I do this? And the answer is yes-ish. Um, I think it's, it's,
00:59:18it's working better than I expected, but, um, it still has a lot of problems. I mean,
00:59:23it would be way better to do this in a first party harness integration or like a, like a first party
00:59:27tool. Um, so I think there are limits that I'm hitting, uh, where, um, where skills might not be
00:59:33effective anymore. I think in general, I would say most skills should probably be written by the
00:59:40individual users. I think those that actually go, go through the effort of packaging a skill and
00:59:45sharing it with others need to invest more time than they currently do. So I guess that's my hot take.
00:59:52I think right now I've, I've seen plenty of skills that are distributed that, uh, do not work well
00:59:59in a model that the author didn't use, for example. Right. And so I think, I think we just have to raise
01:00:04the bar of what's acceptable to ship, uh, to, to, to people. I mean, again, this is like the works on my
01:00:10machine thing. Um, I would rather see less skills in the ecosystem that are really battle tested and proven.
01:00:17Um, and I, I hope we're, we're shifting towards that because right now it's sort of like a white west.
01:00:23Yeah, go ahead.
01:00:37There's no common way to test the skills. Yeah. And then that could be an opportunity. That's a good
01:00:41point. Yeah. I guess I could, I do have the tool for that. That's true. Yes. Um, I could do something
01:00:48with it. Yeah. Uh, right now it's purely built for my own purposes, but, uh, but yeah, the same is
01:00:53true for, for instance, like the, the impeccable installer and compiler. Um, I don't think most people
01:00:58know that it exists, uh, that it can compile to every harness and that it has these substitution
01:01:03techniques and stuff like this. Like I could probably release that standalone as well. Yeah, that's a good point.
01:01:09Yeah. Go ahead.
01:01:17MCP having skills on the server. How would that work?
01:01:25Oh, I see. Yeah. Um,
01:01:29to be honest, I haven't tried it out yet. Um, or I haven't really read too much into it. I think MCP in
01:01:35general, um, um, you know, I worry greatly about context pollution and, um, I do that with skills
01:01:42too. And I think, um, um, I'm not using MCP a lot for that reason. Um, because it polluted my context
01:01:50many times. Um, how do skills work in MCP?
01:01:55Yeah. Oh, you can download a script from MCP server. Yeah. Okay.
01:02:07One of the issues that, uh, I think, um, um, I don't know if I was mentioning about it.
01:02:20Yeah.
01:02:20Yeah, yeah, yeah.
01:02:27So what's the recommended way of packaging them and distributing them? Yeah, it's a good,
01:02:30good topic. Um, so of course, like the harnesses and the, and the frontier labs have their own ways.
01:02:37I mean, um, codex has a marketplace that you can use for distribution plugin marketplace.
01:02:43Um, uh, cloud code has a marketplace as well. I think they started with the marketplace technique.
01:02:48Those marketplaces don't work particularly well. I mean, the cloud code one for sure doesn't work
01:02:53particularly well. I know this for a fact because I mean, the optic mechanism often doesn't work.
01:02:59And people are like, well, my skill doesn't update. And oftentimes there's a caching issue. So it,
01:03:05my experience has been hit or miss with the native methods of distributing. And then of course,
01:03:09it's only for that particular provider. That's why projects like, uh, skills.sh exist. But again,
01:03:16the problem with MPX skills right now, it doesn't, it does, it, it doesn't allow for like, you know,
01:03:22more advanced skill use cases, like, you know, compiled for every different harness. I have a pull request,
01:03:28um, uh, in the, in the, in the repository. Um, and I've, uh, I've, uh, I've bugged Andrew a couple times
01:03:33about it, but, um, he, uh, he, uh, he still has to, uh, get it merged or I agree to, agree with me on that,
01:03:40I guess. Um, I think, uh, we're still discussing. Um, but yeah, MPX skills, I think is a great project in general.
01:03:47I think it'd be great if we could sort of like standardize around it. Um, there's also one from
01:03:53Microsoft that's trying to do that. Um, a project from Microsoft, I forgot the name of it. Um, but
01:03:58there's definitely no, no industry standard for distribution yet. Um, yeah, I'm not, I don't love
01:04:04having to maintain my own CLI installer. I would rather not. Uh, it's annoying. Um, but, uh, it does
01:04:11make it so it plays safe with all harnesses installed, the hooks in the right part of the
01:04:15system, et cetera. So it's, um, yeah, yeah. Okay. I think I'm, uh, way out of time, uh,
01:04:24but come up and speak with me if you like. Yeah, I would say I'll end it here, but you have come,
01:04:29come up if you'd like. Um, let me just, uh, thank you.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기