Nvidia's New Tool Just Fixed Agent Skills
AAI LABS
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Right now, AI agent skills are everywhere. Every agent runs them and you trust them without any
00:00:05checks. But here's the scary part. Researchers studied over 30,000 of these skills and more than
00:00:10a quarter of them had a security vulnerability. So NVIDIA built a tool called Skill Spectre that
00:00:15scans any skill before you install it and tells you exactly how dangerous it is. But here's where
00:00:20it gets interesting. One type of attack can slip right past it and the setting that actually
00:00:24catches it is off by default, so most people never even know it's there. Turning that on normally
00:00:29costs money, but we found a way around it. And by the end, we didn't just scan skills. We built a
00:00:34whole workflow that changes how you find and install them for good. Now before we get into the full
00:00:39workflow, let's give you a quick tour of the tool and what you need to use it. So these are the install
00:00:44commands in the GitHub repo. You can just copy them and hand them to Claude Code and it'll basically
00:00:49install and set up the whole thing for you. Claude Code's gonna install all the dependencies you can
00:00:54see right here. And once all that's done, you can start using Skill Spectre. Inside the GitHub repo,
00:00:59there's this test folder and inside that they've got some dangerous skills you can actually run it on to
00:01:04confirm the tool works. So we ran it on these skills and with every one of them, it tells you not to
00:01:09install. The higher the score, the more dangerous the skill. And with each test, it doesn't just give
00:01:14you a number. It shows you the exact line number, the exact location and the file name where the conflict
00:01:19is, which is basically what pushed the score up. Now this isn't the only way to use the tool, it's got
00:01:24another mode. But before you get why we even need that second mode, you need to know two things: how a skill
00:01:30even attacks you and how this tool actually catches that attack. Now there are 14 categories,
00:01:34but to keep it simple, we've grouped them into six similar ones. So the first way a skill can attack
00:01:39you is with hidden instructions. See, a skill is just a text file full of instructions and your agent reads
00:01:45the whole thing and treats it as orders. The problem is, a bad skill can hide extra instructions in there that
00:01:50you'll never see, but the agent does. They tuck them inside comments, or they use invisible characters,
00:01:55or they scramble the text into a code that looks like nonsense to you, but the AI reads it just fine.
00:02:01So the scanner is built specifically to hunt these hidden instructions down and find them. The second
00:02:06way is impersonation. So your agent has tools it trusts and reaches for by name. Say there's one just
00:02:12called "read" that reads a file for it. So a malicious skill gives its own tool that exact same name,
00:02:17and your agent grabs the bad one thinking it's the safe one it already knows. And the way they pull
00:02:22it off is sneaky. They swap one letter for a lookalike from another alphabet. So they name it "read",
00:02:27but the "A" is actually a Russian letter that looks identical to ours. To you and to your agent at a
00:02:33glance, it's the same word, but underneath it's a completely different tool. And the scanner catches
00:02:38this by checking the real identity of every single character, so it spots that one fake letter and
00:02:43flags it. The third way is when the skill just lies about what it does. The description says one thing,
00:02:48the code does another. So it calls itself a simple formatter and then quietly reaches out to the
00:02:53internet in the background. Or it says it only needs permission to read your files, but the code is
00:02:58actually writing files and running commands too. And this one's way harder to catch. This is where that
00:03:03second mode comes in, but we'll get to that later. The fourth way is the skill steals your credentials.
00:03:08This could be your API keys, your passwords. So a skill goes through all the keys saved on your
00:03:13machine, scoops them up, and sends them off to some server. The fifth way is the skill just runs
00:03:18straight up malware. This includes things like a reverse shell, which basically hands a stranger
00:03:23remote control of your whole computer. And because this kind of malware has known fingerprints,
00:03:28the scanner just matches the code against a big library of those fingerprints. And the sixth way is
00:03:32poison dependencies. So a skill will often use a CLI tool, basically a small outside program it runs in
00:03:39the terminal to handle part of its job. And a bad skill grabs a piece that's actually malicious.
00:03:44Maybe it's a fake package with a name that's one typo off a real popular one. So you pull the wrong
00:03:49one and it runs malware like the last type. So the scanner checks every package the skill pulls in
00:03:54against a live database of known bad ones. And it flags the fake names and those download and run
00:03:59commands to keep your system safe. So in that first mode, it's just matching patterns without any context,
00:04:05which means it ends up flagging stuff that's completely fine. And those are what we call false
00:04:09positives. So that's where the second mode comes in the AI scan and turning it on is simple. You just
00:04:14drop this no LLM flag and it does the second scan here. But if you look inside the code, you'll find out
00:04:20that to run an AI check on a skill, you need to plug in an open AI key. So to get around that cost,
00:04:26we just use Claude Code itself to run that AI check. Now the main agent in Claude Code doesn't actually
00:04:32do it itself. We use Claude's headless mode, which is basically Claude Code running in the background
00:04:38with no chat window, just executing commands on its own. And we're sure most of you know it isn't free,
00:04:43but you do get monthly credits for it with your Anthropic plans. And you can just ask Claude Code to
00:04:48make the change we just talked about and it'll do it for you. Of course you might hit a bug or two,
00:04:52but it's just a single line prompt Claude can set up for you. And if you're enjoying the video so far,
00:04:57subscribe to the channel and hit the hype button. This small gesture of support goes a long way for us.
00:05:03So they've also got dangerous skills in their test folder that actually need the AI check. When you
00:05:07run the no LLM check on one of them, the score comes out as zero, which means it's perfectly safe.
00:05:12But the second you run it with the AI check, the score jumps to 100, it tells you not to install,
00:05:17and it lays out exactly why. But what if instead of just detecting the problems in a skill,
00:05:22the scanner also helped you fix them. So that's exactly why we turned the scanner into a skill. And
00:05:27you might be wondering why is it called Discover Skills? Well, because we didn't just make one
00:05:31separate skill. We made a whole process that helps us discover more skills and make sure they're safe
00:05:36before we install them. So we've been using skills.sh to find new skills for a while now. It's basically a
00:05:42git repo built specifically for skills. So one big shared library you can pull from. And we think they
00:05:47recently shipped a CLI update. So now Claude can just run search queries straight through the command
00:05:53line and pull the best skills it needs before installing anything. And we wanted our scanner
00:05:57running on top of that. So in here, we've got scan.sh, which is the script that actually runs
00:06:02skill specter. Since skill specter is a CLI tool, it has to be run as a command. So we made a whole
00:06:08script and we baked the Claude headless mode fix right into it. So by default, it runs the normal
00:06:13check, but if you want, it'll run the AI check too. And if you open up skill.md, you can see the basic
00:06:19steps laid out. It identifies the target, then scans it, then it shows you the findings. And once it knows
00:06:24what the problems are, it goes ahead and fixes them, then runs the whole loop again after to make
00:06:28sure everything's clean. So for example, this folder we're showing you right now is our AI labs design
00:06:34folder. It's basically our whole design process compressed into one folder with a bunch of skills
00:06:39inside. We've got a whole video on this. And on top of that, the whole system's available in AI labs
00:06:44pro, which is our community. So if you want to support the channel and grab this whole design system,
00:06:49go check it out. And this discovery skill is going to be uploaded in there too. The link's going to be
00:06:54in the description, but we're building on top of this here. So we're adding a new make design.md skill,
00:06:59which lays out the fastest way to pull design tokens out of an app you've already built, basically the
00:07:04colors, fonts, and spacing rules, and merge them into a design.md file. So here we wanted to create
00:07:10the design.md file. So we told it that we wanted to improve it and that it should go search for other
00:07:15tools out there. So it used skills.sh, then we loaded the discovery skill and that pulled back a
00:07:21handful of skills. These are the skills it brought back and the first two looked interesting. So we wanted
00:07:26to dig in. We asked it to install and test both of them. And just like the discover skills workflow
00:07:31says, it won't install any skill without scanning it first. So it installed them and read through them
00:07:36and told us straight up that neither one was going to help with the make design.md skill. But from a
00:07:41safety point of view, the first one got a score of 10, which meant it was safe, and the second got a
00:07:46100, which meant don't install it. So we told it to run the AI check on that second skill. It ran it again
00:07:52through Claude's headless mode and this time the score came back as zero. This means that the skill
00:07:56was safe to use. And that's the whole point of this system. You're not just grabbing skills blindly off
00:08:01the internet. You have a whole process that you can kick off just by using a skill. Now let's have a
00:08:06word from our sponsor. Nimblist. If you use Claude code or codex, you know the problem. You've got multiple
00:08:12sessions running, files changing everywhere, and you're constantly switching between terminal, browser,
00:08:17and editor just to keep track of what your agents are doing. Nimblist is an open source visual workspace
00:08:23that puts everything in one place. I had three agents working on different parts of a project at
00:08:28the same time and instead of jumping across windows, I could see all of them on a Kanban board, jump into
00:08:33any session, review code changes as red and green diffs, and approve or reject them individually. I was
00:08:38editing markdown docs, UI mockups, and architecture diagrams visually right alongside my agent. When I was
00:08:45done, I didn't have to clean up commits manually because it generated git commit messages automatically
00:08:50based on what changed. Tasks stayed connected to the actual sessions and there's even a mobile app to
00:08:56continue the session while you're away from your desk. Nimblist is completely free and open source
00:09:00and you can check it out by using the link in the pinned comment. That brings us to the end of this
00:09:05video. If you'd like to support the channel and help us keep making videos like this, you can do so by
00:09:10using the super thanks button below. As always, thank you for watching and I'll see you in the next one.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video