We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog
AAI Engineer
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Hi everyone, how are we feeling? We're in the home stretch. My name is Sarah and I'm a context
00:00:23engineer at PostHog and I get the delight of working on our beloved wizard every single
00:00:30day. So, what's the wizard? The wizard sets up PostHog for you. It's an agentic CLI tool
00:00:39that reads your code base, it installs the right SDK for your project, it instruments
00:00:44your events, and it sets up dashboards for you. It takes what used to take about an hour
00:00:52or two of setup and it runs that in about five to six minutes and it's free inference
00:00:57on us so that you have a great time onboarding to PostHog. It sounds kind of sick. People
00:01:03love it. But a few months ago we dared to dream what if this became the recommended or default
00:01:10way to install PostHog on your project. And my security alarm bells started going off.
00:01:17I started questioning how secure is this thing because it sounds kind of malware shaped.
00:01:25And in that questioning I learned a lot. So today is all about the lessons I learned, the
00:01:31stuff that kept me up at night while I was building this thing, and the thing that I ended up building
00:01:37because of it. So before I dive into all of the boring security stuff, aka your 2:00 PM cat nap, I want to show you the wizard actually running.
00:01:49If you look up on the screen, it is running for you on a loop. This is the same exact experience that anyone who runs
00:01:56NPX at PostHog Wizard gets on their terminal. Like I said, it is an agent. It figures out what SDK is right for your project. It installs it for you, instruments your events, builds dashboards.
00:02:10I like to call it a little mini implementation engineer in your terminal. And sometimes I show people this and they ask me why an agent? Why don't you give users a good prompt? Why don't you give them a skill that they can invoke in their own tool?
00:02:23And while we do provide those things, the answer is because this developer experience and the capability of the wizard is the whole point. It's the whole product.
00:02:34Because we build a CLI tool that can fully take part in an agent loop and experiencing that for the first time is really powerful.
00:02:43But you can't ship something like the wizard without shipping the stuff that makes the wizard kind of suspect.
00:02:50So let's take it apart. Let's look at the anatomy of the wizard because usually threat models fall right out of the anatomy of the agent.
00:03:01So the wizard is a similar shape to what I'm sure a lot of you are building if you're building agents.
00:03:07It's got models that we've picked for specific tasks. It's got prompts that steer it and it's got a set of tools that we've handed it to get the job done.
00:03:17But it also has some pieces that are really specific to us. It has a context engine fully built in house by my team.
00:03:25It's what allows the agent to do such a good job and give us similar results on every run.
00:03:30I like to call it the wizard's brain. Sometimes we call it markdown in a trench coat.
00:03:35But it's our in-house context engine. There's also a terminal UI that we built ourselves using ink.
00:03:43And now there's a security scanner called the warlock which is what I built when I started snooping around and uncovering the horrors of shipping an agent to production.
00:03:55So if you take the anatomy of any agent that can run commands, it's basically what I like to call the malware starter pack.
00:04:03Because it's almost exactly what you would hand a piece of malware if you were feeling generous or chaotic evil.
00:04:11Luckily, this is the worst case scenario or the nightmare fuel and it's not a confession for me, it's a warning for all of you.
00:04:19Because if you want to ship an agent with hands, an agent that can run commands, you need to make sure that you do not build this.
00:04:28So the V0 of the wizard was born because Josh Snyder, if you know him on our growth team, was watching cursor hallucinate post hog setups in quite possibly the worst ways.
00:04:41And he thought, what if we built an agent that could do a better job?
00:04:46So my team started building on top of it as we validated that it did a much better job than cursor hallucinating.
00:04:52And we thought, what if it could onboard anyone to post hog?
00:04:58It doesn't matter what their framework is, what their stack is, instrument all their events without them having to touch a thing.
00:05:04And then we dared to dream, what if it was the default way to install post hog?
00:05:09We were dreaming of thousands of developers running this a week and yesterday we just hit 8,000 people running this a week.
00:05:16So our dream came true.
00:05:18But we, back in those days when we were dreaming, we had to take our security posture under a microscope and look at what was going on.
00:05:26So I took the ownership of that and I sat down and evaluated where we stood.
00:05:31And early on, I'm talking like a year to nine months ago, we had what I call layer zero because it quite literally is not security.
00:05:41It is just prompts that suggest what the agent should do and steer it and prompts are not security.
00:05:48So I was concerned there.
00:05:51Layer one, it was an allow list.
00:05:53And when I started digging into this allow list, I started to feel a little bit better because it was pretty tightly bounded.
00:05:59But I still had a lot of concerns.
00:06:01And I started panicking because of that context engine that I told you about.
00:06:05We are feeding a lot of context into the agent at runtime.
00:06:09So I built this really hacky regex scanner to look for threat-shaped things going into the wizard and threat-shaped things coming out of the wizard.
00:06:20And I will admit that it was extremely hacky.
00:06:23But I'm telling all of you this very candidly because we are all building things that feel extremely experimental and we are all building things super fast.
00:06:33And I know not all of us have security in our wheelhouse.
00:06:38And some of us are just learning it on the fly like I was.
00:06:43But it's something we need to be thinking about when we are building things that have this shape.
00:06:50So that was our security posture.
00:06:53But I asked the question, are we cooked?
00:06:56Good news, we were less cooked than I thought.
00:06:59Because when I mentioned earlier that allow list, it was pretty tightly bound.
00:07:03We had bash as deny by default.
00:07:06It could only install trusted packages that were vetted by us.
00:07:09It could build.
00:07:10It could type check.
00:07:11It could lint.
00:07:12And pretty much nothing else.
00:07:13It couldn't run random shell commands.
00:07:16And it didn't have access to environment variables.
00:07:20The agent couldn't read your .env file because we blocked it outright and we were rooting secrets through a vault.
00:07:28So I took a breath of relief and realized we were in a better place than I thought.
00:07:34But I wanted to know where the cracks were because with security there's always cracks.
00:07:39So I did the thing that we should all be doing.
00:07:41I tapped our security team and I said, hey, can you audit this thing for me and find those cracks for me?
00:07:48And they found some things.
00:07:51They found some gaps.
00:07:52And the interesting part wasn't the specific gaps or bugs they found themselves, but it was the shape of them.
00:07:58Because almost none of them were obviously evil.
00:08:01They were all two very innocent, well-intentioned things that were shaking hands and opening a hole.
00:08:07So the lesson I learned was that attacks compose, code review doesn't, because us developers all look at diffs one at a time.
00:08:20But attackers look at the whole system and they look for those two things that shake hands and open a door.
00:08:25But there was one more thing that was keeping me up at night.
00:08:29And going back to that context engine, I realized the scariest part of the agent we had built wasn't really a command in our case.
00:08:37It was the helpful-looking stuff that we were feeding its brain.
00:08:43Oh, I think I went the wrong way.
00:08:46Yes.
00:08:47The context mill.
00:08:48So this is our context engine, aka the wizard's brain.
00:08:51And it's how the wizard knows anything at all and why the wizard actually does a good job.
00:08:56It pulls from our docs.
00:08:58It has handwritten prompts that are got-chus and lessons that we learned along the way.
00:09:02And real working end-to-end example apps that help the agent pattern match so that it can install PostHog in a really great way for you.
00:09:11It packages all of that into skill bundles that get shipped to the wizard over our MCP server and loaded straight into the agent's context at runtime.
00:09:22So sit with that for a second.
00:09:24It's a machine whose whole job is to take content and inject it into an agent that can run commands.
00:09:31Now, if you were an attacker, you might say, "Well, what if I just poison the content?"
00:09:36Not the user's code base, not the agent itself, but the actual content.
00:09:41Say someone opens a pull request on one of our open source repos, because at PostHog we build everything in the open,
00:09:48and they inject something in a markdown file or a seemingly harmless code comment,
00:09:55and we have some sort of LLM-powered code review going through that, and it says, "Looks good to me," and ignores it.
00:10:04We may have just shipped a prompt injection payload signed by us into an agent that is running on thousands of developers' machines in a sandbox, but still.
00:10:14So that was the threat that reshaped how I think about security and the wizard, because the dangerous input for us really could come from our own supply chain.
00:10:25So what I ended up doing is I started scanning content at both ends of this pipe.
00:10:30One, when a skill gets built and released, and again, when the wizard actually uses it.
00:10:36My methodology is catch it at the source, assume the source failed, and catch it again at the point of use.
00:10:43So now I get to introduce the Warlock to you.
00:10:48Building the Warlock was not necessarily damage control.
00:10:52Like I said, we had defense in other ways.
00:10:55But I built the Warlock because I didn't like telling people, "Well, this thing is, like, pretty locked down.
00:11:01That doesn't scale.
00:11:02That's not something you want to ship to production.
00:11:04That's not something that you want thousands of developers running every single day."
00:11:08Because when you ship something to that scale, you have way more surface, way more users, way more content flowing in as you expand the capability of the wizard.
00:11:19And we're probably fine.
00:11:21It just stops being good enough.
00:11:23So I pulled that hacky little regex scanner that I threw in there, pulled it out of the wizard, and I made a standalone thing.
00:11:30I called it the Warlock because everything wizard shape needs a bodyguard.
00:11:35And it does exactly one job.
00:11:38You hand it a string.
00:11:40It hands you back a list of findings.
00:11:42Each of those findings has a category, a severity, and a recommended action, and then it stops.
00:11:48I want you to focus on recommended here.
00:11:51Because the Warlock detects it does not act.
00:11:54It will tell you, "Hey, this looks like exfiltration.
00:11:57It's critical.
00:11:58I would block it."
00:11:59But what you actually do with that finding is completely up to you.
00:12:04Because detecting a problem is one job, and deciding what to do about that problem is a totally different job.
00:12:10And the only thing that keeps all of this understandable is keeping those two things separate.
00:12:15So underneath the hood of the Warlock, instead of my hand-rolled regexes, the rules run on Yara, which is the pattern that engine malware researchers have been using for, like, 15-plus years.
00:12:27It's fully deterministic.
00:12:28It's the same input, same output, every single time.
00:12:31It's boring on purpose.
00:12:33And in security, boring is a feature.
00:12:39So what does the Warlock actually catch in the wild today?
00:12:42A bunch of different stuff, but two of these are an absolute, like, nuisance to my soul.
00:12:48The first thing is actually not a rule-shaped thing.
00:12:52It was something that the Warlock flagged that was actually a sub-agent behavior that exposed a vulnerability to us based off of what sub-agents were doing.
00:13:03So basically, we were spinning up agents to do large tasks.
00:13:07They were spawning sub-agents.
00:13:08And those sub-agents were trying to get around the guardrails that we had implemented in the wizard, and they were trying to invent secrets.
00:13:16They were trying to pull secrets from quite literally anywhere in the code base.
00:13:20And we shut it down.
00:13:21We said, "No more sub-agents."
00:13:23And because of the Warlock, we caught that.
00:13:26And I'll empathize with the robot.
00:13:28The robot had a task to do, and it was trying to optimize and please us.
00:13:32But we can't have that.
00:13:35And something else at PostHog that really matters to us is PII.
00:13:39Agents genuinely do not care about exposing data unless you make explicit rules.
00:13:46Left alone, we watched it dump emails, phone numbers straight into events and to an agent that looks like a totally normal thing to capture.
00:13:57And luckily for prompt injection specifically, I'm going to knock on wood here, we have basically never caught an actual malicious prompt injection in the wild.
00:14:08But we do catch a ton of false positives, things like our demo login screens, copy on our example apps, things on our docs.
00:14:17And it's actually made me rethink how I build applications and how I write docs because I don't want to ship anything that looks threat-shaped.
00:14:27But the false positives are honestly the perfect setup for the messiest, most interesting part of this whole thing.
00:14:36So this is the part that I wrestled with.
00:14:39I spent this whole talk preaching deterministic to all of you, and then I went and I added an LLM layer to help sort my false positives and silence some of the noise.
00:14:50And I call it triage.
00:14:51When I was building this triage layer, I had to make a choice.
00:14:56Should the layer be a bouncer or should the layer be an advisor?
00:15:01And the easiest choice probably could have been make the LLM the bouncer.
00:15:06Show it the command, ask it is this an attack, block, allow, and just do whatever it says.
00:15:13And while that's tempting because it seems easier, I can't bet my security model on a coin flip because my model's having a bad day or something happened and it's acting different today than it did yesterday.
00:15:26So instead of the bouncer, I crafted the model to be the advisor.
00:15:32And this was the clean line that I found in a line that I'm still exploring, but I want to leave all of you with.
00:15:38First, for us, detection and enforcement stay deterministic and mechanical.
00:15:43If a rule matches, the gate locks, the session ends, and there is no model anywhere on that path.
00:15:49The block happens before we even ask the LLM's opinion.
00:15:54The LLM only gets to weigh in afterwards if we have not blocked something.
00:15:58It's designed to remove noise.
00:16:00It is not designed to let things through.
00:16:03And if it fails closed, so if the model is having a bad day, all wizard runs are killed, sorry, but we're just protecting you.
00:16:14Enforcement is the part that you bet the house on, so it has to be deterministic.
00:16:20But judgment is the part that adds nuance, so that's really the only place that you can put anything probabilistic in there.
00:16:27So how do we ship real rules for agents?
00:16:34This is the anatomy of one of our warlock rules, and every warlock rule has four parts.
00:16:41Part one is the metadata.
00:16:43It's plain English description, severity, category, action, direction.
00:16:50Part one, is this flowing into the agent?
00:16:52Is this something the agent is writing?
00:16:57Then we have the strings, so these are the actual patterns that you're looking for.
00:17:03And part three is the condition.
00:17:05So this is where the rule is actually allowed to fire.
00:17:10I will walk through this example for you, and we can pretend like we're writing it in our head.
00:17:15Prompt injection being like the classic ignore all previous instructions.
00:17:20Your first instinct here is probably to block the word ignore, but agents read code all day,
00:17:27and ignore can show up in code comments or examples all the time.
00:17:31So you don't want to match the verb alone.
00:17:33You match the verb plus an instruction flavored noun.
00:17:38In the condition, you say fire if any of those patterns hit.
00:17:42And in the metadata, you determine is this critical, what the category is, what the action is,
00:17:50in this case block, and the direction, in this case being input flowing into the agent.
00:17:56But to write good rules that reduce noise, you have to ship tests with them.
00:18:02So you have to write tests that say these are patterns that match.
00:18:06These are ones that should not.
00:18:08And that negative test is the first line of defense against false positives.
00:18:13But you also want to make sure, when you're deciding the severity of that rule,
00:18:19that you track real-world impact, not how scary it looks.
00:18:24RM-RF is scary, but it's also how we all delete node modules like 40 times a day.
00:18:31You decide the real-world impact for the agent that you're building,
00:18:37because a security tool that crashes every time it tries to clean a build folder
00:18:42is a tool that gets turned off and one that catches absolutely nothing.
00:18:47So I'm proud to say this is our security posture now.
00:18:51I can finally come up here and say we have true defense and depth.
00:18:56All my learnings have assembled into this.
00:19:00It's still layered, but every layer is doing a job that it's good at now.
00:19:04We still have prompts, but we only use them for steering.
00:19:07Everything runs in a sandbox.
00:19:09We deny by default.
00:19:11We have a vault so secrets never hit the model.
00:19:14We have the warlock to scan content coming in
00:19:17and to scan output being written by the agent.
00:19:20We also have triage to reduce the noise,
00:19:23and we have telemetry embedded in the entire process
00:19:26so that we see everything.
00:19:28None of these layers stands on its own.
00:19:31Not a single thing here is going to save you,
00:19:33but it's just boring, honest layers,
00:19:36each of them doing one job that it's good at.
00:19:40So if you're building an agent with hands,
00:19:45this is the whole talk in three lines.
00:19:47One, if it isn't enforced deterministically,
00:19:50it is not enforced.
00:19:52Prompts are not security rules.
00:19:54Don't act like they are.
00:19:57Two, the dangerous input isn't just what your user types.
00:20:02It isn't just the commands that you allow it to run.
00:20:04It's everything flowing into the model,
00:20:06including the content that you write yourself.
00:20:09So scan your own supply chain at the source
00:20:12and when the agent invokes it.
00:20:15Three, attacks compose, code review doesn't.
00:20:18Most of our gaps during our audit were two innocent things,
00:20:22shaking hands and opening a door.
00:20:25The wizard, the warlock, and the context mill are all open source,
00:20:29so come find me downstairs.
00:20:32I'm in the expo hall at our booth
00:20:34and I'll show you around, show you what we built,
00:20:37and I want to hear how you guys are securing your agents.
00:20:41Thank you.
00:20:59Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video