Your company brain will leak secrets: how we stopped it for big banks — Tanmai Gopal, PromptQL
AAI Engineer
Computing/SoftwareManagement
Transcript
00:00:00All right, everybody can see.
00:00:15Hey, everybody.
00:00:17Thank you for being here.
00:00:18I'm going to talk about the fact that if you go ahead
00:00:23and build a company brain, it will likely
00:00:25leak company secrets, which is kind of the big fear
00:00:28that we have about building a company brain anyway,
00:00:32which is this case of intern joins the company
00:00:34and then suddenly gets comp details and everybody
00:00:36kind of situation.
00:00:37You want to guard against that.
00:00:40This has kind of been, I guess, pretty much the biggest thing
00:00:43that's been holding us back from just deploying OpenClaw
00:00:47and Hermes all over the place.
00:00:49It's also kind of the reason--
00:00:50it's like this big opportunity that ClaudeTag
00:00:52had with its recent launch a few days ago where
00:00:54it was going to be the company brain,
00:00:56but then everybody's like, well, it's
00:00:57not-- it doesn't look like it's going to be the company
00:01:00brain, right?
00:01:01So I'm going to talk about kind of what makes it challenging.
00:01:03So before we get into that, let's kind of understand and dissect
00:01:07this company brain business a little bit, right?
00:01:09I'm Tanmay.
00:01:10I'm the CEO co-founder of PromptQL.
00:01:13You can check PromptQL out later, but our background as a team
00:01:18building this is we come from the Hustle Graph--
00:01:21we're creators of the Hustle GraphQL engine.
00:01:23It's a very popular open source project in the GraphQL space
00:01:26where we solve a lot of data access problems.
00:01:28We deployed everywhere from like Apple to Meta to JPMorgan,
00:01:32et cetera.
00:01:33And that kind of gave us a lot of this grounding for--
00:01:39and kind of like a love-hate relationship with data
00:01:43and data security.
00:01:44So I'm going to show you stuff that we've been working on over
00:01:51the last year and kind of what we've learned from that so that
00:01:53you can kind of take that and exercise that and try it out
00:01:56for yourself.
00:01:58And of course, at the end of the talk, happy to kind of exchange
00:02:01notes and see what works or what might not work for you.
00:02:06Over the last year, we've only partnered with a small set of
00:02:08people who've exhibited some kind of spike on scale.
00:02:12It's about 15 to 20 folks so far.
00:02:15And now we're just starting to open it up to other people.
00:02:17But over that course of time, we've kind of looked at three
00:02:20different types of set of people who are very different needs.
00:02:22You have AI-native companies that are willing to just do
00:02:25whatever as long as it works.
00:02:27You have kind of tech-forward companies, folks like Instacart,
00:02:30who like best-of-breed technology.
00:02:33So they'll move fast.
00:02:34They'll be tolerable to breaking things.
00:02:36But it just needs to be really, really good, right?
00:02:38And then you have Fortune 100 banks who have like a fuck-me
00:02:41level of security.
00:02:43Thank God that they do, because they're my bank.
00:02:45I definitely don't want Vibe-coded AI agents running
00:02:48inside a bank, because that's where my money is.
00:02:51So they have a lot of security rules.
00:02:54Thank you so much.
00:02:55But we're deployed in places like those as well,
00:02:59with kind of the beginnings or like the frontal lobe
00:03:01of a company brand, right?
00:03:02So we can talk about those kind of learnings.
00:03:05Our own personal usage of kind of building out our company
00:03:10brain, it kind of is about 5,000 pages.
00:03:13So we model it as a wiki.
00:03:14You can model it however you want.
00:03:16You can model it as a set of markdown files on GitHub.
00:03:17You can put it into a, remember, graph frag?
00:03:21You can model it in knowledge graphs.
00:03:23You can do whatever you want.
00:03:25So you can place it wherever you want.
00:03:27But like it's about 5,000 interconnected pages for us.
00:03:32Question for you folks.
00:03:33So suppose you had a company brain that was working.
00:03:36It was working well.
00:03:37It was all set up, right?
00:03:40There would be a kind of daily number of updates
00:03:41that would happen to this company brain, right?
00:03:44Because it was learning stuff from everybody in the company,
00:03:47right?
00:03:47From finance, to HR, to your engineers, to everybody.
00:03:52So if you were to plot the daily number of updates happening
00:03:55to the company brain, what would it sort of look like?
00:04:00Would it sort of look like a roughly downwards trend?
00:04:04Like all of these are like random graphs.
00:04:05But would it sort of like start and then go down?
00:04:09Would it kind of be steady, going up and down as updates spike?
00:04:12Or would it kind of steadily increase upwards?
00:04:16So to kind of think about, like, what would the commit history
00:04:18to your shared skills repo look like, right?
00:04:21How many updates are happening to a healthy company brain,
00:04:26right, every single day?
00:04:27What does that trend look like?
00:04:29Anybody for option A?
00:04:31Anybody think it's option A?
00:04:33OK, cool.
00:04:34Option B?
00:04:37Option C?
00:04:39That's nice.
00:04:41And so when I kind of plotted our thing, right,
00:04:44to see what a healthy company brain looks like,
00:04:47if you look at number one, it's basically saying,
00:04:50we had a lot of enthusiasm.
00:04:53We built the company brain on day one, day two.
00:04:56We gave somebody the task and said,
00:04:57build all the shared skills repo,
00:04:59scrape all the slack, scrape all the emails, build it,
00:05:02and we'll all use it.
00:05:03And then nobody cares, right?
00:05:05Or you have a system which is auto learning.
00:05:07Maybe you have a Hermes that's deployed internally.
00:05:09So I'm looking at that where it's kind of steadily adding more
00:05:11and more comments.
00:05:12It kind of goes up and down depending on who has enthusiasm,
00:05:14right?
00:05:15And then when I plotted our history over just the last two
00:05:19months-- and this is a little bit outdated now--
00:05:22this is what we got.
00:05:24And I was kind of shocked.
00:05:26I was like, why is it continuously increasing?
00:05:30Like, it's a gentle curve, right?
00:05:32But why is it gently just going up?
00:05:34Why is the number of updates per day increasing?
00:05:38And that was fascinating for me to see.
00:05:39Because what I realized was that if you have a system that starts
00:05:42to work, what happens is people start to teach it a lot more.
00:05:46It's like saying, if I taught you the skill for querying data, then
00:05:50tomorrow I'm going to teach you the skill for interpreting that data.
00:05:53And then day after tomorrow, I'm going
00:05:55to teach you the skill of how to take an action based on that.
00:05:57And then after that, I'm going to figure out
00:05:59how to do A/B testing based on-- so people
00:06:00like you'll continuously add more.
00:06:03But because everything is an agent where no amount of learning
00:06:05is perfect, everything has its own kind of steady rate as well,
00:06:08right?
00:06:08So the rates kind of--
00:06:10even your steady rates kind of keep adding up.
00:06:12And that's what I started to notice in our thing as well.
00:06:15This is early, so who knows if it'll kind of peter out eventually.
00:06:18Maybe it'll start to look more like option B.
00:06:20But a healthy brain, of course, the overall size keeps increasing.
00:06:24But even your daily updates per day kind of keep increasing as well.
00:06:28So that's a sign of a good brain that you built, right?
00:06:32A healthy brain that you built for your company.
00:06:35Awesome.
00:06:36So use cases for company brain is how we start to analyze how
00:06:39we build a system that won't leak secrets, right?
00:06:42So two use cases.
00:06:44The first use case is there is a company brain.
00:06:48I want to use it in my AI, agent, whatever, to get work done, right?
00:06:55I'll show you an example of that, right?
00:06:56It's like I got an email with a security questionnaire
00:06:59I need to answer from a customer.
00:07:00And I talk to my AI, and I'm like, look up the company brain
00:07:03and help me answer the security questionnaire, right?
00:07:05So that's a totally valid use case of a company brain.
00:07:08Second, very useful use case, right?
00:07:10Because it's other people's knowledge that is coming to me.
00:07:14Second use case of a company brain is kind of similar to what cloud tag is,
00:07:18is this idea of multiplayer.
00:07:21And if you've been putting agents inside Slack in places where multiple people
00:07:25can interact with it, it's kind of been using it as a shared AI, right?
00:07:29It's like get stuff done.
00:07:32And an example of that could be collaborative incident management, right?
00:07:36So for example, you want to say, like, hey, I want to fetch logs.
00:07:40I want to investigate the-- there's an incident.
00:07:42Go fetch some logs, investigate the code base, raise the PR, deploy to staging,
00:07:46deploy to prod, set up an alert, right?
00:07:48You want, like, multiple people are doing things with the company brain.
00:07:51So those are kind of two use cases of the company brain.
00:07:53One is kind of this, like, shared collaborative knowledge use case,
00:07:56and one is, like, shared AI use case itself, right?
00:08:00Both of those have a huge security kind of problem, right?
00:08:06So to start to secure it, let's kind of define that a little bit more strongly, right?
00:08:15So what exactly is a company brain?
00:08:18And this is my definition of it, right?
00:08:20It's shared context that you'd put in a markdown,
00:08:23that you'd put in a set of markdown files, right?
00:08:26And it's access control rules for the different data and tools
00:08:29that you want to access as given to a coding agent.
00:08:34So that's what I'm calling it for--
00:08:38because I'm speaking so I can define whatever I want.
00:08:41That's my definition.
00:08:42So I'm not saying this is knowledge that is pulled into an LLM
00:08:46that will do tool calls, right?
00:08:48It is not an AI that is doing general purpose stuff.
00:08:51It is an AI that is a coding agent that is solving whatever problem you throw at it, right?
00:08:56Kind of similar to the-- a little bit of the previous talk that you folks might have heard,
00:09:00which is this idea of, like, can we just use a coding agent to solve general problems?
00:09:04It's that, right?
00:09:05In the most trivial case, if you say, hey, write me a tweet, you're writing a small script that's making an AI call to write a small tweet, right?
00:09:13You probably don't need to do that.
00:09:14The AI itself can just, like, return the tweet back to you.
00:09:17But, like, essentially, Claude code being used for everything.
00:09:20Claude co-work is the same architecture.
00:09:22The Codex app is the same architecture, which is this realization that you can use coding agents to solve general purpose problems.
00:09:28So we're building the brain for that.
00:09:30We're not building gigantic knowledge graph, knowledge base for the company and then trying to secure it.
00:09:34That, anyway, doesn't-- hasn't worked-- won't work.
00:09:39So in terms of how we want to approach designing the company brain, right, should we build a company brain?
00:09:50So if you're an enterprise and you're paid to twiddle your thumbs, then you like this idea of building a company brain.
00:09:57Because you're like, yes, let me take on a two-year project and I will build the company brain for J.P. Morgan.
00:10:02That's not going to happen.
00:10:03You can't build a company brain for an organization that's, like, 100 years old, right?
00:10:07You can barely build it for your own family, right, which might just be months or years old, right?
00:10:13So the idea and the way that we want to build a company brain is we want each person who does a little bit of the work in the company to own and build their part of the company brain, right?
00:10:23That's the way we should build it.
00:10:25So that's kind of constraint number two that I'm putting.
00:10:27One was the definition of the company brain, and second is how the approach that we want to take for how a company brain is built.
00:10:33I like kind of this way of phrasing it, which is that we're going to grow a company brain, we're not going to build one, right?
00:10:41We're going to, like, let it come together.
00:10:43Let it come together.
00:10:44The system needs to come together, otherwise it's not possible to build.
00:10:47All right.
00:10:48Broadly, we want to let each person self-serve their bit of the company brain, and so these are kind of, like, the various steps that you want to follow.
00:10:57I'll come back to this in more detail if you have time, but let's start with a particular use case, right?
00:11:03So in this particular use case, what I have is this kind of situation where this is kind of the tangible example I want to take for you folks.
00:11:12I got an email, just a security questionnaire example, right?
00:11:15Hey, I got an email from Dave at Stitch Fix, and that is a bunch of questions I want to answer, right?
00:11:20So it pulls up my email, it says the email has a screenshot of their security onboarding, and then it starts to kind of answer those questions, right?
00:11:30I have no idea how it knew.
00:11:32I was kind of very surprised to see that it answered all of the questions on, like, hey, this is our trust center, this is how our security stuff looks, they have a gateway, right?
00:11:40It does something.
00:11:41All of this is kind of coming from the company brain, right, which is the answer to that, and then kind of go ahead and I'm like, hey, just go ahead and send this.
00:11:50I like this draft, go ahead and send this draft to Dave, right?
00:11:53And then goes and sends that email, really simple example of what I want to do.
00:11:57Now, the challenge here, right, and the issue is that how do we build a system, right, which somebody else can contribute to that a third person kind of uses?
00:12:13How did this knowledge about what our security thing is come in?
00:12:18Presumably somebody else had been working on the same security questionnaire, right?
00:12:22So they had, let's say, a Hermes agent or whatever.
00:12:24They were working on it.
00:12:25You auto-saved some memory.
00:12:27Maybe somebody wrote down a skill.
00:12:28Somehow that piece has to come to my AI agent.
00:12:32How are we going to make that possible, right?
00:12:35Now, let's try obvious thing number one, right, which is that everybody writes shared skills for each other on GitHub.
00:12:42So the first time the security questionnaire was answered by your security person, everybody visualized like your security and compliance person in your head, right?
00:12:50Now, imagine that they, after answering the questionnaire, it sucks to answer questionnaires.
00:12:56After answering this gigantic Excel sheet of a questionnaire, they then went to GitHub and updated a shared skill, right?
00:13:04Many of you are fortunate to work with people who are modeled after our Lord and Savior Christ, who are so nice, who will go and update shared skills in a GitHub repo, right?
00:13:16Most people will not.
00:13:18Nobody is going to write skills for another person in GitHub.
00:13:24Like, that is not something that is natural to us, right?
00:13:27In the day-to-day of doing work, we don't suddenly decide that, ooh, this might be really useful for somebody else I don't even know, I'm not connected to in this situation in the future.
00:13:39Not happening.
00:13:40I can barely get it to, like, curate my own memory and my context.
00:13:44I do not have the time to send it to somebody else, to write it down for somebody else.
00:13:49Second, instead of having a company brain, why don't you do a team brain?
00:13:53Why don't you all just use one shared?
00:13:55Why doesn't the security team kind of use one more shared silo where you can do this, right?
00:14:01So build an agent and have it kind of save to memory itself, right?
00:14:04And that's kind of the architecture that I'm guessing a lot of you folks have with maybe something like a Hermes added to Slack.
00:14:09Does anybody have kind of a team brain situation going where you have an AI that multiple people use that auto-saves memory and auto-adds context?
00:14:18Does anybody have kind of, like, a skill that does that already, just for a small team?
00:14:23One. Anybody else? Okay.
00:14:24Okay, a few of you have that. That's cool.
00:14:26This is nice, but the problem is it's still not a company brain because it's still isolated, right?
00:14:31So it's like one more silo, right? Like, for example, if this gets, like, with claw tag, it has a per-channel memory, right?
00:14:40So in every channel, it gets saved, but now it's another silo in that one channel, right?
00:14:46So now it's, again, locked into one place that can't be used anywhere else.
00:14:50So if somebody got added to the channel, it would work, but otherwise it wouldn't work, right?
00:14:55And so this is the third option.
00:14:59The third option is saying all context goes into a single shared wiki.
00:15:06A wiki is a set of markdown files, and markdown files can link to each other.
00:15:08So imagine a gigantic folder.
00:15:10The folder has lots of markdown files, right?
00:15:13And markdown files can link to each other.
00:15:16So all context, instead of saving it inside a folder or siloing it,
00:15:21you put it in a markdown file, the equivalent of a markdown file,
00:15:25and you let it link with each other.
00:15:28The second thing that you do is you allow each file to have scopes
00:15:33on who can have read/write access to that file.
00:15:38The third thing that you do, which is the most important,
00:15:42you don't let the agent auto-add the memory.
00:15:49You don't let it auto-add because if it auto-adds, you have no idea what happened, right?
00:15:55You can't -- we're back to kind of the same world where some stuff is getting added,
00:16:00and as long as you are in that agent's memory, you're lucky, right?
00:16:04So the third thing that you do is instead of letting the agent auto-add,
00:16:07do something that allows your agent to suggest what is added with what scopes,
00:16:16and then have the human accept or reject.
00:16:21So it's not as heavy as GitHub, where I have to go and write this, update a shared skill,
00:16:28do a PR review, and then get it merged.
00:16:30But it's also not as YOLO as the memory just kind of being auto-written by the agent,
00:16:36right? It's kind of the sweet spot where, while you are working,
00:16:40you pop it up, suggest the right scopes, and let somebody add it.
00:16:45So now what happens is with this very simple addition, right, you are able to let people add
00:16:51to a gigantic wiki, but you let that person take on responsibility for what they can see or not.
00:16:57So if I'm adding something to the finance wiki, I want to make sure -- I'm adding something
00:17:01that's sensitive, I want to make sure it has a finance scope.
00:17:03If I'm adding something that's personal, I want to make sure that that's personal scope.
00:17:06Let me show you an example UX of what that might look like. This is what we do.
00:17:15This is a recent email that I got from one of our sales reps adding me onto a call.
00:17:33I looked at that email, helped answer it, and then I got a little box that suggested
00:17:39a bunch of bullets that told me what it's going to add, right?
00:17:43And when I hit add to wiki -- and so now it's much easier for me to review what is getting added.
00:17:47I don't care. I don't care if it gets added into this markdown file, that markdown file,
00:17:52what links that the agent takes care of. What I care about is are these facts correct?
00:17:58If these facts are correct, I'm going to hit add to wiki and I'm going to be done, right?
00:18:02And during the time of add to wiki, I can choose what scopes need to be added per wiki page or not,
00:18:08right? So each wiki page itself can get a certain set of scopes that you want to decide
00:18:12who gets access to what, for example, right? So for example, my email, this is the wiki page
00:18:17that I have for my emails and how my emails are prioritized, and I can now decide who gets access
00:18:22to this, who are the owners for this and what the RBAC for this is. So whatever the system looks like
00:18:26is up to you folks, but the core idea is that you want to get the agent to suggest a change
00:18:32instead of doing the change. All right. So two rules. One, make sure that everything goes into one
00:18:38company-wide wiki don't back down from this rule. Second, make sure that, like, as a part of that,
00:18:44every change is backed by a human's name. Nothing should be allowed inside the wiki that is Claude added
00:18:51this or like your AI agent added this or Hermes added this. No, Tanmay added this. That name needs to be
00:18:57there so that you can tie it back to this is the person who screwed up and, like, allowed everybody to see,
00:19:04like, everybody's comp, right? And, like, whatever. Now you can take remedial action,
00:19:09right? Whatever that is. Put them on a pip. You didn't know how to edit a wiki. So that is very,
00:19:14very important. And rule number two, once you decide that, you can then go to the second scope of, like,
00:19:19okay, you've got to make it easy for them to do that, which is where this business of scopes come in,
00:19:23where you want to scope each file according to who gets access. You're going to build kind of a system
00:19:26around it. This is what an architecture diagram of that looks like, where you have users,
00:19:32users talk to the agent. Agent, when it's reading context, uses that particular user's claims, right?
00:19:39So if I am reading something for solving a finance problem, it's using the finance claim to read as
00:19:46me because I had access to the finance wiki so I can read it and that is done every single time,
00:19:52right? So the agent is always using the user's credential to read the right part of the wiki.
00:20:00All right. I'm fairly out of time for the second use case. So what I'm going to do is give you a
00:20:06quick flavor of the second use case, but extend this idea. This is the daddy use case. This is like,
00:20:13this is the big daddy use case. This is a really complicated use case because now
00:20:17it's not just one person answering an email. It's a bunch of us using the shared context
00:20:25to solve a problem with various different escalation like privilege levels at the same time,
00:20:31right? And these are kind of the, these are the, these are the interactions with AI where the most
00:20:36amount of company brain knowledge is created, right? For example, I'm going to show you a quick real life
00:20:42example of what it looks like for us. So this is a case from an SRE situation where
00:20:50somebody was like, Hey, our auto learning, our wiki learning, fairly meta was failing. It wasn't
00:20:57working. What's going on? Right? And then it starts doing the investigation and it sucks because it's,
00:21:02it didn't have a skill. It failed. So it's like, bro, don't do this. Please use this open telemetry span
00:21:07name. Used an open telemetry span name. It did a slightly better job, but it was still really slow. So
00:21:13he looked at the code and he's like, Oh, you're using a like query. You're, you're a, you're a dumbass.
00:21:18This is opus 4.5. Um, like, like don't do this. Right? So then he's like, don't use a like query,
00:21:24use an equals to query. Right? And then it does equals to query and it surfaces some details and it,
00:21:29and then he says, Oh, dig deeper into this. And it says, whatever, this is a line of code where the
00:21:33error is coming from. Simple stuff, right? This is now where it surfaces some knowledge and says,
00:21:39aha, I learned that I should use equals to and not like, right? I learned that if you have a custom
00:21:45prefix added to wiki page names, it can cause issues. Right? Um, so it offers these learnings
00:21:50that you can choose to accept. So he kind of went dug in deeper, um, into what the problem was.
00:21:56Somebody else joined the conversation, right? And said, the technical decision that we've made here is
00:22:02wrong. Why is this happening? And now two people start to have an argument, right? They have an argument.
00:22:09Saying, Hey, it should not be like this. It should be like this. But why is it like this? But it should
00:22:12be like this, right? That argument creates knowledge because the actual problem was that somebody made
00:22:17a technical decision that was not documented. Right? When they decide to fix that issue, and they
00:22:22observe that that is indeed the root cause, and they decide that this is the way it's going to be fixed.
00:22:26Hey, we should remove this prefix that's causing a problem, whatever, whatever the thing is,
00:22:31that creates the highest quality context to be added to your brain. Because the previous suggestion was to
00:22:38say, uh, pages should not pages have a prefix, but the fact that pages have a prefix is a problem.
00:22:44Right? So now the thing that you're documenting in the brain is pages should not have a prefix.
00:22:49If they have a prefix, it can cause lookup issues in prod. This happens when multiple people talk to each
00:22:54other. Right? And solve problems together. This is what happens in a Slack thread when two people talk to each
00:22:59other and solve a problem. It creates the highest quality context. But, and so that's kind of what
00:23:05you want here. But the challenge is that the privilege escalation around this becomes very,
00:23:10very serious. If you're building an agent that can do everything surrounded by multiple people,
00:23:18that's scary. Because the engineer was allowed to do the PR work, but now I can use the same agent to
00:23:23deploy to prod. That's too scary. I can't have a conversation where I debug and
00:23:29deploy securely. Right? Especially if you're in a bank. Right? Like the people who are debugging,
00:23:35deploying to staging, setting up an alert, and deploying are not the same. But being the same has
00:23:40a lot of value because that's where all the knowledge is. Right? And so that kind of brings us to the
00:23:45second architecture, which I'm not going to get into too much detail with. But think of it as the same
00:23:49idea where user credentials and claims were used to read context. Instead of that, also use user credentials.
00:23:58Right? When the code is executing tools. So never store credentials in the sandbox.
00:24:05Instead of that, at the HTTP layer, at the SQL layer, inject the user's credentials, allowing the AI to behave as
00:24:13the human in a particular interaction. Right? So there's interesting details here. But that is what
00:24:20allows a shared AI to work with shared context. Right? And those are kind of the two key pieces to
00:24:25work with. So I would summarize, and this architecture is not particularly complicated, but it's very simple
00:24:30to work back from these two rules. Do not store credentials in the cloud sandbox. And second,
00:24:36virtualize all interactions with real data, proxy it, virtualize it, whatever word you want to use,
00:24:41and let users control them. So the user who adds a particular tool should control who gets access to
00:24:48that particular tool. So you can derive this entire thing if you just kind of follow these four principles
00:24:53and work backwards from that. There's only one architecture that is possible that makes sense
00:24:58in how you manage context and what constraints you set up and how you manage tools and what security
00:25:02rules you set up. That is my time. And so I'm happy to chat more after the talk. We have a booth as well,
00:25:10so happy to chat more through that on what the nuances inside this architecture are. I'm Tanmay Goh on
00:25:16Twitter. We're called PromptQL. Do check us out. At the end of the day, with the AI engineering community,
00:25:24we're going to do a product launch. And so I would love to share that, folks, share that with everybody.
00:25:30I'm going to take a picture with everybody on stage so that I can share that. And so let me do that while I'm here.
00:25:40All right. Do folks want to say cheese?
00:25:45Thank you so much. So watch out for that. It's our approach to Cloud Tag, which is PromptQL Tag,
00:25:52which is very similar to the ideas that we chatted about here, except that you're not stuck to Cloud.
00:25:57You can use GLM, and you can use GPT, and then Sol comes out, and we can use that and have a lot of fun.
00:26:03So do check that out. And otherwise, I'll see you folks soon.
00:26:19We'll be right back.