Transcript
00:00:00When it comes to AI, there are enough things to worry about.
00:00:04There also are things to look forward to, don't get me wrong,
00:00:06but there are enough things to worry about.
00:00:09The job transformation, job loss, how it will impact the job market,
00:00:13obviously, how it will impact society with deep fakes
00:00:17and all that slop on the internet and social media being worse than ever,
00:00:23in my opinion, at least.
00:00:24But the thing I want to look at today is cybersecurity and some implications
00:00:30that may not be obvious, which I also worry about.
00:00:34Now, you may have heard that OpenAI accidentally hacked Hugging Face.
00:00:39They published a blog post about this.
00:00:41I'll link it below, of course.
00:00:42And in there, they share the details on how one of their models
00:00:47in its evaluation environment during testing escaped that environment
00:00:52through a zero-day exploit or a zero-day vulnerability,
00:00:56which it detected so that it was able to break out of its sandbox
00:01:00and ultimately, through multiple steps,
00:01:02then compromise Hugging Face systems, the servers of Hugging Face.
00:01:08Now, all the details are in the blog post.
00:01:10That's not primarily what I want to talk about.
00:01:12We'll not do a deep dive into this incident.
00:01:14But you probably heard about it.
00:01:15And it is pretty amazing that the model was able to do that.
00:01:19And it's also pretty worrying, and I'll get back to why exactly I find it worrying,
00:01:23which may not be for the reason you think it is.
00:01:26But that's one example.
00:01:27Then, yesterday, Anthropic also published a blog post about three incidents they found.
00:01:35In their blog post, they start by mentioning the OpenAI incident
00:01:39and that that led them to take a closer look at all the evaluation runs they did.
00:01:44And after reviewing 141,000 evaluation runs,
00:01:50they found three cases, three main incidents that happened
00:01:55where their models also were able to break out of the testing environments
00:02:01and hack other companies, other websites, if you will.
00:02:07And they did not detect this before.
00:02:11Again, the blog post is below if you want all the details.
00:02:13In the first incident, we're talking about Claude Opus 4.7.
00:02:17So we're not talking about Mythos here.
00:02:19This dates back to April, I think.
00:02:21And in there, the model essentially was able to break out of its sandbox
00:02:30because the provider of the testing of the evaluation environment made a mistake
00:02:35or there was a misunderstanding, as they say in this article.
00:02:39And even though the model shouldn't have direct internet access
00:02:43and was told that it doesn't have direct internet access,
00:02:46it did have internet access, it discovered that.
00:02:50And then achieved the goal it was given by actually infiltrating the servers of a real company.
00:02:58That's the first incident.
00:02:59In the second and the third incident,
00:03:00we also have the pattern of the model having internet access even though it shouldn't.
00:03:04So that's the consistent pattern here.
00:03:07In the second incident, in order to achieve its goal,
00:03:09the model built a malicious Python package and published that
00:03:13and went to great lengths to actually publish it,
00:03:18creating an email account which was not trivial because it had to find a provider
00:03:23that didn't require a phone number.
00:03:26But it went through all these steps to finally be able to sign up with the Py
00:03:29Py registry for Python and publish a malicious package.
00:03:33And it did that to achieve its simulation tasks, its evaluation tasks,
00:03:38which were meant to stay on the sandbox,
00:03:40but it did that on the real registry out there in the internet.
00:03:43And therefore, the package was downloaded, for example, by,
00:03:47and I find that a bit funny, by a security company,
00:03:50which routinely downloads packages.
00:03:53And it installed that package, didn't discover that it contained malicious code
00:03:57and indeed got hacked by it.
00:03:59And some data got exfiltrated.
00:04:02And mythos, this was a mythos model, continued and didn't really realize
00:04:07or didn't really care, at least in the end,
00:04:10that it was dealing with the real internet and no longer in a simulation environment.
00:04:16It did actually realize that this would not be okay to do out there in the real world,
00:04:20but it didn't really care to find out if it was maybe doing stuff in the real world.
00:04:25That's the second incident.
00:04:26And in the third incident, it's kind of similar,
00:04:29same pattern of the model having internet access when it shouldn't.
00:04:33And in order to perform its task, which was meant to be done inside of its sandbox,
00:04:37it suddenly did stuff on the real internet,
00:04:40kind of replicating its task outside of the sandbox, if you will.
00:04:44Now, again, the details are in the blog post.
00:04:46What I worry about is, of course, for one,
00:04:51that the models are good at hacking, if you want to call it like this.
00:04:56This is obvious.
00:04:57And in general, I mean, the models we have these days
00:05:01and used in the right agent harnesses are really good at achieving tasks.
00:05:06If you have a system where there is no straightforward way
00:05:11of opening a PDF document, let's say,
00:05:14and you put a PDF there and you tell a model in cloud code,
00:05:19in codex, in pi, whatever, to extract the content of that PDF document,
00:05:24it will most likely find a way of doing so.
00:05:27And if it has to write its own PDF parser,
00:05:31the models are really good at solving a task
00:05:36or achieving a goal if they can validate it.
00:05:41So if you have a loop where the model can find out whether it achieved its task,
00:05:46they are really good at keeping on going, again, when used in the right harness,
00:05:51until that task is achieved, until it's completed.
00:05:54And, of course, hacking in the end is just such a task.
00:05:58If you give a model the goal of compromising system X,
00:06:05it will keep on going until it did so.
00:06:07Now, of course, it is possible that it doesn't find a way
00:06:10if the system is secure enough, but it will get creative for sure.
00:06:15At least, and that's my main point here, actually,
00:06:18when companies like OpenAI or Anthropic use these models.
00:06:22If you or I do it, it will not get there.
00:06:25There are guardrails in place,
00:06:27and unless you find a way of working around them,
00:06:30which may be possible but is tricky and is a task on its own,
00:06:33if you don't find a way of working around them,
00:06:35the models you and I are working with,
00:06:38they can't do it or they are far worse at that,
00:06:42at hacking and at the bad stuff
00:06:44than they are when OpenAI and Anthropic do it.
00:06:48And that is actually one thing I deeply worry about.
00:06:53We have this pyramid of capabilities,
00:06:58as I like to call it.
00:07:00And at the top, we have the AI labs.
00:07:04So they sit here at the top.
00:07:07They have the full power of these models.
00:07:11They can fine-tune them
00:07:13to do exactly what they want them to do.
00:07:16They don't have to impose any guardrails at all
00:07:20if they don't want to, and why would they?
00:07:23And therefore, they can really leverage these models
00:07:26and nudge them in all the directions they need to.
00:07:28They can build amazing research assistants.
00:07:31They can build amazing assistants
00:07:33that can deal with cybersecurity stuff.
00:07:36They have all the power regarding that.
00:07:39They don't need guardrails necessarily.
00:07:41So that is, of course, pretty frightening
00:07:44that you have essentially maybe a handful of companies,
00:07:48maybe just two or three or four,
00:07:51depending on how you count it,
00:07:53that have the full power there.
00:07:55And obviously, they're using that power
00:07:57mostly to develop better models,
00:07:59but they can do all kinds of stuff.
00:08:03Then you have the government.
00:08:06And that is the US and the Chinese government, obviously,
00:08:10because these are the only two countries
00:08:15that have relevant model providers.
00:08:17The government and its agencies,
00:08:20they can, of course, and I'm sure they do that,
00:08:23ask the AI model providers
00:08:26to give them the raw power models
00:08:29or some specifically fine-tuned models.
00:08:34That is certainly happening or about to happen.
00:08:37I have no doubt about that.
00:08:39So that gives their agencies,
00:08:41NSA, CIA, and of course,
00:08:42the counterparts in China
00:08:44full power when it comes to cybersecurity mostly,
00:08:48but again, whatever you need.
00:08:50And then you have you and me, the peasants.
00:08:54Well, actually, maybe we have another layer
00:08:57now that I think about it.
00:08:59Maybe there is another layer in that pyramid.
00:09:02Maybe we are actually here at the bottom
00:09:07and maybe there are big companies,
00:09:12if you want to call them like this here.
00:09:15You could argue that maybe there are privileged companies
00:09:19that are authorized, so to say, by the government
00:09:24to also get access to more powerful models.
00:09:28So that may be a case.
00:09:30We saw that with mythos,
00:09:32that there is a tier of companies
00:09:34that have access to mythos,
00:09:37whereas the rest of us gets fable.
00:09:39So we, the peasants, are stuck with that.
00:09:41Now, we can also divide the peasants.
00:09:45We have the US
00:09:49and maybe in the future, also China peasants.
00:09:53And with that, I don't just mean normal people.
00:09:56I also mean all the smaller, less important companies.
00:09:59So we have those.
00:10:01And I absolutely see a future
00:10:03where we have the rest of the world peasants
00:10:08in Europe and everywhere else.
00:10:11And we have to work with what's left for us.
00:10:15The models that are authorized for non-US citizens,
00:10:19for example.
00:10:20That's the pyramid.
00:10:21And that is what I worry about a lot.
00:10:25And of course, that's related to the other stuff.
00:10:28If we have models that are very good
00:10:31at compromising systems,
00:10:33then as a company,
00:10:35as countries,
00:10:37we need systems
00:10:38that are good at defending
00:10:40against these attacks.
00:10:42And also outside of the cyberspace,
00:10:45if we're talking about warfare,
00:10:47if we have systems
00:10:48that are good at developing weapons,
00:10:50doing intelligence,
00:10:52then countries also need ways of defending.
00:10:55But that's a whole different level, right?
00:10:57But there is this difference
00:11:00between the privileged countries
00:11:02and companies and the rest.
00:11:04And if there are models
00:11:05that are good at hacking,
00:11:06good at compromising systems,
00:11:08but those models
00:11:09are only available here
00:11:12in the top three tiers of the pyramid,
00:11:15then the lower tiers
00:11:17doesn't look so good, does it?
00:11:21So that is what I'm really worried about.
00:11:25That AI is allowing us
00:11:28to do many amazing things
00:11:29that it can maybe lead
00:11:31to break through research.
00:11:33That is all very exciting.
00:11:36It's less exciting
00:11:37that it looks like
00:11:40we'll not all be able
00:11:41to be part of that world.
00:11:43I hope I'm wrong here.
00:11:46I hope I'm very wrong.
00:11:48I hope that maybe
00:11:49if I'm allowed a little joke,
00:11:52Europe will catch up.
00:11:55just for the simple fact
00:11:57that I live there
00:11:58to be a bit egocentric here.
00:12:00But yeah, I hope that I'm wrong
00:12:02and I hope that maybe
00:12:04and that's the one little light I see there
00:12:07that maybe the fierce competition
00:12:09between these companies
00:12:10and between these countries
00:12:11at least buys us more time
00:12:14until that pyramid fully takes shape.
00:12:18But maybe I'm also too pessimistic here
00:12:21or maybe AI doesn't get better and better.
00:12:27I'm not sure about that,
00:12:29but I'm very interested
00:12:30in hearing your thoughts, of course.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video