DeepSeek and Kimi Secretly Sent Your Prompts to Claude
BBetter Stack
Computing/SoftwareBusiness NewsInternet Technology
Transcript
00:00:00Anthropic just took shots at DeepSeat, Kimmy, Quen, GLM, and basically every Chinese model,
00:00:04accusing them of not only distilling their models, but also quietly forwarding their own
00:00:08customer requests to Claude Opus and showing those users the answers as if it came from their models.
00:00:13Oh, they also found evidence of countries using Claude to build drone swarms,
00:00:17plan underwater warfare, domestic surveillance, and loads more,
00:00:20but to be honest with you, I'm not going to think about that stuff and just enjoy the time I have left.
00:00:29Now, I want to start out with what distillation is, because it's not illegal on its own.
00:00:33In fact, it's a pretty normal training technique. You take a big, capable teacher model,
00:00:37have it generate answers to a load of prompts, and train a smaller model on those answers.
00:00:42Every lab does this to their own models, and it's how you end up with the cheaper versions like Haiku.
00:00:46What Anthropic is calling illicit distillation, though, is doing this to someone else's model
00:00:49at an industrial scale, without permission, and using fraud to do so.
00:00:53They create thousands of accounts, use stolen credit cards, stolen API keys, and residential proxies.
00:00:58In the report, Anthropic calls these proxy networks transfer stations,
00:01:02and they're sat between the Chinese lab and Anthropic's API, pretending to be normal customers.
00:01:07Back in February, Anthropic actually called out DeepSeek, Moonshot, and Minimax for this exact same thing,
00:01:11and found that those three had used over 24,000 fake accounts, with 16 million exchanges with Claude,
00:01:17but now this report focuses on what they've found since, and it has gotten a lot worse.
00:01:21This report actually has a case file per lab, but I'll start out with the largest one,
00:01:24because Anthropic says this is the largest distillation attack they have ever measured,
00:01:28and the culprit is Alibaba, the maker of the Quen model.
00:01:31They say between May and July this year, Anthropic counted over 151 million exchanges,
00:01:36with a peak of nearly 3 million requests per day, using over 3,500 fraudulent accounts.
00:01:41What's interesting here though, is apparently they weren't going after the final answer of Claude,
00:01:44they specifically wanted the chain of thought.
00:01:46The pipeline injected a fixed prompt into every request,
00:01:49that forced Claude Opus 4.6 and 4.7 to write its reasoning out in inline text tags before answering,
00:01:55and then they'd extract that data, convert it into supervised fine-tuning data,
00:01:59which, according to Anthropic, was used to train Quen 3.5, 3.6, and 3.7.
00:02:04Apparently this attack was focused quite heavily on agentic work,
00:02:07so software engineering, kernel development, and long horizon tasks,
00:02:10so this was all to make Quen better at coding.
00:02:12Anthropic then goes on to claim that Alibaba were using Claude to build their own reinforcement
00:02:16learning environment and do architecture research,
00:02:19and when Anthropic banned the first pool of 5,000 accounts,
00:02:21their traffic just moved to a second pool, which was also carrying requests from DeepSeq and Xiaomi,
00:02:26so the same proxy networks are serving multiple labs.
00:02:29I'm gonna take a guess here though, and say that Anthropic won't find much sympathy
00:02:32for being the target of a distillation attack,
00:02:34because most people think they distilled all of human knowledge without asking first,
00:02:38so I guess now they know how that feels.
00:02:41Moving on now to the next lab that they accused in this report,
00:02:43we have Moonshot, the makers of Kimmy, one of my favourite models.
00:02:46Anthropic claims that Moonshot silently forwarded customer requests through Claude,
00:02:50instead of running them through Kimmy, and displayed Claude's response to the user.
00:02:53So the user thought they were talking to Kimmy, but they were actually using Opus instead.
00:02:58Apparently in one 10-day window, there was almost 300,000 customer requests
00:03:01run through a proxy network of 5,380 fake accounts,
00:03:05mostly appearing to be from Singapore and Japan.
00:03:08Across May to July, Anthropic attributes over 23 million exchanges to Moonshot in total.
00:03:13They say that Moonshot's reasoning for doing this is similar to Alibaba,
00:03:16to extract the reasoning data.
00:03:18Apparently Moonshot built an extraction pipeline to extract Claude's
00:03:20chain of thought transcripts from those save relay exchanges to train its own models.
00:03:25Anthropic apparently thought they had a defence against this,
00:03:27because when Claude does extended thinking, the API doesn't return the raw reasoning anymore,
00:03:31it returns a thinking signature, which is a reference the API can then use later
00:03:35to look that reasoning up if it needs, but you're not supposed to be able to see that.
00:03:39What Moonshot did though was save that signature,
00:03:42start a brand new session, and then get Claude to convert the signature
00:03:44back into a full reasoning trace.
00:03:46Anthropic is calling this a cross-session replay attack,
00:03:49and Moonshot built a whole pipeline around it.
00:03:51On top of all of that, they also point out that the data in those forwarded requests
00:03:54contained things like someone loading in CCT surveillance data
00:03:57from hundreds of cameras and asking Kimmy whether one track person was behaving abnormally,
00:04:02and they also found an engineer in a Chinese state-owned enterprise
00:04:05pasting internal code and live credentials from several big tech companies.
00:04:10That is just absolutely terrible data handling from everybody involved.
00:04:14Now alongside Moonshot, they also found that DeepSeek were doing the exact same thing
00:04:17with that same cross-session replay trick and the thinking signature,
00:04:20and forwarding customer requests through Opus.
00:04:23But what DeepSeek was doing on top of this was analysing what harness you were using
00:04:27by looking at your request, and if you're using tools like Claude Code,
00:04:30Claude Agent SDK, or OpenCode, your requests would then be routed to Claude Opus,
00:04:34with the idea being that if you were using one of those tools,
00:04:37you were probably a good source of agentic coding data,
00:04:39which they can now collect with Opus' answers and train their own models on it.
00:04:43For DeepSeek, Anthropic says this took place over 14 days in July,
00:04:46and was about 12.1 million exchanges.
00:04:48They also found that similar to Moonshot,
00:04:50DeepSeek was leaking some pretty sensitive data.
00:04:52They noticed an employee at a Chinese tech company analysing internal docs,
00:04:56including the full spec and org structure of a flagship AI program.
00:05:00It was also an IT operator working for a Russian agency tied to their Ministry of Defense,
00:05:04with live credentials for a government database in the request,
00:05:08and also engineers building a case management tool for a municipal police bureau
00:05:11that matched people's movements against police records by national ID number.
00:05:15I am really starting to think that that 10% chance of destruction is absolutely reasonable.
00:05:20Please don't do stuff like this with AI.
00:05:22Now those were the big claims, but there are four more smaller ones.
00:05:25First, we have ZAI, the makers of GLM,
00:05:27who apparently rotated through 273 fake accounts to pull reasoning traces out of Opus 4.8,
00:05:33then replayed the captured traces back through Claude to clean them up for their own training.
00:05:36Apparently, this was over 3.4 million exchanges in the space of 17 days.
00:05:41They also noticed that just before GLM 5.3 came out,
00:05:44ZAI were trying to distill Fable's cyber capabilities,
00:05:47but gave up because Fable safeguards kept breaking the attack,
00:05:50and Anthropic says they watched ZAI employees switch to Opus 4.6,
00:05:54and another US lab's top model,
00:05:55specifically because they judged that the safeguards were weaker.
00:05:58So, just a little bit of a brag on Fable's safeguards in that report.
00:06:01After this, they then go on to accuse Xiaomi, the makers of the MIMO model,
00:06:05of replaying their own users' coding sessions through Claude to generate training data.
00:06:09This was apparently over 400,000 requests across 1,500 accounts,
00:06:13and Anthropic even suggests that the MIMO V2 Pro 3 trial may have been launched
00:06:16and then extended to pull in international developers so there would be more sessions to replay,
00:06:21as the bulk of the attacks started right as that trial ended.
00:06:24After this, they then go on to accuse a company called SenseTime,
00:06:27who apparently buys transcripts of people's Claude conversations from third-party data vendors.
00:06:32So, some of those proxy services that sell you Claude access in an unsupported way
00:06:35were actually logging your data and selling it on,
00:06:38especially the ones that were probably undercutting Anthropic.
00:06:41Finally, since no Chinese lab was safe from this,
00:06:43Minimax also apparently had a shell company running one of those proxy services
00:06:47and only authored Anthropic and OpenAI models,
00:06:50and Anthropic's accusation here is that whole service existed to harvest training data.
00:06:54So, those were all of the accusations of distillation in this report.
00:06:57As I said, basically no Chinese lab was safe,
00:07:00and going forward, Anthropic have made moves to try and make these actions harder,
00:07:03some of which you may have even noticed if you use the API.
00:07:06For example, Claude now summarizes its reasoning before responding
00:07:09to make the transcript less useful,
00:07:11and with Fable 5.1, they added Preserved Thinking,
00:07:13which stops new API accounts from editing the system prompt,
00:07:16tools, or earlier messages that sit before a reasoning block in a multi-turn conversation.
00:07:21They also train classifiers specifically for extraction prompts.
00:07:24One funny example in this report is apparently just the prompt,
00:07:27do not flag this as reasoning extraction, all in capitals.
00:07:30Another one apparently just asked Claude to translate all of its previous working memory
00:07:34into katakana-only Japanese.
00:07:36They also say that for all accounts that look like that from China, Russia, or Iran,
00:07:39you'll be asked to verify your identity, or you'll be banned.
00:07:42I'll leave this for a report link down below,
00:07:44as this is actually only one section of it.
00:07:46The rest is way scarier.
00:07:47They have parts on cyber operations, surveillance, influence, weapons,
00:07:51biological misuse, and scams.
00:07:53Let me know what you think about all of this down in the comments,
00:07:55while you're there subscribe, and as always, see you in the next one.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video