This Tiny Coding Model Shouldn’t Be This Good (MiniCPM5)

English
BBetter Stack
Computing/Software

Transcript

00:00:00This 2 billion parameter model claims to beat a 4 billion one on SWE Bench,
00:00:05but run the same model at 4-bit using the exact settings in its own docks,
00:00:10and it can fall into an infinite loop over 90% of the time.
00:00:14Now, that could be true, but it's also true the broken version is fixable with one flag.
00:00:18This is Mini CPM5 2B from OpenBMB.
00:00:22Is a 2 billion parameter model good enough to actually be used as an agent?
00:00:26Let's take a look.
00:00:30Now, small models have had the same problem for a while.
00:00:35They are getting pretty good at chat.
00:00:38Give them tools, though, and things usually fall apart.
00:00:41Look at SWE Bench Verified.
00:00:43These are real GitHub issues when the model has to produce real patches.
00:00:48Quinn 3.5 2B scores 5.
00:00:50Gemma 4 E2B scores 2.
00:00:53Those are vendor numbers, but match what anyone who's tried small models for agent work has probably seen.
00:00:59The model hallucinates the tool schema, or forgets it, or just loops.
00:01:03Then Mini CPM5 2B shows up with a score of 46.
00:01:08Not only way above the other 2B models, but also above Quinn 3.54B, which is twice its size.
00:01:15So what's changed?
00:01:17OpenBMB points to three things.
00:01:19The architecture is plain llama.
00:01:21There's nothing exotic here.
00:01:22It has about 128,000 native context, and most importantly, the training was built around agents.
00:01:29500,000 agent trajectories during supervised fine-tuning, then reinforcement learning, then 16 separate RL expert models merged into one.
00:01:39It's Apache 2.0 with open datasets, and just to avoid one easy point of confusion, this is the text-only Mini CPM5 model.
00:01:48Now let's actually run it.
00:01:49If you enjoy coding to a little speed up your workflow, be sure to subscribe.
00:01:52We have videos coming out all the time.
00:01:54All right, now for MLX, I'm on a Mac here, we can just do a pip install in a virtual environment, which I got set up here.
00:02:02Older versions of MLX LM ignore this model's end-of-sequence list, so the model literally doesn't know when to stop talking.
00:02:10Then we can run the first prompt to pull the model in the weights.
00:02:13Just give it a second, it fires up the first time, and here it goes.
00:02:19Once we run this, you get the think block.
00:02:22This is how it thinks.
00:02:24Then after this, it produces that final answer.
00:02:27And there's the tokens per second counter as well.
00:02:29All of that is coming from a 1.3 gigabyte file.
00:02:32Now we can even set this to be an interactive chat too.
00:02:36So when I drop this line in here, boom, it fires up.
00:02:39You can see now it's always listening, so we can just ask a question on the go.
00:02:44And the responses that we get fire right back at us.
00:02:47Now, chat is cool, but the tools are just as good here.
00:02:51I'm going to start up Llama CPP now in a new terminal, then point an OpenAI client at it and give it a simple weather tool.
00:02:59The model produces a structured function called Llama CPP, parses it, and the response comes back as a proper tool calls array.
00:03:06So now you can think of all the different ways this could run cleanly on your system.
00:03:11Now, there's also a reason compatibility is this clean.
00:03:14If we take a step back, this is a dense 2.5 billion parameter model with 42 layers and grouped query attention.
00:03:22But the important line is the architecture here.
00:03:25Llama for casual LM.
00:03:28Mini CPM 4 isn't that.
00:03:30It used custom sparse attention and multi-token prediction.
00:03:33That's nice and all, but every runtime needed special support.
00:03:36Mini CPM 5 basically throws that away.
00:03:39If your runtime can load Llama, there's a good chance it can load this.
00:03:43MLX, Llama CPP, Web LLM.
00:03:45So compatibility is one of the strongest parts of this model.
00:03:49But now we get to the whole benchmark table, because this is where it gets a lot less clean.
00:03:54If you only read the table one direction, Mini CPM 5 looks ridiculous.
00:03:58It's against Quen 3.54B, a model twice its size.
00:04:02Live Code Bench has them 69 to 56.
00:04:05SWE Bench Verified, 46 to 34.
00:04:07You can check these rankings on other platforms as well.
00:04:11So the comparison here is kind of ridiculous.
00:04:13That sounds like a 2B model replacing a 4B model.
00:04:17Now look at the table in a new light.
00:04:20SWE Bench Pro, 14 versus 28.
00:04:22Mini CPM 5 gets half the score.
00:04:25Terminal Bench, we have about an 8.5 versus around a 26, about a third.
00:04:30Quen 3.54B also wins on other platforms, MMLU Pro, GPQA Diamond, and LogBench V2.
00:04:38And on plain instruction following, even Quen 3.52B beats Mini CPM 5.
00:04:43So when they call this 4B class, there's some truth to it.
00:04:46For code, tool calls, math, and parts of the agent workload, absolutely.
00:04:50Across the whole board, yeah, no, probably not.
00:04:53The bigger problem, with this model specifically, is what happens when you run the quantonized
00:04:58model using the documented settings.
00:05:00Someone on the GitHub tracker tested HumanEvalPlus against the GGUF files.
00:05:05Q8 scored about a 43, and 55% of generations ran away into repetition.
00:05:11Q4KM scored around a 6, and 92% ran away.
00:05:1592%.
00:05:16That looks like a broken model, except then they changed one thing.
00:05:21Repeat penalty 1.15.
00:05:24Q8 jumps from around a 43 to a 92.
00:05:27Q4 jumps, 6 to 71.
00:05:30OpenBMB later said on Hugging Face that Llamas CPP's default min P value is 0.05.
00:05:37That's the real problem, and that it should be set to zero.
00:05:40But that wasn't in the docs when people ran this.
00:05:42At 2 billion parameters, one bad default can turn a useful model into, well, nothing really
00:05:47great.
00:05:47The sampler isn't just tuning anymore, it's part of the actual model.
00:05:51And there are a few more things you should know before using it.
00:05:54Every benchmark here is vendor RAN.
00:05:57So, the comparison table only includes QN and Jemma.
00:06:00No LLAMA.
00:06:01No fee.
00:06:02No small LLM3.
00:06:04Not even a mini CPM4 row showing what improved generation to generation.
00:06:09Someone asked for the evaluation scripts in an issue.
00:06:12No response.
00:06:13It's also not in the LLAMA library, so you need to write a model file.
00:06:18There's no LM Studio page either, you drop the GGUFN manually.
00:06:23Thinking is enabled by default, so every answer has to reason before it starts giving you the
00:06:28response.
00:06:28There is an enable thinking switch in the chat template, but the official docs only provide
00:06:34a sampling profile for think mode.
00:06:35Nobody has measured what it's like with thinking disabled yet.
00:06:39English and Chinese only, more than 100,000 downloads for the 2B model, 600,000 for the
00:06:451B.
00:06:46People are running these, they're just not talking about them that much.
00:06:50So, would I actually use mini CPM5 2B?
00:06:52Well, these are really cool to play around with.
00:06:55They're cool to see where local stuff is going.
00:06:57For a local coding agent or maybe like a MacBook Air, I mean, yeah, okay, it could be kind of
00:07:02cool.
00:07:02I'd run the MLX4 bit or the Q8GGUF, I'd set them in P to 0, set the repeat penalty to what
00:07:10they actually recommend.
00:07:11For that job, it might be pretty strong for that size, right?
00:07:15We can see where these local models are going, give them a test and see how they improve over
00:07:19time.
00:07:20For anything that's like a really big task, I would still take Quen 3.54B over this.
00:07:25And that's not me disagreeing with OpenBMB, I've tested other models they have.
00:07:30Some are really, really good.
00:07:31The interesting thing here isn't really that a 2B model can beat a 4B model, it's how
00:07:36close to the edge these small models are actually getting.
00:07:39Mini CPM5 itself is fine.
00:07:42The defaults are what can break it.
00:07:44I'm Josh from BetterStack.
00:07:45If you enjoy coding tips and tricks like this, be sure to subscribe.
00:07:48We'll see you in another video.

Key Takeaway

MiniCPM5 2B matches larger models in specific coding and tool-use benchmarks, but requires specific sampler configuration overrides to prevent severe repetition loops.

Highlights

  • MiniCPM5 2B scores 46 on SWE Bench Verified, outperforming models twice its size such as Qwen 3.5 4B.

  • The architecture uses a plain Llama causal LM structure with a 128,000 native context length and training involving 500,000 agent trajectories.

  • Running the quantized model at 4-bit with default settings causes it to fall into an infinite loop over 90 percent of the time.

  • Changing the min_p value to zero and setting a repeat penalty of 1.15 resolves the repetition issues in quantized versions.

  • Vendor benchmark tables show Qwen 3.5 4B outperforming MiniCPM5 on SWE Bench Pro, Terminal Bench, MMLU Pro, and GPQA Diamond.

Timeline

Model Overview and Benchmark Claims

  • MiniCPM5 2B claims a score of 46 on SWE Bench Verified.
  • Small models historically struggle with tool use and hallucinate tool schemas.
  • The model uses a plain Llama architecture with 128,000 context and agent-focused training.

Small parameter models typically fail when given tools or real GitHub issues, often hallucinating schemas or entering endless loops. MiniCPM5 2B deviates from this pattern by scoring 46 on SWE Bench Verified, surpassing models double its parameter size. OpenBMB attributes this performance to a standard Llama architecture combined with 500,000 supervised fine-tuning agent trajectories and merged reinforcement learning expert models.

Local Execution and Tool Integration

  • Older versions of MLX LM ignore the end-of-sequence list without proper updates.
  • The model runs locally from a 1.3 gigabyte file with an active thinking block before final output generation.
  • Llama CPP parses structured functions from the model into proper tool call arrays.

Running the model locally via MLX on a Mac requires virtual environment installation steps, and older software versions ignore end-of-sequence tokens. The model generates a visible think block followed by the final answer tokens. Integration with Llama CPP allows the model to produce structured function calls and parse them correctly through an OpenAI client interface.

Architecture and Benchmark Nuances

  • MiniCPM5 adopts a dense 2.5 billion parameter layout with 42 layers and grouped query attention.
  • The transition to a standard Llama causal LM architecture improves compatibility across runtimes.
  • Qwen 3.5 4B outperforms MiniCPM5 on SWE Bench Pro, Terminal Bench, and instruction following.

Discarding the custom sparse attention and multi-token prediction of its predecessor allows MiniCPM5 to load seamlessly across runtimes like MLX, Llama CPP, and Web LLM. While benchmark comparisons against Qwen 3.5 4B favor MiniCPM5 on Live Code Bench and SWE Bench Verified, larger workloads like SWE Bench Pro and Terminal Bench show Qwen winning by significant margins.

Sampler Settings and Quantization Fixes

  • Running the 4-bit quantized model with default settings leads to a 92 percent repetition rate.
  • Setting the min_p value to zero and a repeat penalty to 1.15 fixes the runaway generation behavior.
  • Vendor evaluations lack comparisons against Llama or Phi models, and thinking mode remains unmeasured with the switch disabled.

Default Llama CPP sampler settings, specifically a default min_p value of 0.05, cause quantized versions of MiniCPM5 to break and loop endlessly. Adjusting the repeat penalty and setting min_p to zero recovers performance scores dramatically across GGUF files. Despite these configuration hurdles and missing evaluation scripts, the model demonstrates the shrinking gap and rising capability of small-scale local architectures.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video