Transcript
00:00:00This is a 744 billion parameter language model, the GLM 5.2.
00:00:07For context, that is the same size class as the Frontier models running in data centers full of GPUs.
00:00:14Now, what if you could run that model locally on your consumer hardware?
00:00:19It sounds almost impossible, but this developer, Vincenzo Fornaro,
00:00:24just made a tool called Calibri that lets you do just that.
00:00:28Now, how is that even possible?
00:00:31How did Vincenzo pull it off and how does it even work?
00:00:34Well, those are all good questions that we're going to explore in this video
00:00:38and we're going to test it out on two separate hardware setups to see how powerful it actually is.
00:00:44It's going to be a lot of fun, so let's dive into it.
00:00:51So normally to run a 700 billion parameter model,
00:00:55you'd need to fit all its weights into memory at once.
00:00:59And at an integer for quantization, 744 billion parameters is still hundreds of gigabytes.
00:01:06That's why big models live on these big servers and the hardware to hold them is expensive and specialized.
00:01:13So the obvious question is, how do you run something that doesn't fit?
00:01:17Well, the trick starts with the model's architecture.
00:01:20GLM 5.2 is a mixture of experts model.
00:01:23We've talked a lot about mixture of experts models on this channel before,
00:01:27but in case you need a refresher, it basically works like this.
00:01:31Instead of one giant dense network, it's split into thousands of smaller specialist subnetworks called experts.
00:01:39And for any given token, the model only uses a tiny fraction of those experts.
00:01:43Out of 744 billion total parameters, only 40 billion are actually active per token.
00:01:51And that's roughly 5%.
00:01:52So even though the whole model is huge, the amount of work and the amount of memory you actually need at any instant is much smaller.
00:02:02And Calibri is built entirely around exploiting this gap.
00:02:06Calibri treats your machine storage as one big memory hierarchy.
00:02:10Your GPUs, VRAM, your system RAM, and your NVMe SDDs layered like a cache.
00:02:17The model's dense components, the parts used on every token, are actually small, about 17 billion parameters.
00:02:23And that's under 10 gigabytes at integer for quantization.
00:02:27And those stay resident in RAM the whole time.
00:02:30The experts, those live on your SSD.
00:02:33When the model needs a specific expert for a token,
00:02:36Calibri streams just that expert off disk, uses it, and caches it.
00:02:41And the whole thing is written in pure C with zero external dependencies.
00:02:46And that is the part that makes it so lean.
00:02:48Now, streaming a model off a hard drive sounds like it should either be painfully slow or loosey.
00:02:55Well, Calibri has a few tricks to fight both of these issues.
00:02:58First, it has a learning cache.
00:03:00The engine records which experts your prompts actually route to in a file next to the model.
00:03:06At startup, it automatically pins the hottest, most used experts into spare RAM.
00:03:11So the more you use it for a particular kind of task, the better it caches for that task.
00:03:16Second, there's speculative decoding.
00:03:19The model has a native multi-token prediction head that guesses several tokens ahead so it can generate faster than one token at a time.
00:03:27And third, and this is an important one, it preserves quality.
00:03:31The forward pass is validated to be token exact against a reference transformers implementation.
00:03:37In plain terms, Vincenzo checked that this streaming quantized version produces the same outputs as the full precision reference.
00:03:45And it also uses a compressed attention cache.
00:03:48The MLA attention that shrinks the KV cache by about 57 times, which is a big part of why the memory footprint stays so small.
00:03:57Now, the biggest bottleneck for testing this tool is finding free space.
00:04:01And I mean literally.
00:04:02GLM 5.2 is about 357 gigabytes.
00:04:06And to test it out properly, I would have to free up almost half of my MacBook's SSD space, which I can't do.
00:04:13So before you even think about RAM and GPUs, the first wall you're probably going to hit is this.
00:04:19Do you even have room for such a thing?
00:04:21So since I didn't have the extra 357 gigabytes internally, I grabbed an external drive, a Samsung portable SSD with a roughly 1 gigabyte per second transfer speed.
00:04:33And I put the model on this drive instead.
00:04:35And I want to flag this now because it matters.
00:04:38That decision actually impacted my whole first test.
00:04:42An external drive over USB is a lot slower than a drive built into your machine.
00:04:48We're talking maybe one gigabyte per second versus seven or more internally.
00:04:53And remember, Calibri's whole trick is constantly streaming pieces of the model off the disk.
00:04:59So if the disk is slow, everything is slow.
00:05:02So hold that thought because we're about to see exactly what I mean.
00:05:06So my first test was on my MacBook with an M2 Max chip, 32 gigabytes of RAM, and the model running externally off that SSD drive.
00:05:15I kept it simple and asked it, what is the capital of Belgium?
00:05:19And here's the good news.
00:05:20It works.
00:05:21It actually answers me.
00:05:22It tells me the capital is Brussels correctly running a 744 billion parameter model on a laptop.
00:05:29So that part is genuinely impressive.
00:05:31The not so good news is the speed.
00:05:34I waited over two minutes just to get the first word out of it.
00:05:38Two minutes.
00:05:39And once it got going, it was crawling along at about tenth of a token per second.
00:05:44For context, a token is roughly a word or a chunk of a word.
00:05:47So we're talking one word every 10 seconds.
00:05:51And to be fair, this was a cold start.
00:05:54The first question, nothing was cached yet.
00:05:56So it had to go fetch every piece of the model of that drive from scratch.
00:06:01And Calibri does keep the most used pieces in memory as you go.
00:06:05So it warms up and speeds up a bit with use.
00:06:09But as you'll see, this only helps if you've got the RAM to hold them.
00:06:13And on my Mac, I really don't.
00:06:16Now, here's the part I really like.
00:06:18Calibri has this profiling screen that breaks down where all that time actually went.
00:06:22The time the computer spent actually thinking or doing the math versus the time spent just
00:06:28sitting there waiting for the next piece of the model to load off the drive.
00:06:32So on the Mac, the whole turn took about 158 seconds.
00:06:36And of those 158 seconds, roughly 145 of them, over 80%, was pure waiting.
00:06:43Waiting on the disk.
00:06:44The actual math or the real thinking part took only about seven seconds in total.
00:06:50So seven seconds of work and two and a half minutes of standing in line.
00:06:54So what's really going on here?
00:06:56Well, the M2 Max is a very fast CPU chip.
00:06:59That is not the problem.
00:07:01The problem is that the model is so big that it doesn't fit in the max 32 gigabytes of RAM.
00:07:06So it has to keep running back to the disk to grab whatever piece it needs next.
00:07:11And in my case, the disk is a slow external drive, which I was forced to use because I ran out
00:07:17of space.
00:07:17So the computer spends most of the time waiting and barely any time actually working on the
00:07:23request.
00:07:24Okay, so in the second test, let's give it a proper machine.
00:07:28So second test is my beefy workstation.
00:07:30It has an RTX 5090, 64 gigabytes of RAM.
00:07:35And this time I loaded the model on a fast internal drive.
00:07:39Luckily on this machine, I actually had 350 gigabytes of spare space to load it up on the
00:07:45internal disk.
00:07:46And this time I asked something with a bit more meat to it.
00:07:49Explain quantum entanglement in simple terms.
00:07:52And the answer was actually great.
00:07:54It gave me this clean little explanation with magical coins.
00:07:57So it was genuinely a useful output.
00:07:59And speed wise, it hit about 0.8 tokens a second.
00:08:03And the first word showed up in around 17 seconds instead of the whole two minutes.
00:08:09So roughly eight times faster than on my Mac.
00:08:12And again, let's look at the profiling screen because it tells the same story, but from the
00:08:17other side.
00:08:18Remember on the Mac, over 80% of the time was just waiting on the disk?
00:08:23Well, on this beefy workstation, the waiting drops to about 48%, basically cut in half.
00:08:29And for the first time, you see that the machine is actually spending real chunks of the time doing
00:08:34math instead of just waiting around.
00:08:36So the compute really improved here.
00:08:39So why is this one much better?
00:08:41Well, it would be tempting to say, well, obviously because it has an RTX 5090 on it.
00:08:46But that's not really the reason.
00:08:48You see, the reason is that this machine has twice the RAM and it was on a faster internal
00:08:54drive, so a much bigger portion of the model can actually just sit in memory, ready to go.
00:09:00Less running back to the disk, less waiting, and more actual work getting done.
00:09:06So the 5090 was actually off in the corner, barely breaking a sweat at this point.
00:09:10And that's the thing I did not expect going into this experiment.
00:09:14You'd assume that the graphics card is what makes or breaks this, right?
00:09:19Well, it isn't.
00:09:20The single biggest thing holding these models back on normal hardware is actually memory.
00:09:25How much of the model you can keep in RAM at once, so you're not constantly waiting on
00:09:30the drive.
00:09:31Which is honestly kind of a rough conclusion right now, because if you've shopped for RAM
00:09:36lately, you know it has gotten really expensive.
00:09:38So no wonder all of these hyperscalers are buying up all the RAM they can get their hands
00:09:44on.
00:09:44So basically, it's not the flashy graphics card that gets you there, it's the RAM.
00:09:49So can you run a 744 billion parameter frontier model on consumer hardware?
00:09:55Yes, full stop.
00:09:56I did it twice, on a laptop and on my desktop, and both of them gave me real correct answers.
00:10:03A couple of years ago, that would have sounded absurd.
00:10:06But Vincenzo genuinely pulled it off.
00:10:08But the honest truth is that running it and running it well are two very different things.
00:10:14If you look at Vincenzo's own benchmarks on the Calibri site, the setups that actually
00:10:19get usable speeds are still serious machines.
00:10:22The fastest configs are things like a 6 GPU rig or a dual Xenon server with a full terabyte
00:10:29of RAM.
00:10:30And notice that the terabyte of RAM server with no fancy GPU at all beats a single RTX 5090
00:10:37by a mile.
00:10:38So it all comes back to the memory, folks.
00:10:41So the dream of a frontier model humming along on an average laptop isn't quite here yet.
00:10:46But we are very, very close.
00:10:48So there you have it, folks.
00:10:50Those were all of my highlights I got while testing Calibri.
00:10:54But I'm curious to know if you guys have tried it or if you have found other clever ways of
00:10:59running frontier models on consumer hardware.
00:11:02I'm really curious to know your thoughts, so let us know your thoughts in the comments down
00:11:06below.
00:11:06And folks, if you like these types of technical breakdowns, please let me know by smashing
00:11:10that like button underneath the video.
00:11:13And also don't forget to subscribe to our channel.
00:11:15This has been Andris from BetterStack and I will see you in the next videos.
00:11:32you