Transcript
00:00:00This project uses Apple Silicon's architecture to run a 26 billion per hour model on just two
00:00:06gigabytes of RAM and it's actually usable. I tried it out locally and was able to get 23 tokens per
00:00:12second. The project created a couple of weeks ago Turbo Fieldfare avoids the large memory footprint
00:00:17of Gemma 4 which is around 14 gig by keeping only a tiny fraction of the model in memory
00:00:23then streaming the experts direct from SSD. The entire thing is written in Swift and Metal which
00:00:29is Apple's low-level GPU API so you can launch it as a Mac app which is incredibly user-friendly
00:00:34and it takes advantage of the specific architecture that you get from Apple Silicon chips. So today I
00:00:41want to dive into how Turbo Fieldfare uses Apple Silicon specifically to squeeze incredible
00:00:46performance out of a 26 billion per hour model. So if we head over to the GitHub repo we can run a
00:00:57series of commands to clone the repo and get the app up and running. And once up and running we need to
00:01:01download the model itself, load the model then we can send our first message. You can see for me on an
00:01:06M3 Max I'm getting 23.4 tokens per second with a memory footprint of just 2.15 gigabytes. This feels
00:01:15completely usable to me. So let's take a deeper look at the architecture and if you enjoy content like
00:01:20this then don't forget to subscribe to BetterStack. First we need to understand how Gemma 4 works because
00:01:25that leads into the system design of the application itself. Gemma 4 is a mixture of experts model. Most
00:01:32models use one big feed forward neural network but Gemma is different. As an MOE layer it instead has
00:01:39128 small feed forward blocks, these are the experts, plus a tiny router. For each token the router picks the
00:01:46top 8 experts and only those 8 run. So 26 billion parameters total but only roughly 3.9 billion actually do
00:01:54any work on any given token. 85% of the file is idle at any instance. You don't need 14.3 gigabytes in RAM,
00:02:02you need whatever the current token happens to touch. Turbo Fieldfare takes advantage of this design,
00:02:08keeping the always needed parts in RAM, leaving the experts on SSD and fetching them just in time. So
00:02:15let's take a look at what Turbo Fieldfare is doing with Apple Silicon. When you install the model it gets
00:02:20split into two separate piles. The first is everything the token needs no matter what, the attention,
00:02:25the router, the embeddings and one shared expert that always runs. That comes to about 1.35 gig,
00:02:32it gets memory mapped straight off disk and it's always resident on memory the whole time the model
00:02:37is loaded. The second pile is the experts themselves, 30 layers with 128 experts each, about 3.36 megabytes
00:02:46a piece, so roughly 12.9 gigabytes and that never gets loaded at all. It just sits on the SSD and gets pulled
00:02:53in a few megabytes at a time as and when the model asks for it. So what actually happens when you produce
00:02:59a token? The model has 30 layers and the token passes through every one of them in order and each layer
00:03:05does the same two things. First up is attention and that's the step where the model looks back over
00:03:11everything that's been written so far and works out what matters right now. If it's about to write the
00:03:16word after the cat sat on the, attention is what makes cat count more than the and the handy thing is
00:03:23that attention runs entirely on that 1.35 gig that's already in memory so we haven't touched the disk
00:03:29yet at all. Then the router takes what attention produced and names the eight experts it wants out
00:03:35of the 128 and that's where a problem lies because you can't know which experts you need until you've
00:03:41already done half the work on that layer. There's no reading ahead and no pre-fetching since the choice
00:03:46depends on this token and every token before it and it only gets made a fraction of a millisecond before
00:03:52those weights are needed. So 30 times per token the CPU has to stop and go to disk which brings us to
00:03:59why this specifically is a mac project. On a pc with a discrete gpu getting a white in front of the gpu
00:04:06means reading it off the ssd into system ram then pushing it across the pci buzz into the card's own
00:04:12vram that's two copies and a buzz hop and doing that thousands of times a second would tank performance.
00:04:19Apple silicon however has unified memory so the cpu and the gpu are looking at the same physical ram and
00:04:25there's no vram to copy into. A metal buffer is just memory that both the cpu and the gpu can see so the
00:04:32cpu reads bytes straight off the ssd into a buffer the gpu is about to run against meaning we can skip
00:04:39a bunch of work and they lean on these optimizations harder with the file format. Normally whites on disk
00:04:45sit in a storage format that needs to be unpacked and converted before a gpu can use them but turbo
00:04:51fieldfet stores them in exactly the layout the metal kernel consumes right down to the 4-bit quantized
00:04:57values. The installer rearranges the data without ever re-encoding it so reading the file is loading
00:05:03the weight with no conversion step in between. So what's the gpu doing while the cpu is off fetching
00:05:09from disk? Well remember the shared expert sitting on the resident pile that runs for every token no
00:05:13matter what the router picked? That's what it works on. The disk read hides inside work the model had to do
00:05:19anyway so you get most of the fetch for free. So we've got multiple work streams happening async which
00:05:25squeezes the time down even further and even more so it's not even going to disk every time either.
00:05:30Each layer keeps 16 of its 128 experts parked in memory so when the router picks one that's already
00:05:37there you get it instantly and when it doesn't the expert comes off the ssd and pushes out whichever one
00:05:43has been used least. That's called an LFU cache which is least frequently used. This is different to an
00:05:49LRU cache where you throw out whatever was touched longest ago because routing here isn't random. Some
00:05:55experts get picked constantly across all sorts of tokens and others hardly come up at all so counting how
00:06:01often an expert gets used keeps the popular ones around better than counting how recently. It does mean the
00:06:07whole design is betting on routing being predictable because if every token wanted a different random
00:06:138 you'd miss the cache nearly every time and lose performance. So the system design here is very
00:06:18specific to take advantage of both the architecture of Gemma 4 and Apple silicon. You can try this out
00:06:24yourself and head over to the repo we've included in the description and i'm excited to see how far we can
00:06:29squeeze these models you know maybe one day we'll have a fable level model running on our watch. We've also filmed a
00:06:35video showing how you can run this type of architecture on even smaller machines you can watch that here
00:06:40otherwise thank you so much for watching guys i hope you enjoyed that one and i'll see you in the next one
Community Posts
No posts yet. Be the first to write about this video!
Write about this video