How They Fixed AI's Biggest Local Hardware Problem (with FreeToken)

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語한국어PortuguêsРусский中文
BBetter Stack
Computing/SoftwareConsumer Electronics

Transcript

00:00:00Did you know you can run Mixture of Experts models three times faster than Ollama on the
00:00:05same GPU setup? There's a tool out there called FreeToken, and it uses some clever tactics to
00:00:11optimize state-of-the-art Mixture of Experts models on consumer hardware. So in today's video,
00:00:16we'll take a look at FreeToken, see how it works, and we'll do a live test demo to see if this claim
00:00:22is actually as good as it sounds. It's going to be a lot of fun, so let's dive into it.
00:00:30So FreeToken is an open-source inference engine, and you can think of it as an alternative to
00:00:37Ollama CPP or Ollama, but it's built specifically for serving Mixture of Experts models on consumer
00:00:43hardware. It's made by a team of researchers from Berkeley and University of Texas, and it's based
00:00:49on their research paper that they recently published. So here's the core idea. Let's take a Mixture of
00:00:54Experts model like DeepSeq v4 Flash, for example, which has a total of 284 billion parameters,
00:01:01and only 13 billion are active per token. Since Mixture of Experts models only activate a few of
00:01:07the total parameters available, that is small enough to hold in your GPU, in theory. But that only makes
00:01:13the compute faster for those parameters. You still have a massive memory problem because you still need
00:01:18to place those 284 billion parameters somewhere. And another problem is that in advance, you don't know
00:01:25which experts you're going to need for the next token. So the full model has to live in regular RAM, and the GPU
00:01:31has to keep fetching whichever experts get picked on the fly over PCIE. Existing tools like Lama CPP
00:01:38handle this by nailing certain layers to the GPU and certain layers to the CPU permanently at load time.
00:01:45The problem is, which experts get used changes every single token. So a fixed preloaded layout like that
00:01:53misses most of the traffic and everything just slows down. But FreeToken's fix is to stop treating this as a
00:01:59placement problem and start treating it as a scheduling problem. So if we think of our GPU's memory as a cache,
00:02:07the way it works with Mixture of Experts is that the most recently used experts usually stay in this cache.
00:02:13The full multi-billion parameter model always lives in your regular system RAM as the source of truth. And the GPU
00:02:21just keeps the most used experts nearby. And it turns out that alone helps a lot because experts tend to be
00:02:28very local. Token after token, the router tends to reuse a lot of the same experts. So most of the requests
00:02:34hit the cache. But there are two areas in the execution where this still creates problems. The first is right when you
00:02:41use a prompt. This is called pre-fill. Since a long prompt tends to touch almost every expert across all those tokens, the sparsity
00:02:49basically disappears and the engine has to stream nearly the entire model in from RAM. But FreeToken hides this by doing double buffering.
00:02:57While the GPU is crunching the current layer, it's already streaming the next layer's experts in the background
00:03:04over PCIe. So compute and transfer happens at the same time instead of one after the other. So you're not just sitting there waiting.
00:03:12And the second problem is during actual generation where the cache inevitably misses sometimes. And that's when the model
00:03:20requests an expert that isn't loaded on the GPU at that moment. And this is where the researchers came up with a clever idea.
00:03:27It's something they call the QSTAR policy. So the way it works is that when there is a cache miss, you've got two options.
00:03:34You either pull the expert over from RAM to the GPU or you just run the calculation directly on the CPU
00:03:41where the data already sits. Neither option is necessarily always better. It depends entirely on your specific hardware's
00:03:49balance between the PCIe speed and RAM speed. So FreeToken actually measures both of those bandwidths on your
00:03:56machine to split each batch of missing experts between sending it to the GPU and just computing it on the CPU.
00:04:03Whichever mix finishes faster on your exact setup. So a laptop with a narrow PCIe link and a desktop with a
00:04:11fast RAM get two completely different splits automatically without you needing to configure that manually.
00:04:18And on top of that,
00:04:18they use a custom weight format, which they call FTW. So the model loads straight off of disk into exactly the
00:04:26memory layout the engine needs, skipping the repackaging step that most tools do on startup.
00:04:32That's why loading a huge model doesn't take forever either. So that's how it works in theory,
00:04:37but I wanted to test it out on my own workstation with an RTX 5090 and 64 gigabytes of DDR5 RAM
00:04:45to see how it performs. So I ran the same task with the same model using both OLAMA and FreeToken
00:04:52to see how these tools differ. And in my case, the whole comparison comes down to one line.
00:04:57Does this model fit into my 32 gigabyte VRAM space or not? So in my particular case, the problem is that I
00:05:04loaded Quen 3.6 35 billion parameter model at 8-bit quantization, which is about 38 gigabytes.
00:05:11And that is six gigabytes over what the card can actually hold. So OLAMA's answer to this problem is to
00:05:18split the model by layer. Roughly 70% of it stays on the GPU and 30% gets pushed onto the CPU, which sounds
00:05:26fine, except every single token has to pass through every layer. So every token takes a detour through
00:05:32the slow CPU. But FreeToken keeps the math entirely on the GPU and it just streams whatever experts are
00:05:39missing over PCIe. So in theory, this means I should get a much faster token speed using FreeToken. So I
00:05:47ran a little task. I asked the model to add some test cases to a sample Python project I have lying
00:05:53around and I hooked up both OLAMA and FreeToken in my open code configuration so I can do the same task
00:05:59on open code and then compare the results. And it took OLAMA 14 minutes and 20 seconds to finish the task.
00:06:06And then I pulled up the statistics from the run and if we look at it, we see that the model using OLAMA
00:06:12was streaming at around 58 tokens a second. When I ran the same task with FreeToken, it finished in
00:06:194 minutes 40 seconds. And indeed we got a much faster speed of 132 tokens per second. And another cool
00:06:27thing is that FreeToken allows you to resize its GPU expert cache live on a running server in about a
00:06:34second without restarting it. So I shrank it down step by step to see how much cash it actually needs. And
00:06:40as we can see here, the dropping from 58% of the model cached down to 40% only cost about 7% of the
00:06:49speed. But if we keep shrinking it past 20% though, then it starts falling off a cliff because the PCIe link
00:06:56gets too oversaturated. So this is a nice visual proof that usually only a small slice of the hot
00:07:03experts are doing almost all of the work. And it's also worth noting that FreeToken is not a silver
00:07:09bullet. It only works for specific cases. So for example, in my case, I also tested a 4-bit version
00:07:15that actually fits inside the 32 gigabytes of VRAM with room to spare. And in this case,
00:07:21OLAMA just loads the whole thing onto the card and that actually achieves a higher speed of 240 tokens
00:07:28per second versus FreeToken's 225. Because in this case, FreeToken's streaming architecture has nothing
00:07:35to stream, so it's just pure engineering overhead. So long story short, if your model comfortably fits on
00:07:42your GPU, OLAMA is most definitely the faster choice. FreeToken's advantage only shows up once
00:07:48you cross your graphics card's VRAM threshold. And it's worth noting that I conducted all my tests in
00:07:54this video using their CLI version, but they also have a very user-friendly desktop app. But it is only
00:08:01available on Windows and Linux at the moment. But it does give you a nice overview of which models your
00:08:07system is able to run. For example, I wanted to run DeepSeq v4 Flash, but it straight out told me that
00:08:14even my workstation is too weak to run such a large frontier model locally. And I tried running the
00:08:19same test with Quen 3.6 on the desktop app, and it couldn't really go into any project folder or modify
00:08:27any of the files. So the app is quite limited at the moment, but it does give you a nice visual overview
00:08:33of what's happening in FreeToken behind the scenes. So there you have it folks, that is FreeToken in a
00:08:38nutshell. I think the biggest advantage of this tool is that it is adaptive. It can automatically profile
00:08:44your machine and adapt a better streaming strategy depending on your hardware setup. But remember,
00:08:50this only works for mixture of experts models, and it is only effective if the experts do not fit
00:08:56comfortably inside your GPU's VRAM. Which for many consumer setups might actually be the case. But what
00:09:02do you think about FreeToken? Have you tried it? Will you use it? Let us know in the comments section down
00:09:07below. And folks, if you like these types of technical breakdowns, please let me know by smashing that
00:09:12like button underneath the video. And also don't forget to subscribe to our channel. This has been Andres from
00:09:18Betterstack and I will see you in the next videos.

Key Takeaway

FreeToken overcomes consumer hardware limitations for large Mixture of Experts models by treating expert placement as a scheduling and caching problem, delivering over 2x faster execution when models exceed available VRAM.

Highlights

  • FreeToken is an open-source inference engine designed for running Mixture of Experts models on consumer hardware by researchers from Berkeley and the University of Texas.

  • A live test with a 38-gigabyte model running on an RTX 5090 completed a Python testing task in 4 minutes and 40 seconds using FreeToken, compared to 14 minutes and 20 seconds using Ollama.

  • FreeToken achieves higher token generation speeds of 132 tokens per second versus Ollama's 58 tokens per second when models exceed available VRAM capacity.

  • The QSTAR policy measures PCIe and RAM speeds to split missing expert calculations between GPU streaming and direct CPU execution dynamically.

  • FreeToken utilizes a custom weight format called FTW that loads straight off disk into the exact memory layout required by the engine without a startup repackaging step.

Timeline

Core limitations of running Mixture of Experts models on consumer hardware

  • Mixture of Experts models feature massive total parameter counts while only activating a small fraction per token.
  • Full multi-billion parameter models must reside in regular system RAM because advanced knowledge of required experts is impossible.
  • Existing tools like Ollama CPP permanently nail specific layers to the GPU or CPU at load time, causing traffic misses and slowdowns.

Models such as DeepSeek v4 Flash contain 284 billion parameters with only 13 billion active per token. Although active parameters fit inside a GPU, the entire model must live in system RAM due to memory constraints. Standard inference engines struggle because active expert routing changes dynamically per token, rendering fixed preloaded layer layouts inefficient.

Caching and scheduling architecture in FreeToken

  • FreeToken treats expert memory management as a dynamic scheduling problem using the GPU memory as a cache.
  • Double buffering streams the next layer's experts over PCIe in the background while the GPU computes the current layer during pre-fill.
  • The QSTAR policy measures hardware-specific PCIe and RAM bandwidth to split missing expert computations between GPU transfer and CPU processing.

Because routers frequently reuse local expert tokens, most requests successfully hit the GPU cache. During long prompt pre-fills where sparsity drops, double buffering prevents waiting by overlapping compute and PCIe data transfers. For cache misses, the QSTAR policy automatically calculates the fastest execution split based on the exact machine's hardware bandwidth profile without manual configuration.

Performance benchmarks and limitations on local workstations

  • A benchmark task using a 38-gigabyte model on an RTX 5090 finished in 4 minutes and 40 seconds with FreeToken versus 14 minutes and 20 seconds with Ollama.
  • FreeToken generates 132 tokens per second compared to Ollama's 58 tokens per second on models exceeding VRAM capacity.
  • Ollama performs faster than FreeToken when models comfortably fit entirely inside the GPU VRAM.

Testing a 35-billion parameter model quantized at 8-bit on a 32-gigabyte VRAM card reveals massive performance gains when models exceed hardware limits. FreeToken dynamically streams missing components over PCIe rather than routing every token through slow CPU split layers like Ollama. However, when models fit entirely within VRAM, streaming architecture introduces unnecessary overhead, making standard tools faster.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video