This Tiny Tool Fits a 744B AI Model on Normal Hardware (colibrì)

Highlights

• Calibri is a pure C tool created by Vincenzo Fornaro that enables local execution of the 744 billion parameter GLM 5.2 model.

• Out of 744 billion total parameters in the GLM 5.2 model, only 40 billion parameters are active per token.

• The dense components of the model stay resident in system RAM while individual expert parameters stream off the SSD and cache dynamically.

• Running the model on an M2 Max Mac with an external USB SSD resulted in a speed of 0.1 tokens per second with a 158-second total turn time.

• Testing on a workstation with an RTX 5090, 64 gigabytes of RAM, and a fast internal SSD produced a speed of 0.8 tokens per second.

• System memory size and disk transfer speed are the primary hardware bottlenecks for running large language models locally rather than GPU computing power.

Timeline

1. 00:00 - 05:05

   Local Execution of Frontier Models

   Running large language models locally normally requires hundreds of gigabytes of specialized memory. GLM 5.2 bypasses this constraint through a mixture of experts design where thousands of specialist subnetworks handle individual tokens. Calibri exploits this sparsity by keeping dense components under 10 gigabytes in RAM and streaming required experts from the disk on demand. Written entirely in pure C with zero external dependencies, the engine includes a learning cache that pins frequently used experts to spare RAM and uses a compressed attention cache to shrink the KV cache by 57 times.

   • The GLM 5.2 model contains 744 billion parameters and matches the size class of data center frontier models.
   • A developer named Vincenzo Fornaro created Calibri to run this large model on consumer hardware.
   • Mixture of experts architecture activates only 5% of the total parameters, or 40 billion, for any given token.
   • Calibri treats device storage, system RAM, and VRAM as a memory hierarchy to stream and cache model components.
2. 05:05 - 07:26

   MacBook Performance and Bottlenecks

   Evaluating the tool on a consumer laptop required an external drive because the 357-gigabyte model exceeded internal storage limits. The slow USB transfer speed constrained the data pipeline, forcing the CPU to spend over 80 percent of the processing time waiting for model pieces to load from disk. Because the 32 gigabytes of system RAM could not hold the necessary model layers, continuous disk access created severe latency before token generation began.

   • Testing on an M2 Max MacBook with 32 gigabytes of RAM required an external Samsung portable SSD with a 1 gigabyte per second transfer speed.
   • The system successfully answered a query about the capital of Belgium using the 744 billion parameter model.
   • Generation speed reached approximately one word every 10 seconds, or 0.1 tokens per second, during a cold start.
   • Profiling data showed that 145 out of 158 total seconds were spent waiting on disk input and output operations.
3. 07:26 - 11:32

   Workstation Testing and Memory Requirements

   Moving the model to a machine with double the system RAM and an internal NVMe drive significantly improved execution speed. The faster drive and larger memory footprint allowed a greater portion of the model to remain resident in memory, cutting disk wait times in half. Ultimately, hardware constraints confirm that system memory capacity is the most critical factor for local execution of frontier-scale language models.

   • A workstation featuring an RTX 5090, 64 gigabytes of RAM, and a fast internal drive generated responses at 0.8 tokens per second.
   • Time to first token dropped to 17 seconds, and disk waiting time decreased to 48 percent of the total duration.
   • System RAM and internal disk speed dictate performance more directly than graphics card capabilities.
   • Optimal performance configurations require server hardware with high memory capacities, such as systems equipped with one terabyte of RAM.