Written with AI assistance from the source video. The video is the authority.
Log in to leave a comment
No posts yet
When running a 744B MoE model like GLM-5.2 locally on standard workstations or MacBooks, the bottleneck hits storage bandwidth and OS virtual memory policies long before VRAM capacity. JustVugg's colibrì, a lightweight C-based inference engine, keeps only a 17B resident layer (9.9 GiB in int4) locked in RAM and streams the ~370 GB of routed expert weights—split into 21,504 chunks—in real time from an external SSD on a per-token basis.
The problem is that running it with default OS settings causes swap thrashing or prompts the kernel's OOM Killer to terminate the process. To keep token generation running smoothly without interruptions, you must manually fine-tune everything from compiler flags to NVMe I/O queues and kernel parameters.
colibrì's core compute engine is a single C11 file of about 1,300 lines with zero external dependencies. Matrix multiplication throughput depends heavily on which SIMD vector instructions the compiler utilizes. Instead of the default -O2 build, you should explicitly specify the target CPU's integer arithmetic extension instructions.
For x86_64 Linux environments, enable AVX-512 VNNI with GCC 12 or newer; for Apple Silicon, link NEON DotProduct via Clang.
On Ubuntu and Debian-based systems, install the build tools first:
`bash
sudo apt-get install gcc-12 libomp-dev
`
On macOS, install the OpenMP runtime via Homebrew:
`bash
brew install libomp
`
On x86_64 Linux machines, build targeting AVX-512 VNNI instructions:
`bash
gcc -O3 -march=native -mtune=native
-mavx512f -mavx512bw -mavx512vnni -mavx512dq
-fopenmp -funroll-loops -ffast-math
-o colibri_engine colibri_glm52.c -lm
`
On Apple Silicon (M-series) systems, compile by explicitly specifying the library paths:
`bash
clang -O3 -mcpu=native
-march=armv8.4-a+dotprod+fp16
-Xpreprocessor -fopenmp
-I/opt/homebrew/opt/libomp/include
-L/opt/homebrew/opt/libomp/lib -lomp
-o colibri_engine colibri_glm52.c -lm
`
If you encounter struct-related build warnings, add the -std=c11 flag to resolve them.
| Platform | Compiler | Vector Instruction Flags | Thread Parallelism |
|---|---|---|---|
| x86_64 (Linux) | GCC 12+ | -mavx512vnni -mavx512bw |
-fopenmp |
| ARM64 (macOS) | Apple Clang | -march=armv8.4-a+dotprod |
-Xpreprocessor -fopenmp |
| Legacy x86 | GCC / Clang | -mavx2 -mfma |
-fopenmp |
colibrì's decoding latency is determined by the disk's 1MB random read speed. GLM-5.2's routed expert parameter files are partitioned into ~19 MB chunks each, and with every generated token, an asynchronous I/O thread reads the necessary chunks from the SSD.
Install fio using your package manager:
`bash
sudo apt-get install fio
brew install fio
`
Measure random read bandwidth using a 1MB block size with Direct I/O enabled:
`bash
fio --name=colibri_stream_bench
--filename=/Volumes/ExternalSSD/glm52_test.tmp
--size=10G
--rw=randread
--bs=1m
--iodepth=32
--numjobs=4
--ioengine=posixaio
--direct=1
--group_reporting
--runtime=60
`
Thunderbolt 4 or USB4 (40 Gbps) interfaces maintain actual bandwidths of 2,800–3,200 MB/s, delivering a decoding speed of 0.08–0.10 tok/s. In contrast, connecting an external drive via a USB 3.2 Gen2 (10 Gbps) port—which tops out at 900–1,050 MB/s—will take over 80 minutes to generate just 100 tokens.
`
+-------------------------------------------------------------------------+
| colibrì C Inference Engine |
| +--------------------------+ +-------------------------------------+ |
| | Dense Weights (9.9 GiB) | | Multi-Token Prediction (MTP) Head | |
| | Resident in RAM (mlock) | | int8 Quantized | |
| +------------+-------------+ +------------------+------------------+ |
+---------------+-----------------------------------+---------------------+
| |
v v
+-------------------------------------------------------------------------+
| Memory & Storage I/O Layer |
| +--------------------------+ +-------------------------------------+ |
| | Async Expert Readahead | | 21,504 Routed Experts (370 GB) | |
| | I/O Queue Depth 3264 | | Streamed from NVMe SSD via mmap | |
| +--------------------------+ +-------------------------------------+ |
+-------------------------------------------------------------------------+
`
Increasing the I/O queue depth (iodepth) for Async Expert Readahead to 32–64 fully utilizes the NVMe controller's parallel channels, reducing disk wait latency.
On a 32 GB RAM system, loading 9.9 GiB of resident weights alongside the KV Cache and filesystem page cache simultaneously triggers kernel memory contention. Under default Linux settings, attempting a 370 GB mmap call either blocks virtual address space allocation or triggers the OOM Killer to terminate the engine.
Create /etc/sysctl.d/99-colibri-memory.conf and add the following configuration:
`ini
vm.swappiness = 1
vm.overcommit_memory = 1
vm.min_free_kbytes = 2097152
vm.vfs_cache_pressure = 50
vm.max_map_count = 524288
`
Apply the parameters immediately after saving:
`bash
sudo sysctl --system
`
Lift the memory lock limit in your shell session to allow mlock() calls for the resident layers:
`bash
ulimit -l unlimited
`
| Parameter | Default | Recommended | Purpose |
|---|---|---|---|
| vm.swappiness | 60 | 1 | Prevents the 9.9 GiB of locked weights in RAM from being swapped out |
| vm.overcommit_memory | 0 | 1 | Prevents kernel memory allocation denial during 370 GB mmap calls |
| vm.vfs_cache_pressure | 100 | 50 | Extends file cache lifetime to reuse frequently invoked expert weights |
| vm.max_map_count | 65530 | 524288 | Removes mapping limits across the 21,504 parameter files |
| ulimit -l | 64 (KB) | unlimited | Grants permission to lock physical memory (mlock) for Dense layers |
On macOS, when free memory is exhausted, dynamic_pager applies memory compression in the background, consuming CPU resources. Close high-usage applications before running to prevent the compression engine from kicking in.
MoE decoding relentlessly hammers the CPU cache and memory bus. On multi-socket workstations, running on the wrong NUMA node creates an interconnect bus bottleneck that cuts token generation speed in half.
First, verify the physical CPU node number directly connected to the external NVMe controller using numactl --hardware. Next, write a shell script that pins threads strictly to physical cores to avoid hyper-threading conflicts.
`bash
#!/usr/bin/env bash
set -euo pipefail
sudo sysctl -w vm.swappiness=1 > /dev/null
sudo sysctl -w vm.overcommit_memory=1 > /dev/null
sudo sysctl -w vm.max_map_count=524288 > /dev/null
sudo sysctl -w vm.vfs_cache_pressure=50 > /dev/null
sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
ulimit -l unlimited
export OMP_NUM_THREADS=16
export OMP_PROC_BIND=TRUE
export OMP_PLACES=cores
ENGINE_BIN="./colibri_engine"
MODEL_PATH="/mnt/nvme_ext/GLM-5.2-colibri-int4-g64-with-int8-mtp"
exec numactl --physcpubind=0-15 --membind=0
"${ENGINE_BIN}"
--model "${MODEL_PATH}"
--threads "${OMP_NUM_THREADS}"
`
On Apple Silicon MacBooks, you must prevent macOS's QoS daemon from offloading worker threads to Efficiency Cores (E-Cores). Elevate the process priority explicitly when executing:
`bash
sudo nice -n -20 taskpolicy -c default ./colibri_engine --model /Volumes/SSD/glm52_i4
`
Once these four configurations are in place, you can reliably run a 744B MoE model via local disk streaming on a 32GB RAM workstation or MacBook without crashing into OOM errors.
