Written with AI assistance from the source video. The video is the authority.
Log in to leave a comment
No posts yet
You've probably had the experience of watching a YouTube video, pasting a model installation command into your terminal, and watching your screen freeze up. The reason local LLMs crash on older MacBooks with 8GB or 16GB of RAM, or on budget laptops, isn't a lack of compute cores. The real culprit is the runtime trying to read data that exceeds its limits when the memory bandwidth is hopelessly narrow. By measuring your device's bandwidth and tuning just the context size, you can run and use code-assistant models even on older machines.
Modern macOS and Ubuntu prevent you from carelessly installing packages into the system Python. The moment you run pip install, it stops and spits out a PEP 668 error (error: externally-managed-environment). If you get annoyed and force-bypass system protection with a flag (--break-system-packages), your OS default tools will eventually get tangled up and force you to wipe the system. Use pipx, which supports independent virtual environment isolation, to install the diagnostic tool.
Deploy the open-source diagnostic tool llmfit, developed by Alex Jones, in isolation to extract your hardware specs first.
`bash
brew install pipx
pipx ensurepath
pipx install llmfit
mkdir -p ~/local-ai-workspace/{configs,logs,scripts}
llmfit --json system > ~/local-ai-workspace/logs/system_specs.json
`
Once you finish this process in the terminal, your device's RAM capacity and memory bus information are organized into the system_specs.json file in just one minute, without touching the system libraries.
The process of a local language model spitting out text is a sequential task where it looks at previously generated tokens and writes the next character. At every step, billions of model weights must be read entirely from the memory bus. In other words, perceived speed is determined by the memory bandwidth number, not the GPU clock.
The tokens-per-second (TPS) output speed is calculated using the following formula:
TPS approx rac{ ext{Memory Bandwidth (GB/s)}}{ ext{Model Weights Size (GB)} + ext{KV Cache per Step (GB)}} imes etaThe in the formula is the runtime's effective memory bandwidth utilization (MBU) value, which is usually around 0.65.
If you load a 4-bit quantized 7B model (approx. 4.5GB) entirely onto regular DDR4 desktop RAM with a bandwidth of around 45GB/s, the computation speed stays at . Text stutters character by character, which is deeply frustrating for practical assistance work. On the other hand, if you load it onto an RTX 4060 8GB model with 288GB/s bandwidth or an M-series unified memory with over 100GB/s bandwidth, it yields 35 to 40 tokens per second, easily outpacing real-time typing speed.
After checking your device specs, narrow down the models you'll use to just two:
Qwen2.5-Coder-7B-Instruct (Q4_K_M) with a size of 4.7GB. It hits 35+ TPS on a 16GB unified memory MacBook or an 8GB VRAM dedicated GPU.Llama-3.2-3B-Instruct (Q4_K_M) with a size of 2.0GB. It consumes less RAM even in a low-power laptop's DDR4 environment and consistently outputs around 15 TPS.Even after successfully launching a model, the process quietly shuts down after a few back-and-forth questions. Exit Code 137 printed in the terminal means that the operating system's memory management kernel (Linux OOM-Killer or macOS JetSam) forcibly terminated the process (SIGKILL) due to a lack of RAM.
When using an external GPU, an even more troublesome Silent CPU Fallback occurs. When VRAM limits are exceeded, instead of killing the process, some compute layers are offloaded to the sluggish system RAM. During this, GPU utilization plummets and crawls at around 1 token per second.
The culprit is the KV (Key-Value) cache, which consumes RAM as the conversation gets longer. If you leave the context open up to 32,768 (32K) tokens based on the Llama-3 architecture, an additional 4.0GB is attached purely as cache memory on top of the 4.5GB weights. This structure is guaranteed to crash on 8GB devices. Binding this length to 4,096 (4K) tokens reduces the cache capacity to 512MB, preventing crashes even in an 8GB environment.
Here is the procedure to check the current state and lock in the limit.
First, enter ollama ps in the terminal. If the PROCESSOR entry is split like 30%/70% CPU/GPU instead of 100% GPU, VRAM has already overflowed and been offloaded to slow RAM.
Create a configuration file to pin the context size. Open ~/local-ai-workspace/configs/Modelfile.coder and write the following contents:
`dockerfile
FROM qwen2.5-coder:7b
PARAMETER num_ctx 4096
PARAMETER temperature 0.2
`
Build as a custom model in the terminal:
`bash
ollama create custom-coder:7b -f ~/local-ai-workspace/configs/Modelfile.coder
`
After the build finishes, type sudo purge in the terminal on macOS to clear the disk cache, and clean up unused WSL instances on Windows with wsl --shutdown to secure at least 2GB or more of basic available RAM.
To prevent internal source code or personal projects from leaking outside, bind the model serving endpoint exclusively to the local loopback (127.0.0.1).
Create a ~/local-ai-workspace/scripts/serve_secure.sh file and add the following code:
`bash
#!/bin/bash
export OLLAMA_HOST="127.0.0.1:11434"
export OLLAMA_ORIGINS="http://127.0.0.1:*,http://localhost:*"
ollama serve > ~/local-ai-workspace/logs/ollama_runtime.log 2>&1 &
`
After running the script, enter lsof -i :11434 | grep LISTEN in the terminal to verify that the listening address shows up as 127.0.0.1:11434. If 0.0.0.0:11434, which can be accessed from the outside, is visible, you must close the process immediately.
For integration, use the VS Code plugin Continue.dev. Open the configuration file (~/.continue/config.json) to separate the chat model and the autocomplete model.
`json
{
"models": [
{
"title": "Local Qwen2.5-Coder (Chat)",
"provider": "ollama",
"model": "custom-coder:7b",
"apiBase": "http://127.0.0.1:11434"
},
{
"title": "Local Llama3.2 (Summary)",
"provider": "ollama",
"model": "llama3.2:3b",
"apiBase": "http://127.0.0.1:11434"
}
],
"tabAutocompleteModel": {
"title": "Local Autocomplete",
"provider": "ollama",
"model": "qwen2.5-coder:1.5b",
"apiBase": "http://127.0.0.1:11434"
},
"allowAnonymousTelemetry": false
}
`
Assign the 7B model with its context bound to question-answering, and assign the 1GB ultra-lightweight model qwen2.5-coder:1.5b to tab autocomplete. Autocomplete delay disappears.
Verification is carried out with the internet disconnected. Turn off Wi-Fi to go offline, open a terminal, and fire a direct query:
`bash
curl -s -X POST http://127.0.0.1:11434/api/generate -d '{
"model": "llama3.2:3b",
"prompt": "로컬 격리 네트워크 테스트",
"stream": false
}' | grep "response"
`
If a normal JSON response returns while the connection is blocked, you've finished setting up a development environment that runs safely within your laptop's resources without any external cloud dependency.