Written with AI assistance from the source video. The video is the authority.
Log in to leave a comment
No posts yet
When you download the code for an open-source AI workspace and spin it up on Mac Docker, you hit a wall right from the first prompt. Even though you are clearly using an M-series chip, the fans just spin loudly, and you get a dismal speed of barely 10 tokens per second. This is because Docker Desktop's virtualization layer fails to pass the Apple Silicon GPU (Metal) into the container, causing it to fall back to bare CPU computation.
If you run the FastAPI backend and a local vector DB together without resolving this bottleneck, the container will crash due to memory leaks and connection exhaustion. Here are four configurations you need to overhaul to run code—which you might have just grabbed based on GitHub stars—at a production level in a local environment.
Docker Desktop on macOS runs on top of a lightweight Linux VM. There is no feature to pass the host's Metal GPU API directly into the Linux guest OS container. Ollama or llama.cpp inside the container fails to allocate VRAM and quietly switches to CPU execution. This is why a Llama 3 8B model, which achieves 40-80 t/s natively on an M3 Max, plummets to a prompt evaluation of 21.45 t/s and token generation of 12.17 t/s the moment it goes inside Docker.
`
[CPU Fallback Mechanism in Virtualized Container]
+-----------------------------------------------------------------------+
| Docker Container (Linux Guest OS) |
| +---------------------+ |
| | Local LLM Engine | --(VRAM Allocation Req)--> [VirtGPU Missing] |
| +---------------------+ | |
| | v |
| +<--(Fallback to CPU Execution)--- [Silent slog.Debug] |
+-------------|---------------------------------------------------------+
v
[Apple Silicon Host CPU (ARM Neon/DotProd)] -> Speed Degradation Occurs
`
The default shared memory (shm) configuration is also a problem. The default allocation of 64MB will immediately throw a Bus Error during large-scale tensor operations. Open ~/.docker/daemon.json to expand the shared memory and lift the resource limits.
`json
{
"builder": {
"gc": {
"defaultKeepStorage": "20GB",
"enabled": true
}
},
"experimental": false,
"default-shm-size": "8g"
}
`
`yaml
version: '3.8'
services:
llm-inference-engine:
image: ollama/ollama:latest
container_name: local_llm_engine
shm_size: '16gb'
ipc: host
deploy:
resources:
limits:
cpus: '8.0'
memory: 24G
reservations:
cpus: '4.0'
memory: 12G
ports:
- "11434:11434"
volumes:
- ollama_storage:/root/.ollama
volumes:
ollama_storage:
`
To extract GPU acceleration inside the container, you must switch to Podman using the libkrun virtual machine monitor and the krunkit driver. This method (Virtio-GPU Venus) passes Vulkan compute requests from inside the container to the host macOS's Metal API. It boosts processing speed up to around 75% of native Metal performance.
brew tap slp/krunkit && brew install krunkit podmanexport CONTAINERS_MACHINE_PROVIDER="libkrun" && podman machine init --cpus 8 --memory 32768 && podman machine startpodman machine ssh "ls -la /dev/dri"`dockerfile
FROM fedora:40
RUN dnf -y install dnf-plugins-core &&
dnf -y copr enable slp/mesa-krunkit fedora-40-aarch64 &&
dnf -y install mesa-vulkan-drivers vulkan-loader vulkan-tools &&
dnf -y downgrade mesa-vulkan-drivers.aarch64 --repo copr:copr.fedorainfracloud.org:slp:mesa-krunkit &&
dnf clean all
ENV GGML_VULKAN=1
`
If infrastructure policies dictate that you must use CPU execution only, use the Q4_0_4_4 quantization format tailored for ARMv8.4-A DotProduct and Neon vector instructions. This defends the prompt evaluation speed at up to 50.63 t/s, reducing the processing time by more than half compared to default Docker CPU execution.
| Runtime Configuration | Prompt Eval (t/s) | Token Gen (t/s) | Setup Difficulty | Features |
|---|---|---|---|---|
| Docker Desktop (Default CPU) | ~21.45 | ~12.17 | Low | CPU fallback due to virtualization limits |
| Docker Container (ARM Q4_0_4_4) | ~50.63 | ~14.01 | Medium | Utilizes ARM Neon vector instructions |
| Podman + libkrun (Vulkan Venus) | ~75% of Native | ~75% of Native | High | Virtio-GPU acceleration inside container |
| Host Native Metal + Container | 100% Native (40~80) | 100% Native | Medium | Direct call structure to host engine |
Open-source backends often have sloppy asynchronous task state management or CPython garbage collector handling. As requests accumulate, memory inflates until the process crashes. Use tracemalloc to pinpoint the exact locations devouring memory.
`python
import gc
import tracemalloc
tracemalloc.start()
def log_memory_snapshot():
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[Memory Debug] Top 5 Allocations:")
for stat in top_stats[:5]:
print(stat)
gc.collect()
`
You need to force memory clearance at the process management stage. Configure the lifecycle so that every time a Gunicorn worker handles 1,000 requests, it automatically releases memory and restarts.
`bash
gunicorn
--workers 4
--worker-class uvicorn.workers.UvicornWorker
--bind 0.0.0.0:8000
--max-requests 1000
--max-requests-jitter 100
--timeout 120
--keep-alive 5
--preload-app
app.main:app
`
Append --max-requests-jitter 100 to prevent all 4 workers from dying and restarting simultaneously, which causes dropped requests, and cut off unresponsive tasks with a 120-second timeout.
If an asynchronous generator continues looping when a user closes their browser mid-response, compute resources are wasted entirely. Check request.is_disconnected() to immediately break out of the loop if the connection drops.
`python
import asyncio
import gc
import logging
from typing import AsyncGenerator
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import StreamingResponse
app = FastAPI(title="Production AI Workspace Backend")
logger = logging.getLogger("stream_logger")
async def robust_llm_token_stream(prompt: str, request: Request) -> AsyncGenerator[str, None]:
try:
for token_idx in range(2000):
if await request.is_disconnected():
logger.warning(f"[Stream Aborted] Client disconnected at step {token_idx}.")
break
await asyncio.sleep(0.01)
yield f"event: message\ndata: {{\"id\": {token_idx}, \"text\": \"chunk_{token_idx} \"}}\n\n"
except asyncio.CancelledError:
logger.info("[Stream Cancelled] Task cancelled by ASGI server.")
raise
except Exception as err:
logger.error(f"[Stream Error] Error during streaming: {str(err)}")
raise
finally:
logger.info("[Stream Cleanup] Releasing context and triggering GC.")
gc.collect()
@app.post("/api/v1/chat/stream")
async def chat_stream_endpoint(request: Request, body: dict):
prompt = body.get("prompt", "")
if not prompt:
raise HTTPException(status_code=400, detail="Prompt string is missing.")
return StreamingResponse(
robust_llm_token_stream(prompt, request),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no"
}
)
`
It is common for a single minor update from upstream to break a local container build. Use a requirements.txt embedded with SHA-256 hashes to prevent package tampering and version conflicts.
`plaintext
fastapi==0.110.0 --hash=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
uvicorn==0.28.0 --hash=sha256:2c6a81387e0037a34ca6a7f8087796030c79eb299f116515822ee483c6604e43
pydantic==2.6.4 --hash=sha256:d82e212f451f2d6c19f5a5e3a89369322e70e1781297587786411516279f64a5
sqlalchemy==2.0.28 --hash=sha256:a611116c2bb45f8f3077e6822ec3a37b384ff6b9a84d4dd88a0e8eb876b5cf12
`
Pre-download Wheel binaries into a local directory so that pip downloads don't fail when rebuilding Docker in offline environments, such as internal corporate security networks or airplanes.
`bash
pip wheel --wheel-dir=./wheels_repository -r requirements.txt
`
`dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY ./wheels_repository /app/wheels_repository
COPY requirements.txt .
RUN pip install --no-cache-dir --no-index --find-links=/app/wheels_repository -r requirements.txt
COPY . .
CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", "-c", "gunicorn.conf.py", "app.main:app"]
`
Pulling upstream Git repository changes in their entirety often wipes out painstakingly tuned local configurations. Branch off an optimization branch and cherry-pick only the necessary commits.
`bash
git remote add upstream https://github.com/opensource-ai-workspace/workspace.git
git fetch upstream
git checkout -b feature/local-mac-optimization
git cherry-pick
`
If multi-agents hammer the database simultaneously with embedding searches, memory lookups, and conversation logging, the PostgreSQL connection pool will dry up instantly. Based on an 8-core Apple Silicon single-disk environment, limit the engine pool to under 17, according to the connection calculation formula ().
`python
from sqlalchemy.ext.asyncio import create_async_engine
DATABASE_URL = "postgresql+asyncpg://postgres:password@localhost:6432/ai_workspace"
engine = create_async_engine(
DATABASE_URL,
pool_size=15,
max_overflow=5,
pool_timeout=10,
pool_recycle=300,
pool_pre_ping=True
)
`
Use PgBouncer transaction pooling to cut off the phenomenon of agents stubbornly holding onto DB sessions while waiting for LLM inference results.
`ini
[databases]
ai_workspace = host=postgres_db port=5432 dbname=ai_workspace auth_user=postgres
[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = plain
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
idle_transaction_timeout = 60
max_client_conn = 1000
default_pool_size = 20
min_pool_size = 5
reserve_pool_size = 5
reserve_pool_timeout = 3
`
`
+-------------------------------------------------------------------------------+
| Docker Internal Isolated Network (internal: true) |
| |
| +--------------------+ +--------------------+ +-----------------+ |
| | AI Workspace App | ---> | PgBouncer Proxy | ---> | PostgreSQL DB | |
| | (FastAPI Backend) | | (Port 6432) | | (Port 5432) | |
| +--------------------+ +--------------------+ +-----------------+ |
| | | |
| | [pool_mode = transaction] |
| | [idle_tx_timeout = 60s] |
| v |
| +--------------------+ |
| | Local Vector DB | (Block External Internet) |
| | (Qdrant) | |
| +--------------------+ |
+-------------------------------------------------------------------------------+
`
To eliminate the risk of data leaks, apply internal: true to the container network to block external cloud communication. Connect to the high-speed Metal Ollama runtime running directly on the host machine only through the host.docker.internal gateway.
`yaml
version: '3.8'
services:
backend-app:
build: .
environment:
- DB_HOST=pgbouncer
- DB_PORT=6432
- VECTOR_DB_HOST=vector-db
- OLLAMA_HOST=http://host.docker.internal:11434
networks:
- isolated_local_net
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "8000:8000"
pgbouncer:
image: edoburu/pgbouncer:latest
environment:
- DB_HOST=postgres_db
- DB_PORT=5432
- DB_USER=postgres
- DB_PASSWORD=secret
- POOL_MODE=transaction
networks:
- isolated_local_net
depends_on:
- postgres_db
postgres_db:
image: postgres:16-alpine
environment:
- POSTGRES_DB=ai_workspace
- POSTGRES_PASSWORD=secret
volumes:
- pgdata:/var/lib/postgresql/data
networks:
- isolated_local_net
vector-db:
image: qdrant/qdrant:v1.9.0
volumes:
- qdrant_data:/qdrant/storage
networks:
- isolated_local_net
networks:
isolated_local_net:
driver: bridge
internal: true
volumes:
pgdata:
qdrant_data:
`
By bypassing the GPU virtualization bottleneck via Podman or host bridging, and controlling the lifecycle through worker recycling and DB proxies, you complete an independent development environment that won't crash even on laptops equipped with M-series chips.