This New Engine Runs Local AI Using 10x Less RAM! (Cactus)

Highlights

• Cactus bypasses the GPU to run AI models directly on the neural processing unit (NPU) of Apple, Qualcomm, and MediaTek chips.

• The engine uses zero-copy memory mapping to stream model weights directly from storage, eliminating the need to load full models into RAM.

• Cactus utilizes a proprietary .CACT file format to replace standard GGUF files for efficient edge device memory mapping.

• A confidence-based hybrid router automatically switches between a local NPU model and a cloud-based Gemini 2.5 Flash model depending on task complexity.

• Real-time local transcription using the Parakeet speech model achieves an average latency of 260 milliseconds on an iPhone 12 Pro.

Timeline

1. 00:00 - 01:30

   Memory Bottlenecks and Zero-Copy Mapping on Edge Devices

   Standard mobile execution environments regularly shut down local AI applications due to heavy memory demands and battery drain. Cactus addresses this limitation by transitioning away from the traditional GGUF format to a proprietary .CACT format. This file structure allows the system to map model weights directly from storage instead of loading the entire model into the device RAM.

   • Mobile operating systems aggressively terminate applications that cause sudden spikes in RAM usage.
   • The primary bottleneck for local AI deployment on mobile hardware is memory overhead rather than compute power.
   • Zero-copy memory mapping streams specific tensors into the active compute cycle from storage only when needed.
2. 01:30 - 02:11

   NPU Optimization and Direct Silicon Communication

   Most local inference engines default to utilizing the GPU for processing. Cactus targets the dedicated neural silicon found on modern hardware architectures from Apple, Qualcomm, and MediaTek. Bypassing standard translation layers removes the software overhead that typically slows down edge inference, allowing specialized models to maximize the hardware efficiency of matrix multiplication units.

   • Cactus prioritizes the neural processing unit over the graphics processing unit for local inference.
   • Direct communication with dedicated silicon bypasses traditional translation layers to accelerate performance.
   • The Cactus dashboard provides specific models pre-optimized for mobile matrix multiplication units.
3. 02:11 - 03:07

   Hybrid Routing and Cloud Failover Mechanics

   Simple queries remain on the local NPU to maintain user privacy, eliminate hosting costs, and ensure low latency. When the engine detects a task that exceeds local capabilities, it seamlessly hands the request off to a frontier model in the cloud. This dual-layer approach maintains the user experience during complex operations while keeping overall production costs low.

   • Local edge models encounter a definitive reasoning ceiling when handling complex tasks or massive context windows.
   • A confidence-based routing system automatically shifts execution between local hardware and cloud servers.
   • The background failover mechanism manages model switching without requiring changes to the underlying application code.
4. 03:07 - 05:13

   Local and Cloud Latency Benchmarks on Mobile Hardware

   Testing the Swift Cactus package via a local application reveals a distinct speed advantage for on-device processing compared to cloud alternatives. While the local Parakeet model streams data with minimal delay on older smartphone hardware, the cloud-routed Gemini alternative requires a full data round trip. This makes the cloud backup ideal for heavy image analysis or massive reasoning tasks rather than real-time processing.

   • Local real-time transcription averages 260 milliseconds of latency using the Parakeet speech model on an iPhone 12 Pro.
   • Cloud-based transcription via Gemini 2.5 Flash averages 2000 milliseconds for a three-second batch process.
   • The network round trip to data servers accounts for the higher latency observed during cloud execution.