I Ran a 35B Param Model On My iPhone (Local AI)

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語한국어PortuguêsРусский中文
BBetter Stack
Computing/SoftwareConsumer ElectronicsCell Phones

Transcript

00:00:00this 35 billion per ram model is running directly on my iphone i'm able to generate 11 tokens per
00:00:06second through some clever engineering this model was never designed for and in this video i'll
00:00:11show you exactly how to do the same what i'm running here is a mixture of experts model it's
00:00:2135 billion parameters which as a normal 4-bit file is about 20 gig and would have to sit inside ram
00:00:28but through a bunch of techniques like streaming experts off ssd on demand the weights that actually
00:00:33have to live in memory come down to only 1.4 gig i'll explain how all of this works and then we'll
00:00:39go through the iphone setup itself most models sit entirely in ram but the great thing about mixture
00:00:45of experts models is only a small portion are ever active at any one time this means the inactive parts
00:00:51of the model can just sit on ssd waiting for their time to shine the trick to running larger models
00:00:56unlimited memory is only streaming those experts into memory when they're needed we're running quen
00:01:023.5 the 35 billion param variant and the a3b on the end means only about 3 billion of those parameters
00:01:10are active for any one token this has 40 layers each with 256 small experts plus one shared expert a router
00:01:19picks eight of those experts per token so a single token only runs around 3 billion of the 35 billion
00:01:26total parameters now the entire model itself is split usually it all sits in memory but this wouldn't
00:01:32actually fit on an iphone instead the parts every token needs so the embeddings the attention the routers
00:01:39and the shared expert are only loaded once and stay resident on ram the entire time and that's about
00:01:441.4 gig and the experts 40 files of around 300 meg each totaling 12 gigabytes sit on ssd so for each token
00:01:53attention runs on the gpu then the router picks 8 experts and the engine reads those 8 straight off ssd
00:02:00into the gpu memory and runs them together with the shared expert itself this happens in every one of
00:02:06those 40 layers so it's 320 small reads for every single token and if you're unfamiliar with attention
00:02:14it's the part of the model that looks back over everything in the conversation so far and works out
00:02:19which earlier words matter for predicting the next one attention runs on every single token no matter
00:02:25what so it has to stay in memory the experts are the part that do the actual thinking about a token once
00:02:31attention has worked out the context but because only eight of those experts are used we can just leave
00:02:36the rest on disk there is no expert cache in the app at all every read goes through the operating system
00:02:42and the ios keeps recently used experts in its own page cache as long as it has spare ram the authors
00:02:49actually built their own 9.8 gig cache and then deleted it and that made things 38 faster because
00:02:55on a mac or an iphone any ram the app grabs is ram taken away from both the gpu and that page cache and
00:03:03that cache is doing most of the work here at 11 tokens per second if every expert came off flash
00:03:09the phone would need over 5 gigabytes a second of reads and the iphone's flash only does about 1.6 a
00:03:15second so the majority of experts are coming off ram the os is managing for us the next trick is tiered
00:03:22quantization quantization compresses the weights of a model so they take up less space but it also reduces
00:03:29the accuracy of those weights and therefore affects intelligence but with this model around 25 of the
00:03:35experts handle around 80 of the work so the hot experts stay at 4 bits and the cold ones get
00:03:41quantized down to 2 bits and that makes them 44 smaller the whole file shrinks 34 from around 19 gig to
00:03:5013 gig meaning more of it fits in that page cache on my iphone 17 this runs 11 tokens per second with the
00:03:57model in thinking mode i will say though the phone gets hot literally from saying hello i could feel
00:04:03the phone heat up in my hand so try not to explode the phone when you're testing this i'm actually a
00:04:08little bit scared of typing in anything too complex because this might actually melt through my hand
00:04:14the particular engine we're running is a project called flash moe from dan woods it was written for a
00:04:19macbook and there's a small ios port called flash ios that wraps the same engine in an iphone app and
00:04:26that's what lets me run the entire thing on my phone i also did initially hit a bug where the model
00:04:31thinking got stuck in a loop it would start off fine and then about 10 words in it would just repeat
00:04:35the same two tokens forever to fix this i updated a function called async pre-read weight so it checks
00:04:41each experts read against the experts own size cold experts are two bit and half the size of the hot ones
00:04:48so before the fix every cold expert failed the size check and got silently skipped the model was effectively
00:04:54running with half the experts and so it just looped and if you're enjoying this one guys you would do us
00:04:59a huge favor by subscribing to the channel so we can keep creating free content every single day now to
00:05:05get this set up on your phone you need to clone the repo apply the pre-read fix i just mentioned to
00:05:11metal infer slash infer.m point the xcode project at your team and bundle id you will need a paid apple
00:05:18developer account for this build a release build and install it on your iphone download the pre-packed
00:05:24tiered model which is about 13 gig push it to the app over usb which should take around seven minutes
00:05:30on the phone open the flash moe app tap the model and then start chatting and if you want to try running
00:05:36local models on your mac for even faster tokens per second check out this next video i'll be warren from
00:05:43betterstack thank you so much for watching and of course i'll see you in the next one

Key Takeaway

Running a 35 billion parameter Mixture of Experts model at 11 tokens per second on an iPhone requires keeping core components in 1.4 GB of RAM, streaming experts dynamically from SSD, relying on iOS page caching, and applying tiered 2-bit and 4-bit quantization.

Highlights

  • Clever engineering techniques like streaming experts off SSD allow a 35 billion parameter Mixture of Experts model to run on an iPhone using only 1.4 GB of resident RAM.

  • The Qwen 3.5 35B A3B model features 40 layers with 256 small experts and 1 shared expert per layer, activating only 8 experts (around 3B parameters) for any single token.

  • Removing a custom 9.8 GB app-level expert cache increased performance by 38% by letting the iOS operating system page cache manage memory directly.

  • Tiered quantization compresses cold experts to 2-bit while keeping hot experts (25% of experts handling 80% of work) at 4-bit, reducing overall model size from 19 GB to 13 GB.

  • The Flash MoE engine achieves generation speeds of 11 tokens per second in thinking mode on an iPhone 17.

  • Fixing a cold-expert size check bug in the async pre-read weight function prevents the model from entering an infinite two-token repetition loop.

Timeline

Streamed Mixture of Experts Architecture on Mobile

  • A 35B parameter Mixture of Experts model runs locally on an iPhone at 11 tokens per second.
  • Streaming inactive model weights directly off the SSD drops the required resident memory footprint to 1.4 GB.
  • Mixture of Experts models only activate a fraction of their parameters for any given token, making them ideal for memory-constrained hardware.

Standard 4-bit quantizations of 35B models consume around 20 GB of RAM, exceeding smartphone memory limits. Mixture of Experts architectures keep inactive weights stored on flash storage and only load specific parameter subsets into memory when required during inference.

Layer Structure and On-Demand Expert Loading

  • Qwen 3.5 35B A3B uses 40 layers, each containing 256 small experts and 1 shared expert.
  • A central router selects 8 experts per token, executing roughly 3B parameters out of 35B total per token.
  • Embeddings, attention layers, routers, and shared experts remain permanently stored in 1.4 GB of RAM, requiring 320 small SSD reads per token.

Every token requires processing through attention layers on the GPU, followed by dynamic loading of 8 specific expert files from SSD into GPU memory. Across 40 layers, this setup executes 320 discrete reads off the 12 GB expert file storage on the SSD for each generated token.

Leveraging OS Page Caching and Tiered Quantization

  • Deleting an internal 9.8 GB app-level cache improved generation speed by 38%.
  • iOS automatically handles hot expert retention within its native page cache using available free system RAM.
  • Tiered quantization compresses the model file from 19 GB down to 13 GB by reducing cold experts to 2-bit storage.

At 11 tokens per second, un-cached reads would demand over 5 GB/s of bandwidth, whereas iPhone flash memory maxes out near 1.6 GB/s. Because 25% of experts perform 80% of the work, keeping those hot experts at 4-bit while compressing cold experts to 2-bit reduces the overall footprint by 34%, allowing more files to fit into system memory. Allocating memory directly to the app starves both the GPU and the OS page cache, making OS-managed caching substantially faster.

Engine Modifications and Installation Steps

  • The model runs via Flash MoE, a Mac engine wrapped into an iOS application called Flash iOS.
  • A bug in the async pre-read weight check skipped 2-bit cold experts and caused infinite repeating loops until patched.
  • Deployment requires compiling the modified codebase in Xcode with a paid Apple developer account and transferring the 13 GB model over USB.

Because 2-bit cold experts are half the file size of 4-bit hot experts, the original size-checking logic silently dropped cold experts during execution. Updating the size check code restores normal context processing. Once compiled and pushed to the device via USB, the application runs entirely locally on the phone.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video