I Ran a Real LLM on a 6-Year-Old Apple Watch

EnglishالعربيةDeutschEspañolFrançaisBahasa Indonesia日本語한국어PortuguêsРусский
BBetter Stack
Computing/SoftwareConsumer Electronics

Transcript

00:00:00This is an LLM AI model running completely offline on an Apple Watch Series 6,
00:00:07which is a six-year-old watch that came out back in 2020.
00:00:11There's no streaming over the cloud involved here.
00:00:14Everything is working on edge and streaming text at around 15 tokens per second on average.
00:00:20So a few days ago, I challenged myself to see if Apple Watches can actually run AI models locally.
00:00:27And to my surprise, yes, they can.
00:00:30And with a few tricks, you can make them pretty efficient as well.
00:00:33Now, the reason this works is because behind the scenes,
00:00:36it's running a 90 million parameter model called Falcon H1.
00:00:41And because it's so small, it's surprisingly performant.
00:00:44It even supports tool calls.
00:00:46And thanks to the watch's own voice input, it's really easy to prompt it as well.
00:00:51So in today's video, we're going to take a look at how to compile
00:00:55and run an AI model on an Apple Watch locally.
00:00:58We'll talk about why this works and what the best practices are when you're deploying these models.
00:01:03And then we'll run a quick demo to see how it performs and where it still struggles.
00:01:08It's going to be a lot of fun.
00:01:10So let's dive into it.
00:01:15So a few months ago, I did another video where I tested whether a first generation Raspberry Pi could run an AI model completely offline.
00:01:23And it turns out that it can.
00:01:25But in my test, it was struggling at around 0.3 tokens a second.
00:01:30So it worked, but it wasn't really usable.
00:01:33But that got me thinking, what if I put the same model on my six-year-old Apple Watch?
00:01:38What kind of performance would I get out of a proper Apple wearable?
00:01:43And the answer turned out to be about 50 times faster.
00:01:46The Raspberry Pi 1 had a single 700 MHz core and 512 MB of RAM,
00:01:54while the Series 6 has two cores at 1.8 GHz and a full gigabyte.
00:02:00So on paper, this was always going to be a more powerful setup.
00:02:04But I didn't expect it to be better by such a long shot.
00:02:07Now, if you're thinking about how to run inference on a watchOS device,
00:02:12the first thing that naturally comes to mind is Apple's own Core ML.
00:02:16It is Apple's on-device machine learning framework.
00:02:19And the idea is that you convert a model into Apple's format,
00:02:23hand it over, and let the system deal with it.
00:02:26But that doesn't work well with every model.
00:02:28And Falcon H1 is one of those models,
00:02:31because half of its architecture doesn't use attention layers.
00:02:35The way Falcon works is that alongside the usual attention,
00:02:39every layer also runs a state-space model called Mamba 2.
00:02:44It keeps one fixed-sized chunk of memory and updates it one token at a time.
00:02:49Deciding what to keep and what to overwrite as each new token comes in.
00:02:54And that sequence of updates is called a scan.
00:02:57And because each step depends on the one before it,
00:03:00it has to run in order.
00:03:02And nothing gets cached or stored for later,
00:03:04because it all gets compressed into that state.
00:03:07So it's less precise than attention,
00:03:09but it's very memory efficient.
00:03:11And a thousand tokens in,
00:03:13it's using exactly the same amount as it was at token one.
00:03:17And Falcon runs both of these at once,
00:03:20the attention layer and the Mamba layer,
00:03:22and adds the results together,
00:03:24which is great for small devices.
00:03:26The problem is that Core ML has no state-space building block.
00:03:31It only knows how to convert attention layers,
00:03:33so there's nothing for the Mamba layer to convert into.
00:03:37And the deeper issue is that Core ML wants a fixed graph decided up front,
00:03:42while the scan is a loop whose length depends on how long your conversation is.
00:03:47So your options are to unroll it at a fixed length,
00:03:51which locks you into one context size,
00:03:53or carry the state yourself,
00:03:55and thread it through a conversation pipeline
00:03:58that fails by converting perfectly,
00:04:00and then giving you subtly wrong numbers.
00:04:03So Core ML doesn't work for our specific case,
00:04:06but there's another option we can use,
00:04:08which is Llama CPP.
00:04:10But that one has another problem on its own.
00:04:13It ships with a script that builds for macOS,
00:04:16iOS, tvOS, and visionOS,
00:04:19but watchOS is not on that list.
00:04:22But nowhere in the project does it say that it doesn't support Apple Watches,
00:04:26and there's not a lot of information about this specific use case on the web either,
00:04:31so I decided to test it and see if we can actually get Llama CPP to work on watchOS.
00:04:37And fortunately for us, it does work.
00:04:39All it needs is a handful of build flags and a one-line guard in the source.
00:04:45So here's where the confusion arises.
00:04:47Everyone assumes the blocker here is the architecture.
00:04:50WatchOS doesn't use standard ARM64.
00:04:54It uses something called ARM6432.
00:04:57And that name makes it sound like it uses some kind of 32-bit cut-down half-speed variant of ARM,
00:05:05which is not actually the case.
00:05:06Because the instruction set is completely intact,
00:05:10it's the same full 64-bit ARM.
00:05:13Nothing is removed, including Neon,
00:05:15which is the ARM's vector unit that allows batching math operations.
00:05:19So the actual math performance runs exactly as fast as it would on any other ARM chip.
00:05:26The one thing that is different, however, is how the chip refers to memory.
00:05:30So every location in memory has an address, which is really just a number,
00:05:35and on a normal 64-bit system, those numbers are 64 bits long.
00:05:40But on the watch, they're only 32 bits.
00:05:42And Apple did that because a watch only has a gigabyte of RAM anyway.
00:05:47So making every one of those addresses have the size is freeing up memory across the whole system.
00:05:53But that does put a hard ceiling on things because a smaller number can only count so high.
00:05:5932 bits gets you to about 2 gigabytes, and that's it.
00:06:03A watch physically cannot address more memory than that,
00:06:06which means it can never run a large model no matter how aggressively it's quantized.
00:06:12But Falcon is only 57 megabytes, so I'm not remotely close.
00:06:16But this is something you should keep in mind if you want to deploy other bigger AI models on the Apple Watch.
00:06:22And the nice part is that none of this requires porting anything or changing anything.
00:06:27Arm 6432 is just a target you tell the compiler to build for.
00:06:32You just need to pass different flags.
00:06:34So to compile Llama CPP correctly, specifically for watchOS, you need to use this command.
00:06:40But in the watch app project we'll be working with, it is already compiled.
00:06:44And I'll leave a link down in the description below to the GitHub repo that you can clone and run it for yourself.
00:06:50And as you can see on Xcode, the project is quite simple.
00:06:54We just have a models folder where you can import different GGUF files of whichever model you want.
00:07:00Just be aware that it is a very small wearable device,
00:07:03and you won't be able to successfully run a multi-billion parameter model on it.
00:07:09But for this specific demo, I have two AI models added here.
00:07:12One is the previously mentioned Falcon H1, which has 90 million parameters.
00:07:18And I also added a slightly bigger 135 million parameter small M2 model just to see how they compare on the same set of tasks.
00:07:27And then in the tools section, there is also a tools samples file where you can check out some of additional tools that I've added for the use of this app.
00:07:36And if you just rename this file to tools, it will register these specific tool calls instantly.
00:07:41And you can also modify this file to add or remove any other specific tools that you might want in this project.
00:07:48All right.
00:07:49So now let's see how it works.
00:07:51All right.
00:07:51So let's open the app.
00:07:53And here you can choose which model we want to prompt.
00:07:56So first, let's try it with the Falcon H1.
00:07:59And we can just push this button to start speaking.
00:08:02What is the capital of France?
00:08:05And when we're done, we can just click done and then press the ask button.
00:08:11And as you can see, it's using the Wikipedia tool call.
00:08:14And it answered almost instantly that Paris is the capital of France, which is correct.
00:08:20And it did it with a speed of 24 tokens per second.
00:08:24Now let's try the same thing with the small model.
00:08:27We can ask the same question.
00:08:29It's using Wikipedia again.
00:08:31And this one was a bit slower, 14 tokens per second, because this is a bigger model.
00:08:37So that is expected.
00:08:39Now I also added a weather tool call.
00:08:42What is the current temperature?
00:08:44So let's ask that.
00:08:45And as you can see, it's using the weather tool call.
00:08:48And the current temperature is 14.5 degrees Celsius.
00:08:52And I do have to say that for this tool call, I hard-coded the location as Ottawa,
00:08:58just because I didn't want to add the location permission on this app.
00:09:02But if you use this in your own application, you would want to set it to reference the current
00:09:07location rather than hard-coding it.
00:09:09Now let's ask it something more difficult.
00:09:11Can you explain the depth-first search algorithm to me?
00:09:17Wow.
00:09:17It gave me the answer almost immediately.
00:09:20And look at that.
00:09:21It is streaming so quickly.
00:09:23And the longer it goes, the slower it becomes, because I guess we're pushing the boundaries
00:09:29of the memory the watch can hold.
00:09:32So I think we started off at like 20 tokens per second, and now it has dropped to like 10.
00:09:39Yeah, you can see how the token count per second is dropping.
00:09:42So it's probably not ideal for very long responses or difficult questions that require multiple
00:09:50paragraphs of explanations.
00:09:52So as you can see, both of the models have no problems answering questions with detailed
00:09:58responses and are doing it at a relatively quick pace for such a small wearable device.
00:10:05So I'm super happy about these results.
00:10:08So there you have it, folks.
00:10:09We now have a functional inference application.
00:10:12This means that even old devices from 2020, like the Apple Watch Series 6, are powerful enough
00:10:19to run AI models locally.
00:10:21Apple recently came out with their new Series 12 watch.
00:10:25And although it does have an on-device neural engine that can perform certain light AI tasks,
00:10:32I'm surprised that Apple still hasn't shipped a general-purpose local model you can prompt freely
00:10:38on the device.
00:10:39I think tests like I just did and similar experiments have shown that the industry is now in a stable
00:10:44enough state to actually have the capacity to run LLMs locally.
00:10:49So my honest question is, why are device makers still hesitant to make the jump to on-edge
00:10:55LLMs?
00:10:56Do you have any thoughts on that?
00:10:58Let me know what you think in the comment section down below.
00:11:01And folks, if you like these types of technical breakdowns, please let me know by smashing that
00:11:05like button underneath the video and also don't forget to subscribe to our channel.
00:11:10This has been Andres from BetterStack and I will see you in the next videos.
00:11:25Bye.
00:11:26Bye.
00:11:26Bye.
00:11:26Bye.
00:11:26Bye.
00:11:26Bye.
00:11:26Bye.
00:11:27Bye.

Key Takeaway

A six-year-old Apple Watch Series 6 executes a 90-million parameter LLM completely offline at 15 tokens per second through Llama CPP compilation and ARM6432 architecture.

Highlights

  • An Apple Watch Series 6 runs an offline 90-million parameter Falcon H1 model locally at an average speed of 15 tokens per second.

  • WatchOS utilizes an ARM6432 architecture that supports full 64-bit instructions while restricting memory address lengths to 32 bits.

  • Apple's Core ML framework fails to process hybrid attention and state-space models because it lacks a built-in state-space building block.

  • Llama CPP successfully executes on watchOS after applying targeted build flags and a single-line source code guard.

  • Memory consumption remains constant across long conversations in state-space architectures due to fixed-size chunk compression.

Timeline

Watch Hardware Performance and Baseline Testing

  • The Apple Watch Series 6 executes local AI models completely offline without cloud streaming.
  • Performance reaches approximately 15 tokens per second using a 90-million parameter model.
  • Wearable hardware processes inference 50 times faster than a first-generation Raspberry Pi.

Offline edge computing on wearable devices becomes viable through small-scale language models. A six-year-old Apple Watch featuring two 1.8 GHz cores and 1 GB of RAM exceeds the processing capacity of older single-core microcomputers by a wide margin. Small parameter footprints enable acceptable generation speeds on restricted hardware.

Core ML Limitations and State-Space Architecture

  • Apple's Core ML framework rejects hybrid models that incorporate state-space layers.
  • The Falcon H1 architecture combines standard attention layers with Mamba 2 state-space blocks.
  • Mamba 2 compresses conversation history into a fixed-size memory chunk updated token by token.

Standard on-device machine learning frameworks require fixed graphs established prior to execution. State-space models update sequentially via loops where conversation length dictates step counts, conflicting with Core ML requirements. Hybrid designs utilize constant memory regardless of sequence length, maintaining efficiency on small devices despite lacking native framework support.

Llama CPP Adaptation and ARM6432 Constraints

  • Llama CPP operates on watchOS following custom compiler flag additions and a single source code adjustment.
  • The watchOS operating system runs on ARM6432, maintaining full 64-bit math instructions including Neon vector processing.
  • Memory addresses use 32 bits instead of 64 bits to conserve system resources, enforcing a hard ceiling of 2 gigabytes.

Compilation barriers on watchOS stem from missing default targets rather than hardware incompatibilities. The instruction set preserves full vector math execution speeds. Reduced address lengths save system RAM across the device but restrict maximum model sizes permanently, regardless of quantization levels.

Model Demonstration and Token Generation Speed

  • The Falcon H1 model achieves 24 tokens per second on simple retrieval queries utilizing tool calls.
  • Larger 135-million parameter models run at slower speeds of 14 tokens per second.
  • Generation speed decreases during long responses as token counts push memory boundaries.

Local execution handles external tool integration including Wikipedia lookups and hard-coded weather data. Simple factual queries resolve almost instantly with high generation speeds. Complex explanations cause performance degradation over time as conversation context expands within tight hardware constraints.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video