A $10 Chip Is Running a 45M Parameter AI Model (Needle 2)

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語한국어PortuguêsРусский中文
BBetter Stack
Computing/SoftwareConsumer Electronics

Transcript

00:00:00This is Needle 2. It's a 14 megabyte Argentic LLM model. Let me repeat that in case you missed that.
00:00:07It's 14 megabytes and it has 45 million parameters. Basically this LLM is so tiny it can actually run
00:00:14on any edge device down to a bare ESP32 S3. It was developed by Cactus Compute who have now
00:00:21open sourced this model under the Apache 2.0 license and I tested it myself and it actually
00:00:28works. So in this video we're going to take a look at Needle 2, see how it works and then we're going
00:00:34to run some fun inference tests on an actual microcontroller to see how it performs. It's
00:00:40going to be a lot of fun so let's dive into it. So a couple of weeks ago I did another video where
00:00:49I deployed a 29 million parameter model on an ESP32 S3 but that was more of a gimmick and not
00:00:56an actual LLM because it was so bare bones it was trained on children's stories so that's all it
00:01:03could ever output. This LLM however is an actual model that understands your context but before you
00:01:10get too excited there is a caveat to it. This is not going to be your regular chatbot LLM. Instead
00:01:16this model is specifically made to work as a tool call dispatcher. So the way it works is that you
00:01:22predefine a set of tool calls and that could be anything from rotating a servo to any other mechanical
00:01:29or executable operation that you can think of. And then you can prompt the model in natural language
00:01:35and it will be smart enough to understand based on your command which tool call to execute. And that's
00:01:40basically the gist of it. Now this is really cool because on a board like a Raspberry Pi 5 it can achieve
00:01:47up to 500 tokens a second so it can be a very efficient driver for let's say robotics projects. But on bigger more
00:01:55performant devices can Needle 2 actually operate as a chatbot? Well the answer is no because it's not really a speed
00:02:04question because Needle wasn't trained on general knowledge or conversation at all. Its entire training data is
00:02:10device actions like smart home commands, mobile actions, wearable actions, that kind of stuff. So
00:02:17even on a beefy device if you ask it what's the capital of France it won't be able to answer. It's going to be
00:02:23really good at picking the right function and filling in the right arguments but that's it. And that's the
00:02:28trade-off they had to make to get it this small. But let's actually talk about how they pulled off
00:02:3345 million parameters that are actually competitive with models five or seven times bigger. Because
00:02:40normally when you shrink a model that much it just gets dumb. But Cactus didn't want that so they rebuilt
00:02:47a few pieces of the architecture from scratch. So the big change is something they call an engram.
00:02:53Normally all the model's knowledge lives in its weights and every single one of those weights has to be run
00:02:59through a matrix multiplication for every token and that is expensive. But Cactus moved a chunk of that
00:03:06knowledge into hashed lookup tables instead. So the model can just go grab a row of memory instead of
00:03:12doing math on it. And they also swapped the normal MLP layers for something built on a hadamard transform,
00:03:19which is a fixed mathematical operation with almost no learnable parameters attached to it. Normally that
00:03:25mixing step is done by huge matrices that the model has to learn from scratch. And that eats up most of
00:03:32the small model's parameter budget. But a hadamard transform skips all that learning because it's a
00:03:38fixed formula that does all the mixing for free. So Cactus gets that benefit without spending any of
00:03:44their 45 million parameters on it. And then there's quantization, which honestly is the part that makes
00:03:50all of this deployable at all. Needle 2 runs on two bits per weight and it was actually trained on two
00:03:57bits, not quantized down after the fact. The problem is that a lot of small models fall apart when you
00:04:03compress them post training because they were never built to survive that compression. But Needle was
00:04:09trained against its own compression from the start, so their default version is already optimized. So does any of
00:04:16this actually hold up? Well, on Cactus' own benchmarks, Needle 2 is genuinely going head to head with a model
00:04:23five times its size. And on a couple of the harder tool calling tests, it actually beats it outright,
00:04:30despite running a two-bit precision against LFM's 2.5 full 16-bit precision. And on the mobile actions
00:04:38benchmark, Needle picks the correct function name 98.3% of the time, and that's higher than every single
00:04:45model it's compared against. So it's not perfect at everything, but the job it was actually built for,
00:04:51it's really, really good at. So let's test it out to see how it actually works. So for this purpose, I built
00:04:57my own custom inference engine that actually runs on this exact board, an ESP32-S3. And I'll leave a link to the
00:05:05GitHub repo in the description below if you want to try it out for yourself. So here in the terminal,
00:05:10we can see that we have a little TUI application. So now I can just prompt it with a natural language
00:05:17request and see how it works. So first let's prompt it with something simple. Let's say to flash a red
00:05:25light for three seconds. And as soon as I launch this, you can see it's mapping the model straight off
00:05:32the flash chip and setting up its working memory. And once it has done so, it goes through the reasoning
00:05:39phase. And we can see that it identified that the color is red, the mode is flash, and it also identified
00:05:48that we need to do it for three seconds. So once it's done with the reasoning phase,
00:05:53it will proceed to writing the actual tool call. And this takes a bit of a moment because it is slow
00:06:00on an ESP32. But nonetheless, this is pretty cool. And look at that, we now get a flashing red light for
00:06:08three seconds. And we can also see that it has a confidence score that tells us how confident it was
00:06:15about this particular tool call. And this is built in needle two. And this is pretty cool because based on
00:06:21this confidence score, we can program the chip based on different scenarios. So as you saw, this took about
00:06:2739 seconds to complete, which might sound slow. But remember that this is a 45 million parameter model
00:06:35running on an eight megabyte microcontroller with no GPU. Now, let's see what happens if I ask it
00:06:41something totally out of context. So let me just ask it, what is the capital of France?
00:06:48And as you can see here, it's reading our request again. And now in the reasoning phase,
00:06:54it says that there's no tool available for country facts. So the tool call is going to be empty. And
00:07:00look at that, it was very confident that there are no tools that can confidently be executed based
00:07:07on this prompt. So I would say this is still a very intelligent response. This time, let's ask it to
00:07:14shine a purple light for seven seconds. And let's see if it understands that. And it identifies that
00:07:21color is indeed purple. And it understands that shine means that the mode has to be solid and not flashing.
00:07:29And it also understands that it has to be for seven seconds. So then it proceeds to write the actual tool
00:07:36call. And once it has done so, look at that. It's now shining a purple light. Should be seven seconds.
00:07:46There you go. How cool is that? And interestingly enough, this time it wasn't as confident.
00:07:52So the confidence score was 0.57, but it's still managed to reason through the task accordingly and
00:08:00execute exactly what we asked. And there's one more thing I want to show you. I built this project
00:08:06specifically to be reusable. So you can swap out the flashing light commands for something else. Like,
00:08:12for example, if you have a robotics project that is using servos, you can basically replace these
00:08:17commands with your own. And I've documented a three step process how to do that in the project's readme
00:08:24file. So if you want to reuse it for your own projects, just follow those three steps and you're
00:08:29good to go. Let's try another variation. Display a yellow light for five seconds. And there you go.
00:08:36It's flashing a yellow light for five seconds. And this had the lowest confidence score of the mall,
00:08:420.46. And I found interesting that it thought that displaying meant flashing a light and not displaying
00:08:50it for a prolonged period. So there you go. That shows that the model is not perfect. It might still
00:08:56misunderstand your intentions, but nonetheless, we got a yellow light and we got it for five seconds. And
00:09:03most of these tool calls take around 40 seconds to process with 1.86 tokens a second on average. So
00:09:10that is a pretty cool result in my opinion. So there you have it folks. That is needle two in a nutshell,
00:09:16a 45 million parameter model, 14 megabytes running completely offline on a chip that costs about 10 bucks.
00:09:24And folks, if you like these types of technical breakdowns, please let me know by smashing that
00:09:28like button underneath the video. And also don't forget to subscribe to our channel.
00:09:33This has been Andrus from BetterStack and I will see you in the next videos.

Key Takeaway

Needle 2 operates as a 45-million-parameter tool call dispatcher running entirely offline on a 10-dollar ESP32-S3 microcontroller using two-bit quantization.

Highlights

  • Needle 2 is a 14-megabyte LLM containing 45 million parameters that runs locally on edge devices like the ESP32-S3 microcontroller.

  • Cactus Compute open sourced Needle 2 under the Apache 2.0 license.

  • The model operates as a tool call dispatcher rather than a general-purpose conversational chatbot.

  • Needle 2 reaches speeds up to 500 tokens per second on a Raspberry Pi 5.

  • Hash lookup tables and a hadamard transform replace traditional weights and heavy matrix multiplications to reduce parameter usage.

  • Quantization at two bits per weight enables deployment directly on low-cost hardware.

  • Inference execution on an ESP32-S3 microcontroller averages 1.86 tokens per second and completes tasks in about 40 seconds.

Timeline

Model Overview and Architecture Innovations

  • Needle 2 is a 14-megabyte model with 45 million parameters designed exclusively for tool call dispatching.
  • The architecture shifts knowledge storage from model weights into hashed lookup tables.
  • A fixed hadamard transform replaces standard MLP layers to eliminate learnable parameters.
  • Quantization uses two bits per weight from the initial training stage onward.

Traditional LLMs require extensive memory and computational power for every token generation. Needle 2 bypasses this limitation by restricting its training data strictly to device actions like smart home and mobile commands instead of general knowledge. By utilizing hashed lookup tables and hadamard transforms, the model cuts down heavy matrix multiplication costs. Two-bit quantization ensures the entire model fits onto tiny microcontrollers without breaking down during compression.

Microcontroller Inference Testing

  • An ESP32-S3 microcontroller runs the model completely offline through a custom inference engine.
  • Natural language prompts map directly to hardware tool calls such as flashing lights with specific colors and durations.
  • Out-of-context prompts result in empty tool calls accompanied by high confidence scores indicating no matching tool.
  • Inference tasks on the ESP32-S3 process at an average speed of 1.86 tokens per second over approximately 40 seconds.

Testing the model on a bare ESP32-S3 board demonstrates practical edge deployment capabilities. When prompted with specific color commands and time durations, the model reasons through the request and executes the corresponding hardware instruction. Confidence scores generated during inference help determine execution reliability across different scenarios. The reusable project structure allows developers to substitute light control commands with custom robotics operations like servo movements.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video