This Tool Finds the Perfect AI Model for Your Hardware (llmfit)
BBetter Stack
Computing/SoftwareConsumer ElectronicsInternet Technology
Transcript
00:00:00A quick question. Do you know how many AI models are out there right now?
00:00:04Well, according to Hugging Face, there are over 3 million publicly available models out there.
00:00:10And let's say you chose a model that you really like, you go to their Hugging Face page,
00:00:15you try to download it, and there's like 9 different quantizations, 4 different parameter
00:00:21sizes, and no indication on whether any of that will actually run on your particular machine.
00:00:26Ideally, you would want to know if this model is going to throw an out-of-memory error on you
00:00:31before you download the full 8 gigabytes. So how do you decide which one to use?
00:00:36Well, luckily, there is a tool for that now. It's called LLM Fit, and it looks at your actual hardware,
00:00:43your GPU, your RAM, your CPU, and then it tells you out of hundreds of models which ones will genuinely
00:00:49run well on your machine, and how well will they perform. And as you can see here, I got a bunch of
00:00:55different devices on the table. So in this video, we're going to take a look at LLM Fit, see how it
00:01:01works, and then I'm going to test it out on a bunch of these machines and see which models fit each of
00:01:06them. It's going to be a lot of fun, so let's dive into it.
00:01:13Now, at its core, LLM Fit is just a simple terminal tool. You run one command, and then it detects
00:01:19your system, your CPU cores, RAM, and GPU, including multi-GPU setups. And then it scores your setup
00:01:26against a bunch of available models from their database. And not only that, but it also gives you
00:01:32an indication how many tokens per second you can expect on your local setup when running this
00:01:37particular model. And of course, to properly estimate your hardware capabilities, it uses the best tools
00:01:43for the job, depending on your GPU cards and your infrastructure. Plus, it also evaluates if you have
00:01:49any acceleration backend installed on your machine. And it also tries to estimate the best quantization
00:01:55level of that particular model based on your hardware requirements. And each model is scored against
00:02:01four criteria. Quality, speed, fit, and context. And then those get combined into a composite score.
00:02:08But the way they are weighted against each other isn't always fixed. So for example, chat cares more
00:02:14about speed, while reasoning weighs quality much higher. So the same model can rank differently,
00:02:20depending on what you're actually trying to do with it. But how does LLM Fit actually determine
00:02:25those speed numbers? Well, the way it works is that every token means reading the entire model's weights
00:02:31off your VRAM once. So speed really comes down to how fast your VRAM can feed the GPU.
00:02:38LLM Fit identifies the real memory bandwidth from your specific card. So it takes that number and then
00:02:44runs it against the model size to estimate how many tokens per second you'll actually get. And if you
00:02:49want to get into the weeds of all that, there's also an advanced config panel where you can tune all of
00:02:54that yourself. And they added that after someone on GitHub pointed out that token per second estimates
00:03:00were coming in too high for certain models. And if you have some other GPU that is not in the bandwidth
00:03:06table, it falls back to per backend constants, CUDA, Metal, Rock M and so on. So you're never left with a
00:03:14zero estimate. You might just get a slightly less precise one. And one more cool thing is that they
00:03:19have a community leaderboard where you can check performance results from real users using the same
00:03:24hardware as you. So you can actually compare it against a practical benchmark and not a theoretical one.
00:03:31All right, this tool sounds great. So I want to test it out on a bunch of different devices because I'm
00:03:36really curious what kind of models can you actually run on older devices. So we're going to start with
00:03:42this first gen Raspberry Pi. And this thing is from 2012. Single core, 700 megahertz and 512 megabytes
00:03:51of RAM. A couple of months ago, I did a video where I successfully deployed a 19 million parameter model
00:03:56on it. So let's see if LLM fit will suggest something similar in our test. So a couple of things actually
00:04:03happened here. First, it did run, but it was so painfully slow. This board just doesn't have the
00:04:09horsepower to run a Rust 2i comfortably. But anyway, let's look at the recommendations and they are
00:04:16genuinely hilarious. First of all, it recommends DeepSeq R1 as the best choice with almost 400
00:04:23billion parameters. And that would take 230 gigabytes of disk space. But at least it's being honest when
00:04:30calculating tokens per second, putting it at only 0.8. And the memory percentage needed for this thing is
00:04:38also bonkers. Now, as I said, I did test out this board before running a model called Falcon H1 Mini.
00:04:46So I tried to search up the same model and see if it's in the database. And it actually is there,
00:04:51but oddly enough, it only gives us a score of 51 compared to the enormous DeepSeq R1. And this is
00:04:58the funniest part. It says that it can reach 241 tokens per second. Now, in my original video,
00:05:06it clocked around 0.3 tokens per second. So this is what 800 times more than that. So obviously something
00:05:13is wrong here. And I guess for boards as old as this one, it just can't calculate a realistic candidate,
00:05:20which is fair. I didn't even think this app would run on a Raspberry Pi at all. So I'm not too concerned
00:05:26about that. Now we're going to move on to a Lockfox Pico Ultra W. This is not even really a computer in
00:05:34the normal sense. It's a tiny embedded board built for camera modules, security systems, that kind of
00:05:40thing. But it does have Linux on it. And it's single core Cortex A7, 256 megabytes of RAM. So since this
00:05:48Lockfox board comes with a specific Linux version pre-installed on it, I could just SSH into it from
00:05:54my MacBook and run LLM fit remotely. And here we see something similar to the Raspberry Pi board.
00:06:01It recommends the same DeepSeq R1 model with 400 billion parameters, and it sets the same score.
00:06:08So I'm not sure. I feel like this app is just not meant to calculate model recommendations for such
00:06:14small development boards. Next, we're going to test it on a 2015 13-inch MacBook Pro, 8 gigs of DDR3 RAM,
00:06:23Intel Core i5. This is actually the first one in the list that is a real computer people use to do real
00:06:30work on. So I'm curious whether it will find anything worth running at all, or if this is still going to
00:06:36be a dead end. So as expected, this is the first machine where the lineup of recommendations actually
00:06:42starts looking quite usable. So now we get a list full of small models that are all flagged as being
00:06:48good. So that's a great sign. Mostly all of them are quantized down to eight bits, estimated anywhere
00:06:55from 40 to 170 tokens per second, depending on the model. But, you know, to be honest, I think this is
00:07:02too generous of an estimate. I don't actually think Llama 3 could reach that high of a token count on such
00:07:09an old 2050 MacBook. But I don't know. Let me know in the comments down below if you've actually tested
00:07:15a model like Llama 3 on such an old MacBook. And another cool thing you can do in this tool is to
00:07:20filter down use cases to check if there are any good coding models, for example. And when I strip that
00:07:27down, we get the best recommendation being Quen 3.6 with 12 billion parameters and 2-bit quantization.
00:07:35Although I'm not too happy about 2.3 tokens a second, but I guess that's a trade-off you're making
00:07:41for running 12 billion parameters on such an old laptop. Next, we're going to test it on my M2 Max
00:07:47MacBook Pro with 32 gigabytes of unified memory. And this is where things should start looking a lot
00:07:53more normal. It has real GPU cores and real memory bandwidth and LLM Fit actually understands Apple
00:08:00Silicon properly. So I want to see what it recommends here and how close the speed estimate comes to what I
00:08:06actually get normally. So on my M2 MacBook, as we can see, the best recommendation is DeepSeq R1.
00:08:13Although I'm not sure what kind of model this is. It says DeepSeq R1, but it also has Quen tied to it.
00:08:18So I'm not sure. Is this some kind of a symbiotic model of some sorts? But judging by this tool,
00:08:23I guess this is the best case scenario for my MacBook. But once again, if we filter down,
00:08:27looking specifically at the best coding model, we can see the Gemma 4 12 billion version comes out on top.
00:08:34And I actually do agree because I recently did a video on this model and it was a beast,
00:08:39especially on MLX setups. So I do agree with that result. And finally, I'll test it on my beefiest
00:08:45machine, the RTX 5090, 64 gigabytes of DDR5 RAM, Intel Core Ultra 9, 285K. This is the one where I
00:08:55actually expect LLM Fit to start recommending the big models. So let's see what it thinks I can get away
00:09:01with. So on my beefy machine, I get a bunch of recommendations for different Quen 3.8 versions.
00:09:07And as you can see this time, I'm getting a perfect score on almost all of them. So there's plenty of
00:09:12models to choose from. And amongst all the Quen versions, we also get Gemma 4 12 billion version.
00:09:18But here I also want to filter out to see what are the best coding models. And even with this filter,
00:09:23I'm still getting a bunch of Quen 3 models. But then if I sort it by context size, we can see that
00:09:30there's actually Kimi K 2.7 D flash version, which was released by Nvidia. So that is really interesting.
00:09:38And this is something closer to the actual model that I would choose for this specific task.
00:09:43Not trying to be harsh on Quen, but I'm going to be honest, I'm not a biggest fan of their models.
00:09:47So Kimi K 2.7 would probably be a better pick, in my opinion, based on these.
00:09:53So there you have it, folks. That is LLM Fit in a nutshell. I think nowadays, when there are so many
00:09:58AI models to choose from out there, we tend to get in an analysis paralysis mode. So it's nice that
00:10:04there are some tools out there that can at least make this task of choosing the right AI model a bit
00:10:10easier for us. So what do you folks think about this tool? Have you tried it? Will you use it?
00:10:15Let us know in the comment section down below. And folks, if you like these types of technical
00:10:20breakdowns, please let me know by smashing that like button underneath the video. And also don't
00:10:25forget to subscribe to our channel. This has been Andrus from BetterStack and I will see you in the next videos.