Transcript
00:00:00You can get roughly 4 billion LLM tokens a month for free and use them through a single
00:00:05OpenAI compatible endpoint. All this without constantly switching providers, which is pretty
00:00:10sweet. This is free LLM API. And yeah, that name is a bit obvious. It pulls free tiers from 28
00:00:17providers, then roots and fails over between them automatically. It's already passed 18,000 stars
00:00:23on GitHub, and I'll get it running in under a minute. Then we'll find out whether 4 billion
00:00:28free tokens is actually useful or whether it's just the punchline.
00:00:36Now, here's the weird situation we're in. There's actually a lot of free LLM capacity available.
00:00:43Grok has a free tier. Cerebrus has one. Google has one. Mistraw and NVIDIA open route are free
00:00:50models. There's plenty out there already. So you'd think the problem is already solved. Well,
00:00:54as many of us know, it's not because now you have different API keys, different dashboards,
00:01:01different rate limits, different model names, and slightly different APIs. And if you use coding
00:01:06agents, you already know what happens next. You're 10 minutes into a task, your provider hits a limit,
00:01:11and now you're stopping what you're doing just to swap keys, change models, or edit that config.
00:01:16The inference is free, though managing the free inference is the annoying part. That's what free
00:01:22LLM API does. You give it your provider keys. It tracks quota, health, and rate limits. It routes
00:01:28requests between providers. And if one hits a limit, it can jump over to another one all on its own. But
00:01:35your tools only see that one endpoint. So cursor, cloud code, codex, CLI, or even your own scripts don't
00:01:42need to know what's happening behind it. Let's set it up. If you enjoy coding tools to speed up your
00:01:46workflow, be sure to subscribe. We have videos coming out all the time. Alright, honestly, the best thing
00:01:52I found was just to run this with Docker. So a simple git clone, then CD into it. I generated a 32-byte
00:01:57encryption key which I needed, then just ran Docker Compose. Once it's running, we get this local
00:02:03dashboard. I'll go through in a second. But first up is the API keys. You can see here, I added my API
00:02:10keys from all of these options. We got Google, Grok, Cerebrus, Mistraw, and OpenRouter. You can add as
00:02:17many as you want from this list here. These credentials are stored encrypted locally. And after they're added,
00:02:24free LLM API builds a model catalog from the providers you've configured. Then here in models, you can see how
00:02:31many monthly tokens I'm at. This is quite a lot. And all the models that this will be running to get me all those
00:02:37tokens. If I scroll down a little bit, I can even choose my routing strategy, which I set to balance, and they are
00:02:44currently ranked by intelligence. You can run a quick test here in Playground. I'll fire this up, this prompt. Okay, it
00:02:51returns. Now I'm good to go. But this isn't actually the part that I care about. The useful part is what I
00:02:57no longer have to configure. Instead of pointing every coding tool directly at Grok, then Google, then
00:03:03Cerebrus, I point the tool here instead. It's one API key. I'm using that free LLM API. Now, that's
00:03:10basically the whole integration part. And if you're using the normal OpenAI client, it's the same idea.
00:03:16I want to give it a longer coding task here just to see how and what it does. So I wrote out this Python
00:03:23script to use my API key. And you can see here in my client instance, I pass on the free LLM API key
00:03:31I got back. I'll just have it build out a CLI task manager in Python with all of these requirements.
00:03:38Okay, it's a it's a simple project, but let's just run it. And I want to see its reasoning. It also
00:03:43should write out multiple files and return to me a readme file too. I'm going to run it and we're
00:03:49going to wait a second. There's nothing really surprising here yet. My application isn't choosing
00:03:54the provider anymore, the router is. So if this provider gets rate limited or becomes unhealthy,
00:03:59free LLM API can move the request somewhere else instead of my agent just dying halfway through the job.
00:04:05And now my script encoding agents aren't sitting behind one free small quota. They're sitting
00:04:10behind the pool. Then here is the final output I got back from my code, which is just a terminal output.
00:04:16So I made the files on my own. Now I'm going to run and test it. It all works really well. But you
00:04:22can see this project was simple as I only used a few tokens here, right less than 2000, give or take.
00:04:29Now all this sounds great, but it creates an obvious question. How does it decide where
00:04:34anything actually goes? Now it's all actually pretty simple. Your provider keys stay encrypted locally.
00:04:40Your request comes into free LLM API. Then the router looks at the providers you have available.
00:04:47It knows which ones are healthy and which ones are running into rate limits or quota problems. Then it sends
00:04:53out the requests. Your tool still talks through a familiar API surface, including an open AI compatible
00:04:59endpoint. So from the application's point of view, nothing complicated is actually happening. Think of it
00:05:05like this. Okay. Our coding agents really see one door behind that door. Free LLM API decides which
00:05:12provider should actually answer. And that explains why this project is really interesting. Now it's handling all
00:05:17this stuff for us. Free tiers used to be something you'd have to test a model a few times. Now there's
00:05:24enough free inference spread across enough providers that the combined capacity is pretty useful. So the
00:05:30problems changed. The problem used to be, where can I get free inference? Now it's how do I use all this
00:05:37without constantly having to manage it? That's the problem. Free LLM API is built around. And for us devs,
00:05:43there are plenty, pretty obvious places where that matters. Agent loops where extra seconds of latency
00:05:50isn't a big deal. If we are just playing around, okay, great prototyping before moving to a paired
00:05:56model, running multiple coding agents without constantly burning paid credits. There's a ton of
00:06:01reasons where this can become really useful. Even if we're just learning for just playing around on side
00:06:06projects, basically workloads where cost matters more than perfectly consistent latency. And that's an
00:06:11important factor between all of this. Because once you understand that, you also understand why this
00:06:15isn't just open router or light LLM with a different logo. So why not just use open router? Why not just
00:06:21use light LLM? Well, because the overlap is still there, but the goal is different. So if I start with
00:06:27open router, if you want one managed API with access to a lot of models, open router is incredibly
00:06:32convenient. It also has free models, but it's hosted. Your requests go through open router and you're using the
00:06:38capacity available through their platform. Free LLM API works differently. You're combining your own
00:06:44provided accounts and your own free tier quotas. That's what gets pulled. Then there's light LLM.
00:06:51If I'm building a serious multi-provider gateway for an application, that's a very different conversation.
00:06:57It's broader. It's more configurable and it's much more production oriented, but flexibility also means
00:07:03you're the one configuring the routing strategy and maintaining the setup. Free LLM API has a much narrower
00:07:09job. Take the free capacity I already have and just make it easier to use. That's all that's really
00:07:14happening here. So I wouldn't look at this as a replacement for every LLM gateway. I'd look at it as
00:07:20free tier maximization. But the same thing that makes this useful also creates its biggest weakness.
00:07:27Let's start with what it gets right. First, the pooled capacity. One tiny free tier isn't that useful. A bunch
00:07:32of them together. Now you can actually get work done. Second, one key, one endpoint. This is probably
00:07:39the biggest quality of life improvement. I can configure a coding agent once and mostly stop caring which
00:07:46provider is answering each request. Huge win. Then local first. My provider credentials stay on my machine
00:07:54instead of being handed to another hosted routing service. And finally, the failover. This matters
00:07:59more to me than the number of tokens we're getting because free tiers are inconsistent. They have rate
00:08:05limits. They have daily caps. And they change. So the valuable part isn't just having several providers.
00:08:11It's being able to move away from one when it stops working. But here's the catch. Pulling inconsistent
00:08:16free services doesn't just fix everything and make them consistent. At the start of the day,
00:08:21maybe your preferred free model is available. Later, you've burned through that quota. And what's
00:08:27happened? Well, now your request may be rooted somewhere you wouldn't have chosen in the first place.
00:08:32So model quality can vary. Latency can vary. And you're not suddenly getting unlimited access to
00:08:37frontier models. If your workflow absolutely depends on the best models every single time,
00:08:42this doesn't solve that. I would not put production infrastructure on top of this and pretend free
00:08:46tiers are paid infrastructure. Absolutely not. A provider can change its quota. A model can disappear.
00:08:52A lot can change really fast. I'll put the repo below. If you have free tier keys sitting
00:08:57around doing nothing, this is probably worth trying just to see how much useful capacity you can
00:09:01actually get out of them. If you enjoy coding tips and tricks like this, be sure to subscribe to the
00:09:05BetterStack channel. We'll see you in another video.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video