Someone Built Ollama for Audio… and It Runs Locally
BBetter Stack
Computing/SoftwareConsumer ElectronicsInternet Technology
Transcript
00:00:00Somebody is quietly trying to do for audio what Llama CPP did for chatbots.
00:00:05Run every model on your own machine.
00:00:08One binary, zero Python.
00:00:10This is Audio CPP, and it's doing just that.
00:00:13While it's only about a month old, and it still crashes sometimes,
00:00:17it still might be where all this is headed.
00:00:20Let's take a look.
00:00:26Now, rewind a couple of years.
00:00:28Running an AI model locally used to be a pain.
00:00:32A fresh environment for every model, PyTorch versions clashing, CUDA never matching.
00:00:37Then a tiny C++ library called GGML showed up.
00:00:41It's the engine under Llama CPP and Whisper CPP,
00:00:44and suddenly you could run a language model or transcribe an hour of audio as a single native binary.
00:00:51No Python, no cloud.
00:00:52It was just fast, local, and it got done.
00:00:55But there was a hole nobody was filling.
00:00:56Whisper CPP only listens.
00:00:59Speech in, text out, it's done.
00:01:01It can't speak back.
00:01:02It can't clone a voice.
00:01:04It can't make a sound that wasn't already there.
00:01:06So this project, Audio CPP, asked the obvious question everyone skipped.
00:01:11What if one binary did the whole job?
00:01:13Text-to-speech, speech-to-text, voice cloning, voice conversion,
00:01:17figuring out who's talking, even generating music.
00:01:21Roughly 30 model families, one runtime, it's all in this.
00:01:25If you enjoy coding tools that speed up your workflow, be sure to subscribe.
00:01:28We have videos coming out all the time.
00:01:30Now, it's easier if I just show you guys.
00:01:32Here is a 10-second clip of my own voice.
00:01:35I hand that reference straight to Audio CPP.
00:01:38Pure C++ inference on G, G, M, L, no Python runtime.
00:01:43Nothing really leaving the machine at all here.
00:01:46I type a sentence I just made up.
00:01:48This is me talking on my MacBook about Audio CPP.
00:01:51The weather is beautiful.
00:01:53And that's me, cloned, reading words that didn't exist a few seconds ago,
00:01:58running entirely on my Mac with the metal back end.
00:02:01Nothing is leaving the machine.
00:02:03One more, different input so you can hear the consistency or where it kind of fails.
00:02:07This is another AI voice test for local voice generation.
00:02:11And there we go.
00:02:12Quick important note, I'm timing this live on purpose.
00:02:15Every speed number this project brags about was measured on an NVIDIA 5090, not a Mac.
00:02:22Hold that thought so I can come back to it in a second.
00:02:24So how does this even pull it off?
00:02:26Well, underneath, it's GGML, doing the math.
00:02:30With backends, you can swap out.
00:02:31Regular CPU, NVIDIA, Vulkan, and Metal for Apple Silicon.
00:02:35You drive it with flags.
00:02:37Pick a task, pick a model family, pick a backend.
00:02:40There's a command line tool for quick runs.
00:02:42And this is the kind of sneaky part, we could say.
00:02:45A server mode that speaks the exact same API as OpenAI's audio endpoints.
00:02:50So code you already wrote for the cloud, pointed at localhost, it's the same shape, it's just
00:02:56without the bill.
00:02:57Now, here's why its position is actually really cool.
00:03:00Whisper CPP only listens.
00:03:03Ollama and Llamacpp are text brains.
00:03:06They're not for audio.
00:03:07Koki and Piper only talk.
00:03:10Ffmpeg shoves audio around, but doesn't run a single neural model.
00:03:16CPP is the first thing standing in the middle, trying to beat all of them.
00:03:20In one static file, you can copy to another machine and just run.
00:03:23Now, one catch here.
00:03:24A couple models wear an MLX label, but it doesn't actually use Apple's MLX.
00:03:30It's plain GGML on a metal backend.
00:03:33All right.
00:03:33It's too new to even preach this, so I'm just going to hit on the things that aren't that
00:03:38great yet.
00:03:38Still early.
00:03:39First up, those big numbers.
00:03:41Ten hours of audio in three minutes, five times faster than PyTorch.
00:03:45Every single one is a 5090.
00:03:46There's no one single published on an Apple Silicon yet.
00:03:50That's the real reason I timed this right now instead of just reading from the state of
00:03:54this.
00:03:55Next up, there's no ready-made download for this.
00:03:58The only pre-built releases are Windows.
00:04:00On a Mac, you can pilot yourself with a metal build script and Xcode tools.
00:04:05It's doable.
00:04:06Took a little bit of time, though.
00:04:07It's not just one click.
00:04:09Then, that no Python promise, it's true when you run it.
00:04:12Downloading and converting the model still leans on a Python helper, but it's not quite
00:04:16there.
00:04:17And lastly, this is essentially one person.
00:04:19A few people are running this now, at a very early version, and the issue tracker is full
00:04:24of real crashes, memory leaks, and a few models that come out sounding like robotic.
00:04:28This is a fast-moving experiment, so we're going to treat it like one.
00:04:32So, who should actually use this?
00:04:34Well, it's really early, right?
00:04:35So, if you love local AI, you don't flinch at compiling something, and you'll test the
00:04:40output out yourself.
00:04:41This is a blast, and the license was clean.
00:04:44It's Apache 2.0.
00:04:45GitHub says no license, but open the file.
00:04:47It's plain Apache.
00:04:48The detector just tripped over the copyright line, it looks like.
00:04:51So, you can legally build a product out of this, but if you need something rock-solid to ship
00:04:56now, let it work out its issues, right?
00:04:59It's not quite there yet.
00:05:00But zoom out, though.
00:05:02Lama CPP didn't change everything because its first release was flawless.
00:05:06It wasn't.
00:05:07It changed everything because it took a complete Python mess, and it turned it into one fast
00:05:12local binary.
00:05:13And the whole ecosystem just piled on top of that.
00:05:16Audio CPP is making the exact same bet, but for sound.
00:05:20Local audio AI collapsing from a dozen fragile Python setups down to a single binary you own.
00:05:26That's the shift that we're kind of watching here, even while the binary's still rough.
00:05:31I've dropped the links for this in the description below.
00:05:33Head on down, check it out.
00:05:34Takes a little bit of time to set up, but it might be worth it.
00:05:37If you enjoy coding tips and tricks like this, be sure to subscribe to the BetterStack channel.
00:05:41We'll see you in another video.