This Is What Voice Coding Was Supposed to Be (Qwen Audio Agent)
BBetter Stack
Computing/SoftwareInternet Technology
Transcript
00:00:00Voice coding falls apart the moment the agent has to do actual work.
00:00:04Refactor this file, run the build, and then silence.
00:00:08The agent disappears and we're left staring at the mic icon waiting for it to come back.
00:00:14Quen Audio Agent works differently.
00:00:16Claude code can be refactoring the background while you keep talking to the voice agent.
00:00:20And when the task finishes, it can interrupt you and tell you it's done.
00:00:24And here's the weird part.
00:00:25Quen Audio Agent isn't a model, it ships zero model weights.
00:00:29It's a runtime built around one thing most voice coding setups completely miss.
00:00:34Keeping the conversation alive while the actual work happens.
00:00:42Now, let me set up why that matters.
00:00:45Because I think most people have the wrong idea about voice and coding.
00:00:49The usual setup is dictation.
00:00:51Whisper listens, transcribes, pastes text into your editor.
00:00:55That's useful, but look at what actually it replaces.
00:00:58It replaces the whole typing part.
00:01:01You sit there, you still wait.
00:01:03You've swapped the fingers for your mouth and changed really nothing.
00:01:06It does speed things up, though.
00:01:08Full duplex means something different.
00:01:10It means both sides can talk at the same time.
00:01:13You can cut in mid-sentence.
00:01:14And more importantly, the thing on the other end can keep the conversation alive,
00:01:19while work happens somewhere else.
00:01:21The second half is the whole reason this exists.
00:01:24If you enjoy coding tools that speed up your workflow, be sure to subscribe.
00:01:27We have videos coming out all the time.
00:01:29All right, now let me show you why this is different.
00:01:32I'm going to make it talk for way too long on purpose.
00:01:35Not going to lie, the setup was actually really frustrating.
00:01:38It was all in Chinese, so I had to go to the Alibaba Cloud English site, set the desktop
00:01:43UI for front-end and back-end worker with the Alibaba API key, and then add some anthropic
00:01:49API credits for the back-end.
00:01:51I did have to translate some of that.
00:01:52Once it's all synced, though, it does work pretty good.
00:01:55I can just speak without pressing anything.
00:01:57It's always listening unless I turn off the mic.
00:02:00So here we go.
00:02:02I'm going to give you a code repo that I need you to work on.
00:02:05It starts explaining that.
00:02:07Okay.
00:02:07And then halfway through, I'm just going to go here.
00:02:10Stop.
00:02:11Too long.
00:02:11I get it.
00:02:12Understood.
00:02:13And it cuts off just like that.
00:02:15It listened to this.
00:02:16Now it's all gone.
00:02:17I can refresh from here.
00:02:19It doesn't finish the sentence because I cut it off.
00:02:21There's no extra second of audio fighting me after I start talking.
00:02:25Here, the runtime actually kills the old turn and moves on.
00:02:29I'll paste in the path to the repo that we are going to work on here.
00:02:34Now watch what happens with something simple.
00:02:37What's 19 times 24?
00:02:39Don't touch the repo.
00:02:41I got an immediate answer, though, because Claude Code never needed to see that.
00:02:44The voice agent can handle the easy stuff itself, which is great because I don't want every
00:02:49random question turning into a big task or racking up my API credits for Anthropic.
00:02:54But now I'm going to give it something that actually takes a bit more time.
00:02:58The code repo that I gave it is just TypeScript practice problems.
00:03:01I could have chose something harder, but let's just keep it straightforward for now.
00:03:05I'm going to say here, refactor record problem TS.
00:03:09Make sure it works correctly.
00:03:11Now watch this.
00:03:13The task leaves the voice agent, Claude Code picks it up, and this work card appears.
00:03:18It's the proof that Claude actually has the job now.
00:03:21And normally, this is where a voice coding setup could fall apart.
00:03:25You've asked it to do something real, so now you have to wait.
00:03:29Except here, I don't have to.
00:03:31I can say, also, can you go make me a normal JS file with an async function
00:03:36to handle HTTP requests?
00:03:40It answers and does both while they're both running side by side.
00:03:44Claude is still working.
00:03:46Now I'm going to fire out a random question.
00:03:48Also, what is the weather like in Dubai?
00:03:52Okay, I can read it from here.
00:03:53Thanks.
00:03:55Another answer, same conversation.
00:03:57And I can even ask about the task that is being worked on without
00:04:02having to wait for it.
00:04:04Hey, how's the refactor going?
00:04:05Don't wait for it to finish.
00:04:07Just the status.
00:04:11Now it can tell me Claude is editing, running tests, or still working.
00:04:14There are basically two things happening at once.
00:04:17I'm talking to one agent.
00:04:18Another agent is doing the work and neither one has to stop because
00:04:22the other one is busy, which means while Claude finishes, I can just keep going.
00:04:27The coding task finished.
00:04:28The result came back into the live conversation and I didn't have to check anything.
00:04:33I haven't seen an open source desktop runtime handle that this cleanly.
00:04:38So underneath all this, the setup is actually pretty simple.
00:04:41On one side, we've got the voice agent.
00:04:43Okay, great.
00:04:44That handles conversation, interruptions, status canceling, a lot of the fast stuff.
00:04:50On the other side, Claude code is connected over ACP.
00:04:54That's the side touching files, running tools, and doing the slow work.
00:04:58Now, mechanically, it's really two lanes here.
00:05:00And once you see that, this design is a bit more obvious.
00:05:03There's a lightweight front end voice agent whose only job is to hold the conversation.
00:05:08It answers the easy questions immediately.
00:05:10What's the status?
00:05:11Cancel that.
00:05:11Nevermind.
00:05:13Then there's your real coding agent on the back end, and they talk over the agent client protocol, the ACP.
00:05:19Real work gets handed over as an async task and runs on its own.
00:05:23When it finishes, the result gets injected back into the live conversation.
00:05:27So the voice layer never blocks on the slow thing.
00:05:31And because it's a protocol rather than an integration, the back end is swappable.
00:05:36Claude code, codex, open code, and a bunch of others.
00:05:40Now, one detail I want to flag because it shows someone was actually paying attention to this.
00:05:44Interruption isn't handled as an event here.
00:05:47It's a state machine.
00:05:49Detect user speech, mark an interruption, emit an interpreted signal.
00:05:52Actively surpass any audio and transcripts still in flight.
00:05:56Then start the new term.
00:05:57Their own design note says the engineering quality of interruption determines the reputation.
00:06:03And they're right.
00:06:04The thing that makes a voice assistant feel broken isn't slow responses.
00:06:08It's interrupting it and still hearing another second and a half of the former sentence.
00:06:13That kind of gap is actually rather annoying and a big gap.
00:06:16This has a slight one sometimes.
00:06:18Now, where does all this sit next to what we already know?
00:06:21Well, the OpenAI real-time API is the layer underneath this.
00:06:25It's the thing that does speech in, speech out.
00:06:28This runs on top and is provider-pluggable.
00:06:32PipeCat and LiveKit agents are frameworks.
00:06:34They give you the parts and you assemble the pipeline.
00:06:38This is assembled and pointed at a specific job.
00:06:41And to whisper to LLM to TTS pipeline you wrote yourself is the thing that a lot of us have.
00:06:47A lot of us build out.
00:06:48Now, let's get real here for a second because there are two things here that the readme will not lead with.
00:06:52First, the runtime is open source.
00:06:55The voice is not.
00:06:56The code is Apache 2.0.
00:06:58But the default path, the best sounding one, roots through Alibaba's Dash scope with a paid API key.
00:07:05You get some free credits, but it's still paid.
00:07:07The optimized real-time voice is cloud API only.
00:07:10There is no self-hosted version of the good audio generation.
00:07:14So, open source voice agent is only partially true here.
00:07:17There is a real escape hatch and I'll give them credit for documenting it.
00:07:21A fully local mode using a hugging face speech-to-text pipeline with an MLX backend on Apple Silicon running on MPS.
00:07:30In full local mode, you need no cloud key at all.
00:07:33It's a second Python install and the tune profile in the docs is Chinese only.
00:07:38Okay, so translate it.
00:07:39But the path exists and it's pointed at MPS specifically.
00:07:43Second thing, and this one matters more, there are no published latency numbers anywhere on any hardware I went looking.
00:07:51Okay, now a final quick points here.
00:07:53It's Chinese first.
00:07:54The wake word is this, which translates to this.
00:07:58The main design article explaining the full duplex architecture is Chinese only.
00:08:02The tuned voice profile is Chinese.
00:08:05The model underneath claims a hundred plus languages for recognition and a few dozen for speech.
00:08:10So, I had to translate the page to understand the gist of it when I was signing up for all this.
00:08:15Inversion 2.0 is already listed in development with the architecture being rewritten.
00:08:19And the stars are running ahead of the usage.
00:08:2223, 2400 stars here, but around 4000 NPM downloads a month.
00:08:26And on the latest release, 92 downloaded the Mac version build against 502 on Windows.
00:08:32Now, if you already work inside Cloud Code or Codex, which I assume most of you all do,
00:08:37and you want it to stop being chained to the keyboard while long tasks run,
00:08:41this is the only thing I've found doing the parallel task plus line conversation together,
00:08:46or at least the one that does it really well.
00:08:48Get the DMG.
00:08:50It's a signed universal build.
00:08:52No CUDA, no Python, nothing to compile.
00:08:54It takes about 10, maybe 15 minutes to get everything integrated.
00:08:57If you want a fully open, fully local voice stack, this isn't it.
00:09:00But the MLX path is a real starting point, and it's more than most projects offer.
00:09:05And the idea worth taking away whatever happens to this particular repo is,
00:09:09the thing that makes voice usable for real work isn't better transcription.
00:09:12Transcription got solved.
00:09:14It's whether the system can keep talking to you while it's busy.
00:09:17That's a concurrency problem, not a speech problem.
00:09:20And almost nobody is treating it like one.
00:09:22I'm Josh from BetterStack.
00:09:23If you enjoy coding tools like this, be sure to subscribe.
00:09:26We'll see you in another video.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video