Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Nelsi Ghidour: Hi everyone. I'm Nelsi Ghidour, co-founder and CEO of Gradium. Gradium is a
00:00:19startup based in Paris. Most of our background is from research. In particular, we have invented
00:00:26algorithms such as speech-to-speech models, codecs, and so on. Basically, we started from
00:00:36a research project called Qtai, a non-profit research lab that has been focusing on voice
00:00:41since day one. In particular, we released in 2024 the first full duplex speech-to-speech
00:00:46model called Moshi, the first real-time speech-to-speech translation system called Hibiki, and the
00:00:51first TTS models that can run locally on a smartphone. Basically, I will just say a few
00:00:58words about what we do, but we are a model company that trains models for building voice
00:01:03agents and voice applications, so we do TTS, API and on-device, speech-to-text, speech-to-speech
00:01:08translation, and much more to come. What we do is that we train foundation models for audio
00:01:13and then we can apply them for a lot of different tasks. So, I'll go quickly on voice agents because
00:01:19it's a fourth talk about the topic, but basically now we have these voice interfaces that we can
00:01:27use to do a lot of things across a variety of products and types of interactions with NPCs,
00:01:32with customer agents, language learners, coach, and so on and so forth. And in this talk, I try to go
00:01:39through the history of this technology and where I see it going in the next years. And maybe to start,
00:01:47I think we can take a look at the announcement of Siri back in 2011. And you'll see that it's actually,
00:01:55you know, I think it aged pretty well. What is the weather like today?
00:02:06Here's the forecast for today. It is that easy.
00:02:13Lots of things. We've integrated with the stocks, so you can ask it about the stock market. Something like,
00:02:18how is the NASDAQ doing today? NASDAQ composite is down right now at 2321.70.
00:02:29Again, you can ask this from the lock screen anywhere. Just press the button and ask. You can ask about,
00:02:32you know, the NASDAQ, the DAO. So, what you just saw is was kind of a voice agent. It was a bit
00:02:39constrained, but it was technically a voice agent. And the architecture behind it, so you have seen a
00:02:43thousand times today the STT, LLM, TTS. Back then, it was even worse, right? So, there was no LLM,
00:02:49obviously. So, there was what was called natural language understanding. So, you will go from the
00:02:54transcript and try to do basic classification of what is expected, what is the app that is supposed to
00:03:01be controlled, what is the action to trigger, and so on. So, it was a very complex pipeline and
00:03:08very constrained to very specific use cases. So, it was what was called close-ended dialogue back then.
00:03:18Then, the next generation of models, I could say, I think, was illustrated pretty well by the
00:03:25OpenAI voice mode, not the advanced voice mode, the original one. In which case, you just have
00:03:31a conversational experience that is open-ended. So, it's much less constrained than Siri. You can talk
00:03:35about pretty much anything. At the same time, you don't have the same agent capabilities, let's say,
00:03:42in the sense that it cannot really give you the NASDAQ or give you the weather.
00:03:50We want to hear a bedtime story. Tell us a story about the super-duper sunflower hedgehog named Larry.
00:03:57Start with telling us a little bit about him.
00:04:03Larry was a unique hedgehog unlike any other. He had bright sunflower petals instead of spines.
00:04:09Residing in Meadowville, Larry spread joy and color wherever he rolled.
00:04:15So, in that context, basically, that was, you know, kind of second generation of models where
00:04:19all the logic, the very complex logic was replaced by an LLM. And if we compare Siri and the voice mode from
00:04:28is open AI, not the advanced one, the original one, the big strength is open-ended dialogue but the
00:04:36advantage capabilities were inexistent. The naturalness was obviously much better than with Siri.
00:04:41The latency was still pretty bad. So, we are talking about several seconds and that was documented in the
00:04:46previous presentation. And so then, you know, we're in the third phase, which is a voice, what we call voice agents.
00:04:55And here, I'm going to show a quick demo of a drive-through bot vibe coded with our with our voice models.
00:05:02Hi there. What can I get started for you today?
00:05:05Hi, I'm actually in a hurry and I'd like to order a chicken sandwich quickly.
00:05:10We have the classic, spicy, deluxe or grilled chicken sandwich. Which one would you like?
00:05:16Uh, let's go with the classic chicken sandwich.
00:05:19Anything else for you?
00:05:20Uh, what else do you have in the menu?
00:05:23We have sandwiches, sides, drinks and desserts. What can I get for you?
00:05:29Can I get some mac and cheese maybe?
00:05:31You have a classic chicken sandwich and mac and cheese. Your total comes to eight.
00:05:37So, now, you know, unlike what we just saw, it's an actual agent. It's taking actions. It's
00:05:43keeping track of the order. It's then going to make you pay. So, it's an actual voice agent that can
00:05:48do real tasks. So, here, instead of having an LLM that is just a conversational interface,
00:05:54we have a real agent that is empowered with tool call, reasoning, planning and all this stuff. So,
00:06:02what we see now is we have gained back agenting capabilities and actually they are much more
00:06:09powerful and generic than before while keeping a very good level of naturalness and that's where
00:06:17speech-to-speech LLM came. In particular, what we could see here is the latency is better with cascaded
00:06:24system, but it's still higher than you will have the human conversation. And as also as explained before,
00:06:30the naturalness is fundamentally limited by the fact that you go through text. So, you lose a lot of
00:06:35information about what is said, the tone, the emotion of the user and so on and so forth. So, now that we
00:06:41have tackled intelligence and agenting capabilities, speech-to-speech seems like a natural next step for
00:06:46naturalness and latency. And so, here, it's an announcement from the OpenAI advanced voice mode.
00:06:53Hey, ChatGPT. I'm Mark. How are you?
00:06:56Oh, Mark. I'm doing great. Thanks for asking. How about you?
00:07:01Hey, so I'm on stage right now. I'm doing a live demo and frankly, I'm feeling a little bit nervous.
00:07:06Can you help me calm my nerves a little bit?
00:07:09Oh, you're doing a live demo right now? That's awesome.
00:07:14I think we all remember it was very impressive. Very impressive release.
00:07:18And in that context now, all the steps of STT, LLM and TTS have been absorbed into a single one.
00:07:26And so, now, intelligent, you know, like naturalness is
00:07:32still very good. Actually, it can be better because it can understand non-linguistic information.
00:07:37Latency is really, really nice. Honestly, it doesn't make sense to go better than that.
00:07:44Interestingly, and everyone who has used any speech-to-speech model can
00:07:48attest that the intelligence is still much more limited in that context than the cascaded counterpart.
00:07:57So, the speech-to-speech models are fundamentally still limited compared to the textual models.
00:08:03Another limitation is turn-taking. So,
00:08:07people tend to mix speech-to-speech and full duplex. But basically,
00:08:13when you do have a speech-to-speech model like GPT real-time, it's still based on fundamental
00:08:17turn-taking. In the sense that it's going to segment the conversation into either the model is
00:08:21speaking or the model is listening. And to give, to show you how this can make an interaction
00:08:27unnatural, I'm going to show a small demo with what is called backchanneling, which is these very human
00:08:32things that you do when someone talks to you is that you say, hmm, yeah, and so on.
00:08:37Hey, how's it going? I'd just like to brainstorm a bit about it with you.
00:08:40Oh, that's a great topic. Yeah, I'd love to help you brainstorm. Are you thinking more?
00:08:47Exactly. Yeah, I was saying. No, no, I didn't mean to interrupt. You know, I was just saying,
00:08:50yeah, like that. You can just keep going. You know, don't mind me. That's something I typically do.
00:08:55Ah, no worries at all. Yeah, I was just going to say we could break it down into a few aspects.
00:09:00Sure. Yeah, exactly. No, no, please stop interrupting. You know,
00:09:03it's called backchanneling. Humans do it all the time. It shows that you're just following the
00:09:07conversation. That shouldn't, you know, like interrupt you in your flow. Just,
00:09:10just going. Ah, got it. Thanks for letting me know. No problem. Oh, come on. Yeah, so you see,
00:09:17you know, it's still very annoying. You can have lightning speed latency. Fundamentally, this is
00:09:25an issue that can not be resolved when you're using turn-taking. So here, that's the walkie-talkie.
00:09:31Any real-time model today, I mean, now there is a bi-directional one that will come from OpenAI,
00:09:36but it's called half-duplex. So the model is listening or speaking. A human conversation
00:09:42has a constant flow between two people. People do backchanneling. People interrupt one another,
00:09:48talk on one another, and so on. If you're having a relative on the phone, there is up to 20% of the
00:09:54time where you are both speaking at the same time. And that makes, you know, this very flexible dynamics
00:10:00in the conversation makes it much more comfortable for humans. And so to understand how we can make a
00:10:07model full duplex, I'll give a very short presentation of how we train such models.
00:10:14So the way you create a speech-to-speech model, half-duplex or full duplex, is the following one.
00:10:18So you start from a text LLM, which is probabilistic models over words. And instead of predicting the
00:10:24next word based on the past, what you want to do is rather predict the next audio based on the past
00:10:30audio. The issue now is that if you pass a raw audio to your model, which is, you know, a waveform,
00:10:36it's air pressure variations, basically you take this sentence, it's eight words. It takes around
00:10:45three seconds to pronounce it. And so at 24 kHz audio, instead of having eight words, the audio form
00:10:50is 72,000 time steps that you will need to feed to your LLM. Given that LLMs have quadratic complexity
00:10:57with sequence length, so the complexity is a square of the sequence length, a 10,000 times longer sequence
00:11:03is 100 million times more expensive to process. So there is no way you can train an LLM on a raw audio.
00:11:08So the way you addressed it is by creating neural codecs, or you can also call them audio tokenizers.
00:11:14And basically it's an encoder that takes an audio and compresses it in a very dense compressed
00:11:20representation, a bit similar to text. And then you have a decoder that can reconstruct high quality
00:11:25audio from it. So now you have gone from the audio domain into an abstract representation domain
00:11:31where you can train an LLM exactly like you will train it on text. And the speech-to-speech model
00:11:36from Japan AI that was shown before works in this fashion. So instead of having text tokens into your
00:11:42model, you have audio tokens that represent either the LLM or the user and you put them one after the
00:11:48other and the model predicts the audio tokens that should be saved by the model being given the context
00:11:54from both sides on the conversation. However, you can see that it's still a sequence between user
00:11:59and system which is still half duplex. So how did we made the first full duplex model ever?
00:12:06Very simple. We call it multi-stream language models. That's the technology now used also by thinking
00:12:11machines for their interaction model and most likely by the directional model of OpenAI is that instead of
00:12:18having a transformer that models one sequence of tokens, it models two of them so that both
00:12:24parties can be active at the same time, inactive at the same time, one active and one inactive.
00:12:29And I just show a very quick demo of how it sounds like but that's the release of Moshi in August
00:12:362024 where we did announcement live on stage talking to it for the first time.
00:12:41And you'll see that the model often guesses the end of the question and swerves over the speaker and both
00:12:48speaking at the same time is not breaking the flow like we saw with GPT. The whole thing is just
00:12:54extremely resilient to the most chaotic situations. So the planet is Sirius 22. Can you plot a trajectory
00:13:01course to it, please? Yes, sir. Okay. How long is it going to take us to get there? I've mapped it out.
00:13:06Okay. It's approximately five months to get there. Okay. That's not too bad. Do you think we have all
00:13:12we need on board the ship to start the mission? Yes, sir. We have everything we need. So back then
00:13:17it was even a bit irritating to people because it was interrupting you all the time. But the thing is
00:13:21that you can use, you could still use it in extremely noisy environments with a lot of noise, people
00:13:26coughing and so on and you know the flow is just constant. You don't get this very irritating break of the
00:13:32conversational flow. So these full duplex models, they are the highest level of naturalness you can
00:13:38expect. That's the same conversation we have a human. The thing is with our models, it was even more
00:13:45stupid than speech-to-speech models that were already less intelligent than cascaded systems. It's probably
00:13:50fine for some use cases if you just want to have a chit-chat, you know, the model doesn't need to be very
00:13:55intelligent. But if you want to make an actual full duplex voice agent, there is no way we can give up on
00:14:00intelligence just to gain speech-to-speech abilities. So how do we finally make models that tackle all
00:14:06these aspects jointly? And I think interestingly, if you look at the history I showed, there is a tension
00:14:13between naturalness and intelligence. So every time we improve naturalness or humanness of the models,
00:14:19they were less intelligent than the cascaded system. The cascaded agents, they are basically as smart as
00:14:24the best text models. So if you have a voice agent that is powered by the latest model from
00:14:29Entropic or OpenAI, it's going to be extremely smart. I have all the same reliability for tool
00:14:32call and so on. Speech-to-speech has this naturalness aspect. However, you give up intelligence to get
00:14:39that. And the reason why you give up intelligence is remember that LLM is a model that has a certain
00:14:45number of weights that we call the capacity. And if you take a text model and now not only has to
00:14:52handle text but it also needs to understand speech and produce speech, it's taking some of its capacity
00:14:57and this capacity now is taken from the intelligence. So fundamentally, there is a cost of adding a new
00:15:04modality to a text model that is going to be paid in intelligence. So where do we go from here? There are
00:15:10two paths that are in front of us and both are going to be explored at the same time. The first one is
00:15:15scaling the model. So making your speech-to-speech model bigger, better pre-trained, better post-trained and
00:15:22so on, will likely progressively increase its intelligence until it's good enough for a lot of use cases.
00:15:29The second one is splitting the model between naturalness and intelligence. The first one, I'm not at OpenAI,
00:15:37so I don't know because they don't publish their model. I guess OpenAI is the path one. So it's a
00:15:42frontier text LLM with a lot of science around post-training, instruct tuning to fine-tune it on audio
00:15:48and teaching it to be quite smart while using audio. The nice thing about that is you have a single model
00:15:55to orchestrate, so it's quite easy to deploy. And one big aspect, however, is that it's an extremely
00:16:06complex and costly process to go from the text model to this speech-to-speech model. The second path
00:16:12is to split it. It's an approach that we introduced in one of our recent papers called Mochirag and that
00:16:16has been reused by, in particular, the thinking machine interaction models, where basically the
00:16:21idea is that now you have two models. The first one is a small, maybe even on device, full duplex,
00:16:28extremely natural speech-to-speech interface. And its only role is to keep a very natural conversation
00:16:35and be able to delegate all the thinking, tool calling, reasoning, agenting capabilities to a
00:16:42background text model. And so the way to see it is you have a background text LLM that receives
00:16:48asynchronously queries from hundreds to thousands of small voice interfaces and just gives them their
00:16:53text, you know. And basically what we did is a very small full duplex model that just needs to know
00:17:00when it doesn't know so that it can delegate to the background model. And the reason why we believe
00:17:05mostly in this approach, and I go back to it later, it's our, let's say our culture is more the first
00:17:16one, the bitter lesson. So every time we've been pushing for end-to-end systems and so on. But now
00:17:21I think the hybrid approach has two main advantages. The first one is cost. So speech-to-speech models are
00:17:28notoriously quite expensive. And when you think about it, it's a loss of money to do chit chat with
00:17:35gigantic speech-to-speech models that can resolve differential equations and so on. So it doesn't
00:17:41really make sense economically to get all your workflow through this gigantic multimodal mixture of
00:17:47experts. At the same time, we see that people are very attached to their ability to control the
00:17:52backend to be able to switch SONET 5 was released a few minutes ago. People want to switch the backend
00:17:57and the intelligence and get a lot of optionality on that, right? When you're using a speech-to-speech model, your
00:18:05hands are a bit tied with this model provider. And to give you an idea of that, until recently, the
00:18:10advanced voice mode from VNI was powered by GPT 4.0, despite the fact that there have been several
00:18:15generations of the text model since then, because this process is so expensive and so long. For this
00:18:22reason, we rather bet on the hybrid approach, because that will give something that is not only very
00:18:28natural and very nice for demos and impressive, but also will be a viable alternative from an economic
00:18:34point of view and agentic capabilities point of view to the best cascaded systems that are still most of
00:18:41the market today in voice. So what now? You can use our models on Gradium.ai. You can apply to Gradium. We
00:18:49are recruiting research scientists and engineers, and thanks for your attention.
00:19:13Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video