Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

English
AAI Engineer
Computing/SoftwareInternet Technology

Transcript

00:00:00My name is Bo. I'm going to be here presenting Real-Time Voice Agents with Frontier
00:00:18Intelligence. Effectively going to be talking a little bit about how we at
00:00:22Elise AI architected our voice agent harness to get real-time voice with the
00:00:28frontier level of intelligence that we need. So before I start, I think I wanted to
00:00:36kind of draw some parallels about why we decided to go with cascaded voice agents
00:00:42and especially kind of comparing that to self-driving cars which I was working in
00:00:48before. So to me cascaded voice agents makes a lot of sense when you view it in
00:00:53lens of kind of breaking it down into perception which is for self-driving cars
00:00:59it's you know the bounding boxes, the camera, the lidar. For voice it's going to be the
00:01:05transcription basically effectively turning these like signals from the real world
00:01:09into elements of data that the language model or whatever brain you are working on can
00:01:17process. Second one is the planning stack which is pretty straightforward.
00:01:23This is where the language model will take in the outputs from the perception stage and produce
00:01:29the outputs that you want to produce out back out into the real world. And finally there's the controls
00:01:36layer where in self-driving you'll be taking the trajectory that the planner would output and kind of
00:01:43turn it into the real controls to kind of build like drive the car. Here we're turning the text into
00:01:50audio that we use to express our voice agents thoughts. And yeah so here I'll kind of be like kind of diving into
00:02:00each one of these elements and we've made a few kind of interesting tricks on each of these areas to
00:02:08improve the speed of our voice agents without sacrificing the intelligence.
00:02:13So the first one is going to be the transcriber layer. So we came up with this concept called like
00:02:19the streaming speculative transcriber where effectively we are layering a fast streaming
00:02:24transcriber like flux on top of or kind of below a scribe v2 or a accurate batch transcription
00:02:33which kind of takes in more context. It's a little bit slower but it will give you more accurate detections.
00:02:39So kind of walk through a setting now so in this in this case the agent just asked you know providing
00:02:44can you write your name and date of birth and the user is going to say this and we'll see how that plays
00:02:49out timing wise. So first we're going to get you know the shore detection. We'll get it from we'll get it
00:02:57from the streaming layer. The accurate layer the corrective layer is not going to fire because it's the same text.
00:03:05We're going to get some more streaming text detections and in this case the corrective layer is actually
00:03:11cancelled because we got new new text so you know more context more audio is going to beat the old accurate one.
00:03:21And here's where kind of the first correction comes in. So because the scribe v2 layer understands
00:03:27you know the context of the question is able to understand that this is talking about a name and this is a date of birth.
00:03:34And then a couple more detections. These are just punctuation. We don't care.
00:03:37And so in the end we kind of release this text over to the agent.
00:03:44And moving on to the language model there. So here since we're kind of using these slow but
00:03:52intelligent LLMs we really want to reduce the number of round trips and the thing that causes us to do a
00:03:58lot of inferences is tool calling. So one way to get rid of that is by having background agents do the tool
00:04:05calling for you and kind of push the tools back into the context of the main agent so that it thinks it
00:04:13made the tool call but it really didn't. So remember from like detection from before. So what will happen is each one of these detections is going to trigger
00:04:26an early kind of generation of the agent and we but we won't actually admit this out until we're confirming that
00:04:38the user has finished speaking. So in this case the user says sure. The agent kind of knows that the user's about to say something else.
00:04:45Our background tool calling here which is going to be helping us figure out the name and the date of birth from the user detection
00:04:51is not firing so nothing much there. The next instant detection comes in. It says that you know still not really a name.
00:05:02Our agent kind of plays along and continues there.
00:05:06Now kind of more context comes back. The agent kind of feels like there should be a name.
00:05:11It's going to ask to spell it out because it's probably thinking there's some transcription error here.
00:05:16Still no name or date of birth. And then finally this you remember this is kind of our corrected
00:05:22final instant detection from the transcriber from the scribe v2.
00:05:27Here our eager kind of agent generation that was made without any tool calls is going to get cancelled
00:05:34because the background agent finally is able to find the name and date of birth it's looking for.
00:05:38So it's going to re-trigger and now the agent actually has the context it needs.
00:05:45And you see here it's kind of we're doing it. The tool call here is a little bit
00:05:49some intelligence there. We're going to be like you know correcting mistranscriptions of name,
00:05:53kind of doing some like phonetic matching here.
00:05:57And yeah and then we'll kind of once we've understood that this is the end of the user utterance,
00:06:02we'll kind of emit it out. So pretty standard.
00:06:06Okay and then the next layer here is going to be text-to-speech. So with text-to-speech the goal
00:06:12is to kind of take what the agent said and the agent's going to be admitting this in a streaming
00:06:16fashion. So we're going to need to produce audio as quickly as possible. And ideally what you can do is
00:06:24before the agent has even finished generating the full text you can have the audio play. So it's kind of
00:06:31hiding the latency of finishing the generation. So I'm going to kind of play the streaming
00:06:39the streaming agent output now. So it starts with U. And actually before I dive further there's
00:06:46this new concept that we're introducing here called the prefix cache. So the prefix cache is going to be
00:06:52looking at the agent stream and seeing if we already have generated audio for that sequence of words
00:07:02from like a prior generation or maybe like the same generation in this call as well.
00:07:10So it sees the word U. We for this prefix cache we're going to be you know we don't want to like
00:07:17immediately hit on every single word. We're going to be waiting for a little bit more words. So after
00:07:24three words the prefix cache gets our first hit. And over here on the right this is kind of our
00:07:31text-to-speech standard provider you know Cartesia is a text-to-speech engine with WebSocket support.
00:07:36So we're piping the agent through the cache and also piping it through WebSocket.
00:07:45More tokens come in, more cache, more sending to the WebSocket. Not much to say here.
00:07:51And okay so now we get our first kind of first unique thing which is we found a token that actually
00:07:59causes a cache miss. It makes sense if we're kind of caching previous generations. You said your name
00:08:05is is a pretty common thing but once we add in the name suddenly that's going to result on the cache miss.
00:08:12At this point we're actually going to yield out our cached audio. So you said your name is is going to be
00:08:19emitted as the rest of the streaming text is coming back. So at this point the user hears the agent.
00:08:25The user doesn't really know what's going on. It just looks like really fast response times to them.
00:08:32And now the kind of remaining text flows through. And at this point we've already admitted from the cache.
00:08:38The cache has done its job. The rest we can kind of throw into Cartesia. And here's kind of the trick
00:08:46where Cartesia has seen the entire transcript up to this point. To Cartesia like it doesn't know
00:08:56about the existence of this prefix cache. It's just going to generate this full sentence with you know
00:09:01standard natural prosody. But what we do is when the generation comes back since we've already played
00:09:08the audio here we can actually suppress the audio from Cartesia here. And just play out the remaining
00:09:13stuff. So the user there's might be a tiny bit of a hiccup. You know I'll play some audio later and you'll
00:09:21know that you probably won't be able to notice. And effectively we're kind of taking this and
00:09:28or taking this part and just emitting it directly after this audio has finished playing. So it looks seamless to the user.
00:09:35And yeah so I guess at this point kind of this is a lot of technical details. I'll think the best way is
00:09:43just kind of play a call and see this in action. So here I recorded a call and let's just hear what happens.
00:09:53So here we are going to talk to you guys. This is Bo, OB-GYN. Elise speaking.
00:09:59Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm.
00:10:03That's exciting news. Can you provide your name and date of birth?
00:10:08Sure, it's Elise trial. Date of birth, 2303.
00:10:15Thank you. Are you a new patient with us?
00:10:24Yes. Welcome to Bo, OB-GYN. Can I text you a link to upload your insurance information?
00:10:34Sure. Okay. Sent. Let me know when you've received it.
00:10:48I've received your insurance information. The earliest appointment I have is Thursday,
00:10:53July 2nd at 10 a.m. Actually, give me a second. Let me check my calendar.
00:11:01Sure. Take your time. Do you have something for next week?
00:11:09For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on Tuesday,
00:11:14July 7th at 2 p.m. or 3 p.m. Do either of those work for you?
00:11:20Yeah, 2 p.m. works. Great. Your appointment has been booked. We look forward to seeing you then.
00:11:29Thanks. Bye-bye. All right. Yeah, that's pretty much it.
00:11:35Yeah, you can kind of see our -- all this like streaming and, you know, a lot of things are
00:11:43happening in the background. And, yeah, this is what really makes, like, voice agents interesting.
00:11:48And there's a lot of effort that can be done in the harness to really kind of get a
00:11:55natural conversation, which is what we're after.
00:11:59Okay. Yeah. So, I guess, briefly, you know, in the last part, I want to just talk a little bit
00:12:04about Elyse. So, I think Elyse, you know, our headquarters are in New York, and kind of we're
00:12:09trying to expand our presence here in the Bay Area. We've -- I think it's maybe like a different style
00:12:15of company that I think people are -- like, think of when they think about AI startups in San Francisco,
00:12:23where we're actually very focused on just, like, helping people and helping people where they need
00:12:31it, like kind of the life's most critical areas. We work on housing, healthcare, and we're doing really
00:12:37well, and, you know, here's -- there's a link here to kind of join our team, and there's going to -- we're
00:12:45going to be posting a lot on Twitter, so you can follow us @Elyse.ai as well. Yeah, that's it.
00:12:57So, we're going to be posting a little bit more on Twitter, and we're going to be posting a little bit more on Twitter.

Key Takeaway

Combining streaming speculative transcription, background tool calling, and prefix-cached text-to-speech synthesis enables real-time conversational voice agents with frontier-level intelligence.

Highlights

  • Cascaded voice agent architecture breaks down processing into perception, planning, and control layers.

  • Streaming speculative transcribers layer a fast streaming transcriber like Flux beneath an accurate batch transcriber like Scribe v2.

  • Background agents handle tool calling asynchronously to eliminate round-trip latency for main language model inferences.

  • Prefix caches match previously generated audio sequences to hide text-to-speech generation latency.

  • Audio synthesis pipelines suppress overlapping segments from text-to-speech providers like Cartesia to maintain seamless prosody.

Timeline

Voice Agent Architecture Parallels

  • Cascaded voice agent architectures mirror the perception, planning, and controls structure of self-driving cars.
  • Perception translates real-world audio signals into processable data elements through transcription.
  • Planning takes perception outputs via a language model to determine appropriate responses.
  • Controls convert generated text outputs into audio to express agent thoughts.

Voice agents require a modular design similar to autonomous vehicles to balance low latency with advanced reasoning. The perception stack handles transcription, the planning stack runs the language model, and the controls stack generates the audio output. Special optimization techniques applied across each layer improve speed without reducing intelligence.

Streaming Speculative Transcriber Layer

  • Streaming speculative transcribers layer fast streaming models underneath accurate batch models.
  • Flux provides fast initial text detections while Scribe v2 adds corrective context-aware transcriptions.
  • Corrective layers cancel out older streaming outputs when superior context or complete audio arrives.

Transcription speed determines overall response latency in voice agents. Pairing a fast streaming model with a slower, highly accurate batch transcriber resolves mistranscriptions and punctuation errors on the fly. Contextual understanding allows the system to correctly identify names and dates of birth despite initial audio ambiguity.

Language Model and Background Tool Calling

  • Background agents execute tool calls asynchronously to prevent inference round-trip delays in the main agent.
  • Eager agent generation runs continuously during user utterances without waiting for explicit tool execution.
  • Eager generations cancel and re-trigger once background tool calling resolves specific data like names and dates of birth.

Tool calling introduces significant latency when using intelligent large language models. Offloading tool execution to background agents allows the main agent to maintain a continuous conversational stream. Phonetic matching and mistranscription corrections occur inside these background processes before final text emission.

Prefix Caching Text-to-Speech Optimization

  • Prefix caches check agent streams for previously generated audio sequences to bypass full synthesis.
  • Cached audio emits immediately while remaining text streams through text-to-speech engines like Cartesia.
  • Overlapping audio from external synthesis engines is suppressed to maintain natural prosody and seamless playback.

Text-to-speech latency drops significantly by caching frequently used phrases and word sequences. When a cache hit occurs, pre-rendered audio plays instantly while new text routes to external WebSocket providers. Suppressing duplicate audio segments ensures the user experiences zero perceptible hiccups during playback.

Demonstration Call and Company Overview

  • Demonstration recordings showcase real-time voice agent interactions handling appointment scheduling and insurance uploads.
  • EliseAI headquarters are located in New York with expanding operations in the San Francisco Bay Area.
  • Application domains focus on high-impact sectors including healthcare and housing.

Live call demonstrations prove the effectiveness of streaming transcription, background tool execution, and cached text-to-speech pipelines. The system successfully books ultrasound appointments and requests insurance information with minimal conversational delay. Development efforts center on building practical AI solutions for critical daily life infrastructure.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video