How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit

Highlights

• General leaderboards fail to predict real-world voice agent performance because they score individual components in isolation rather than evaluating an entire conversation stack.

• LiveKit's turn detector uses audio rhythm, pitch, and intonation alongside transcripts, cutting unwanted user interruptions down to 4.5% compared to 9.9% for leading alternatives.

• Time to first byte and time to first token metrics are misleading because audio generation requires complete sentences, making time to first audio and time to first sentence the correct measurements.

• Decoupling conversation threads from backend tool execution allows voice agents to keep talking or ask follow-up questions while database lookups or payments process in the background.

• Running automated end-to-end simulations with test callers, background noise, and varied interruptions reveals which model stacks actually succeed for specific domain use cases.

Timeline

1. 00:00 - 04:20

   The Reality of Real-Time Voice Agents

   Everyday interactions with automated phone lines often break down when callers interrupt or talk over the system, frequently driving users to press zero for a human operator. Background tools like code generators tolerate long delays, but real-time conversational agents require instantaneous and natural responsiveness. Voice represents the original and most intuitive modality for human-computer interaction, as highlighted by frameworks like LiveKit powering advanced voice modes for major platforms.

   • Traditional asynchronous background AI tools can tolerate delays, but real-time voice agents fail immediately when users speak over them or interrupt.
   • High-stakes domains like financial services demand flawless handling of shifting variables such as payment dates and amounts.
   • Voice serves as the most natural modality and the primary entry point for aligning artificial general intelligence with human intent.
2. 04:20 - 06:50

   Why Standard Leaderboards Fail Voice Applications

   Picking models purely based on public leaderboards fails because those rankings score isolated components rather than the complete conversational experience. A minor transcription error by a speech-to-text model early in a transaction alters the downstream large language model context silently. Furthermore, benchmark datasets use pristine studio audio rather than the noisy, imperfect environments callers actually use.

   • Developers traditionally choose models using top leaderboard rankings without accounting for how individual components interact in a live conversation.
   • Cascaded pipelines consisting of speech-to-text, large language models, and text-to-speech models compound transcription errors that cascade downhill.
   • Leaderboards test models on clean audio, ignoring real-world conditions like speakerphones, background traffic, and children.
3. 06:50 - 09:29

   Correcting Metrics: Latency, Audio, and Tokens

   Provider dashboards often quote extremely fast time to first byte measurements, but actual audio output frequently trails behind with hidden pauses. Similarly, large language model token generation speeds do not equate to audio delivery because speech synthesis requires a complete sentence structure. Maintaining a tight gap between the caller's last syllable and the agent's first audible response preserves the natural flow of conversation.

   • Dashboard metrics reporting time to first byte can still hide up to three-quarters of a second of subsequent silence.
   • Text-to-speech models require an entire sentence rather than raw tokens to begin speaking, making time to first sentence the necessary metric.
   • Human conversation relies on natural cadence and pacing within a finite latency window where delays under six hundred milliseconds feel genuinely human.
4. 09:29 - 13:37

   Multitasking and Asynchronous Tool Execution

   When an agent initiates a backend task like checking a payment status, dead silence mimics a system freeze and breaks immersion. Humans naturally multitask or fill dead air by narrating their actions, and voice agents can replicate this behavior using asynchronous programming patterns. A single async await line separates the conversation thread from the tool execution, enabling the agent to keep talking or request additional details while data saves in the background.

   • Backend database lookups and transaction checks introduce unavoidable latency that agents must actively manage rather than handling through dead silence.
   • Asynchronous awaiting of tool calls decouples the conversation thread from backend processing, handing the microphone back to the agent immediately.
   • Demonstrated hotel booking workflows prove that agents can collect user data continuously while backend reservations save in parallel.
5. 13:37 - 15:02

   Turn Detection and Conversational Rhythm

   Determining when a caller finishes speaking poses a major design hurdle for conversational agents. Traditional transcript-based models mistake brief pauses inside numerical amounts or complex statements for end-of-turn signals. Incorporating audio features like pitch and intonation allows the system to recognize unfinished thoughts and wait for the user to complete their sentence.

   • Standard turn detectors rely solely on transcripts and punctuation, frequently cutting off users prematurely during pauses within numbers or amounts.
   • Advanced turn detection evaluates audio rhythm, pitch, and intonation to determine whether a speaker has genuinely finished talking.
   • LiveKit evaluation data shows audio-aware turn detection reduces unwanted user interruptions from 9.9% down to 4.5%.
6. 15:02 - 20:47

   Building a Custom Evaluation Framework

   Determining the ideal model stack requires testing complete, working agents with simulated callers that execute real scenarios, alter inputs, and introduce background noise. Developers can execute both text-based simulations for business logic and audio-based simulations for speech recognition pipelines. By continuously swapping model stacks under identical input conditions, teams build personalized benchmarks that reveal which configurations deliver the fastest and most reliable results for their specific use case.

   • Optimal model stacks are discovered by running end-to-end conversational simulations rather than relying on generic leaderboard scores.
   • Automated simulation testing pits complete agent implementations against virtual callers that introduce interruptions, background noise, and goal changes.
   • Empirical testing of the hotel receptionist agent demonstrated that inference-optimized serving significantly accelerates specific models like Gemma 3 31B.