"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語한국어PortuguêsРусский中文
AAI Engineer
컴퓨터/소프트웨어AI/미래기술

스크립트

00:00:00Okay, hello everyone. So, my name is Midam Kim. I am an ML engineer from ServiceNow and I'll be
00:00:25talking about a linguistic framework for VoiceAI. So, quick background on me, so you
00:00:37know where I'm coming from. Like I said, I'm an ML engineer at ServiceNow, but I'm
00:00:42also a researcher, lifelong researcher, of speech communication in the wild. So, my
00:00:48motto is doing linguistics and what I'm going to be doing today is to hand you
00:00:54that lens of linguistics. So, have you experienced VoiceAI failures? Yeah, like
00:01:05everyone. So, I'm gonna introduce an example that I experienced myself. So, the
00:01:15bot asked me, "Could you please spell your first name?" And then I slowly start to spell
00:01:22my name. Yes, it is M-I-D-A-M. And the bot says, "Confirming with you, is it M-I-D-A-N?" And then I say, "No, it is M-I-D-A-M."
00:01:40And then I get slightly annoyed, more annoyed, because my name is M-I-D-A-M, not M-I-D-A-M. And then it asks me about, "Now, what is your account number?" And then I start getting confused. What is that account number thing? And then I try to find information about that. So, I try to find information about that.
00:02:09So, which one? It must be, and I slowly start spelling the account number. So, it is A-X-4-5-1. And then I take time because I'm not used to reading this strange number. And then the bot cuts me off. And then it says, "I couldn't find your record." And then, without even
00:02:14trying, it asks me to repeat that again. "Could you please repeat that?" And then I get stressed, "I couldn't find your record." And then it says, "I couldn't find your record." And then, without even trying, it asks me to repeat that again. "Could you please repeat that?" And then I get stressed out. And then it says, "I couldn't find your record." And then,
00:02:21A, X, 4, 5, 1 and then I take time because I'm not used to reading this
00:02:30strange number and then the bot cuts me off and then says I couldn't find your
00:02:36record and then without even trying it asked me to repeat that again could you
00:02:42please repeat that and then I get super annoyed and then I could say can I talk
00:02:46to a person I just don't want to deal with you anymore so this is a very
00:02:51typical pattern of voice AI unfortunately at this point so I just
00:02:57want to navigate how we can solve this problem with linguistics so voice AI is
00:03:06booming but users are still often preferring human agents over voice
00:03:11agents how can we mitigate this issue but in the first place what are the
00:03:19actual problems so I think we can think about a fundamental framework to
00:03:25understand this end-to-end architecture of voice AI which is called linguistics so as
00:03:35all of us already know human communication is a joint activity like
00:03:41the thing that we're doing right now so I give you my sounds and words you hear
00:03:48them and then if it is a conversation you're gonna give me your sounds and your
00:03:53words and then this is going back and forth through interaction and then in this
00:04:02process we're continuously processing and updating our mental models so that's a
00:04:09joint activity for human communication and I would like to say in the voice AI
00:04:18human communication it also has to be a joint activity like this because that's the
00:04:25only thing that we know about human communication as a human being we have
00:04:30been evolving thousands of years as communicators and this is what we know
00:04:35so we expect the same thing to bots so let me go over the failure scene of my call
00:04:45with the voice agent in this framework so you see there's listen and speak for each
00:04:53party so I start spelling my first name and then the bot does not hear that the
00:05:04difference between M and N correctly so it's an SET failure in the listening level and then the TTS applies only
00:05:14English-centric reading rules to my name M-I-D-A-M would read as M-I-D-A-M in the American
00:05:22English version so I'm confused but at this time I'm kind of generous because that
00:05:29happens a lot so even with human beings so I'm okay but then when it brought brought
00:05:36up account number thing because I don't know what that is I'm confused again but I'm adaptive I can find
00:05:44I can look for it so I found the number start reading it but the STT did not recognize the word
00:05:52unit correctly so it cuts me off and finally it eventually talked over me so I get really irritated
00:06:06and then when it asked me for the repetition of the same information and then it is clear
00:06:14that the spot is not tracking the mental model with me and then very rudely it does not even try
00:06:22interactive clarification which is a common strategy by human beings so I don't want to deal with this
00:06:28anymore so I say can I talk to a person so let's go over the the framework again so these are the
00:06:38linguistic components that are expected and well maintained in human to human conversation
00:06:47so there are listening channels a listening channel and speaking channel and there are different components
00:06:52like sounds words interaction and mental model so the first component is does a bot recognize the user's
00:06:59speech well and all of these technical terms will fall under this and then there was there's going to be this
00:07:09second component which is words in the listening channel so does a bot understand the user's words
00:07:17and then the third one is does a bot wait until the right timing for its turn it's about it's going to be about listening channel interaction
00:07:28and then the last part is mental model so does a bot understand the user's intention in the listening part
00:07:38and then we can also go to the speaking channel so it's going to be about pronunciation for the sound
00:07:43and also does a bot understand the the words users are does a bot choose the words the user can understand
00:07:53and in the interaction part does a bot speak with the right timing and lastly does a bot speak with the
00:08:01information the user actually need
00:08:05so there are a lot of engineering or linguistic or cognitive science terms that are here that that are here
00:08:13so you can now see that all of those have their right spots in this linguistic framework
00:08:23and importantly these components are interdependent not separate or independent from each other
00:08:30they're interdependent and they're aligned so when you want to do good things about sounds
00:08:36you have to think about words level and then when you want to do good things about these sounds and words
00:08:43you also have to account for interaction so turn taking or turn detection
00:08:50and then finally you want to have good task completion which is the goal of these mental model layer then you have to have all of these
00:09:02without all of those without any of those any of those components your voice agent will fail
00:09:09and then finally uh it has to be well aligned all of these have to be well aligned
00:09:17and additionally you have to keep your mind keep in mind that this is happening on the timeline
00:09:26what i mean by that is it is silently tracked unlike in chat in chat you see the history of what was
00:09:34said uh as text but in voice agent experience uh you say something and the bot says something you go back and forth
00:09:45and then see all these waveforms the air via the vibration in the uh in the air they're all gone
00:09:53and only that user's mental model is a thing that's left and that matters
00:10:00so sounds words interactions vanish the moment they're spoken
00:10:04but the mental model proceeds and grows over the timeline
00:10:09so this is what you have to
00:10:12target for user satisfaction
00:10:16and then what can we do
00:10:19for the bot to meet the standard of the user
00:10:25so what we can do uh would include of course choosing good asr models or configurations and do some post-processing
00:10:34choosing good tts models count configurations and pre-processing
00:10:38and carefully curate the vocabulary that can be shared between the bot and the user
00:10:46and do a good job of turn to turn detection latency and turn taking
00:10:52um and very importantly we have to it would be great if we can do good emotion detection and handling
00:10:59and context retention and by context what i mean is context about all of these
00:11:07and importantly uh it has to be dynamic because things are always changing uh throughout over the course of the call
00:11:16so we would have to do this management dynamically along the timeline
00:11:21for different kinds of people
00:11:24so kids or different kinds of people like these will have different expectations that we have to satisfy
00:11:32not just when they're happy but also when they're not happy
00:11:36so only then you can pursue a dynamic and truly scalable orchestration of voice ai
00:11:42so it's a very difficult job to do
00:11:48we always say that voice is the most natural way of communication but it is actually not easy
00:11:54behind the scene it is thanks to this linguistic orchestration
00:11:59when your bot is not good at it it's a catastrophic failure
00:12:07so paying attention to this linguistic framework would have lots of business implications because then you can
00:12:17decrease all of these user frustration task failures live agent escalation or abandoned calls or silent failures
00:12:28so in service now we have made a good benchmark into an benchmark called eva bench so you can try that to diagnose your voice agent's status
00:12:43key takeaways
00:12:45so voice ai is a joint activity
00:12:49between the bot and the user not just a pipeline
00:12:54and we must serve users needs in multiple layers real time
00:12:59it's not that i have given you a fix today because there's nothing like that
00:13:04it's just uh the fix is in you and your system
00:13:10but what i have given you is today is the linguistic framework you can try
00:13:16to diagnose your system and to build your system upon
00:13:21you can try eva but also you can learn linguistics and hire linguists
00:13:28another thing i want to remind you of is that
00:13:31business implications are linguistic implications and vice versa
00:13:35in this voice ai scene
00:13:37because voice is fundamentally a linguistic and very human and cognitive experience
00:13:45i would like to ask you a longer term question
00:13:50speakers adapt so i uh i'm pretty sure that in this talk in my talk with you guys today
00:13:59you have learned something about me about my speaking style what kind of accents i speak
00:14:05what kind of words i'm using so next time i see you guys in person you would find it more comfortable to
00:14:12talk to me because you have paid attention to me right so speakers are always adapting so the user
00:14:18will be adapting to your voice agent throughout the call so is your system ready for them to
00:14:27use you better use your voice agent better at the next time
00:14:31and languages always change so is your voice agent ready for language change in one year or six months even
00:14:44is your voice agent better at the next time so thank you
00:14:57so thank you
00:14:57so thank you
00:14:57so thank you
00:15:04you

핵심 요약

Voice AI systems fail when treated as simple pipeline software instead of a real-time joint activity, requiring an interdependent linguistic framework across sound, word, interaction, and mental model layers to retain context and avoid live-agent escalations.

하이라이트

  • Voice AI interactions fail because speech components like acoustic sounds and individual words disappear from physical memory immediately after being spoken.

  • Human communication functions as a real-time joint activity where both parties continuously update and align internal mental models.

  • A typical failure cascade occurs when Speech-to-Text misinterprets phonetic sounds, cuts off natural user pauses, and fails to offer interactive clarification.

  • Evaluating voice agents requires an end-to-end framework covering sounds, words, turn-taking interactions, and mental model tracking across both listening and speaking channels.

  • Addressing linguistic alignment in voice architecture reduces live agent escalations, abandoned calls, and silent task failures.

  • The ServiceNow EVA Bench provides an end-to-end benchmark to diagnose and evaluate voice agent performance.

타임라인

Anatomy of a Voice AI Failure Cascade

  • Users frequently default to human agents due to recurring voice AI interaction breakdowns.
  • A single phonetic misinterpretation often triggers user annoyance and conversation degradation.
  • Premature interruptions during data retrieval destroy the user's focus and increase cognitive load.

Voice interface failures usually follow a compounding pattern rather than an isolated error. A simple phonetic error—such as an STT engine mistaking 'MIDAM' for 'MIDAN'—initiates user friction. When the agent prompts for unknown account details, the user requires extra processing time to retrieve the information. Cutting off the user mid-reading due to aggressive turn-detection timers leads directly to call abandonment or requests for human agent escalation.

Human Communication as a Joint Activity

  • Human speech relies on continuous, real-time updates to shared mental models.
  • Acoustic sounds and specific words vanish instantly from physical memory after utterance.
  • Failing to attempt interactive clarification breaks standard human conversational expectations.

Thousands of years of human evolution have trained speakers to expect real-time mental model updates during conversation. Unlike text chat, where transcript history remains visible on screen, spoken audio vibrations disappear into the air immediately. Only the internal mental model remains over the timeline. When a voice bot asks for repeated information without offering clarification strategies, it breaks the joint activity framework.

The Four-Layer Linguistic Framework

  • Voice architecture spans two channels: listening and speaking.
  • Four interdependent layers govern conversation: sounds, words, interactions, and mental models.
  • Task completion relies on the seamless alignment of all lower-level acoustic and turn-taking components.

A complete voice AI model mapping separates communication into listening and speaking channels. Each channel operates across four distinct levels: sound recognition or pronunciation, word comprehension or selection, turn-taking timing during interaction, and intent mapping within the mental model. Because these layers are deeply interdependent, an acoustic recognition error at the sound level directly degrades the mental model layer, causing overall task failure.

Technical Orchestration and Business Outcomes

  • Dynamic orchestration requires real-time handling of ASR, TTS, vocabulary selection, latency, and emotion detection.
  • Business metrics directly reflect the system's linguistic capabilities.
  • ServiceNow offers EVA Bench as an end-to-end diagnosis benchmark for voice agents.

Building effective voice AI requires dynamic system adjustments along the timeline to handle user emotion, diverse accents, and shifting contexts. Misalignments in linguistic design translate directly into negative business metrics, such as elevated abandoned call rates, frequent agent escalations, and silent failures. Employing frameworks like EVA Bench and incorporating formal linguistic principles enables systems to adapt as human speech patterns and vocabularies evolve over time.

커뮤니티 글

아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!

이 영상에 대해 글쓰기