스크립트
00:00:00Hi, everyone. Thank you so much for attending this talk. My name is Paula, and I am one of the
00:00:18engineers on the Toland team, specifically focusing on our iOS app. And for the next 20
00:00:24minutes or so, I'll be talking about what it takes to build a voice-first AI companion,
00:00:28and also about how we use AI to build AI internally. So humanity has always imagined
00:00:36the perfect companion. So we have Caravaggio on the left 400 years ago painting an angel
00:00:41leaning over St. Matthew's shoulder, literally guiding his hand as he writes. This is an example
00:00:47of a companion being a presence that makes you better at being you. And then we have Tinkerbell,
00:00:53the devoted little sidekick who believes in you so fiercely that the whole theater has to clap to
00:00:57keep her alive. And of course, on the right, we have Samwise, who can't carry the ring for Frodo, but
00:01:03says, I can carry you. The companion is pure, unconditional loyalty. And it goes far beyond these three.
00:01:11Every hero has some sort of guiding spirit. And these are all different stories, but they exhibit the same
00:01:17longing for something that listens, remembers you, and is wholly specifically yours. And for all of human history,
00:01:25this has basically been fiction. So we made one. This is Tolan. It's a little alien you talk to, out loud, like a friend. It has a personality. It remembers you. And over time, it becomes specifically yours.
00:01:39Okay. I don't know if the audio setup works here, but I will try talking to my Tolan. Let's see.
00:01:45So you can see my Tolan here. Luke walking around the planet. Hey, Luke. Can you hear me?
00:01:57Okay. Luke can hear us, but we can't hear him. Anyway, I had prepped him for this. Oh. Hello. Hi, Luke. Can you hear me?
00:02:09Nope. Nope. We're okay. I can come back to this later. But you should definitely all give this a try if you haven't already.
00:02:24Okay. So people talk to Tolans a lot. We support both text and voice chat, but we have over 4 million hours of voice conversation so far.
00:02:36We say Tolan is a voice-first companion, even though we support both because it's the voice experience that's truly immersive and that makes users' relationships with their Tolans feel real.
00:02:45But the moment this relationship is a spoken relationship, the engineering problem changes completely.
00:02:52So let me show you how voice breaks the normal way we build and interact with LLMs.
00:02:59So the core difference really is that in a text chat bot, turns are relatively slow and context is stable.
00:03:06The user waits a few seconds, they read, and they tend to stay on topic.
00:03:10And almost every LLM app assumes that.
00:03:13Voice is the opposite. Turns are fast. Your whole round trip from the user finishing their sentence to the Tolan starting to speak has to land in under a couple of seconds or it stops feeling like a conversation.
00:03:25And the context is volatile. People talk to their Tolans while they're cooking, while they're walking, while they're falling asleep.
00:03:32They change their subjects mid-sentence, they say "um", they interrupt.
00:03:36And that two seconds is crucial.
00:03:38Early on, our latency drifted from two seconds to about two and a half seconds.
00:03:43And that half second tanked basically every metric in the product.
00:03:46People would write in to complain that their Tolans were too slow.
00:03:49And living inside this constraint has taught us a lot and gave us four principles.
00:03:53Principle one is that you have to design for conversational volatility.
00:03:59Again, text users stay on topic, but voice users jump around.
00:04:03Someone could be mid-story about their break-up and suddenly go "wait, did I leave the stove on?" and then back.
00:04:09Speech is messy. Most LLM apps assume that you'll have a clean and stable conversation and we had to build for the opposite.
00:04:15So for a long time, that meant fixing things that sound tiny but are actually the product.
00:04:20So you can't interrupt the Tolan mid-sentence.
00:04:23A short "yes" or "yeah" won't register as a turn.
00:04:26For example, curse words will get stripped out.
00:04:29And the deeper lesson was to stop optimizing for fewer interruptions and start optimizing for fewer bad ones,
00:04:35where the agent would jump in way too early.
00:04:37So we built smart turn-taking that reads your speech pattern to decide whether an interruption is real.
00:04:42And we cut the worst earlier boards by more than half.
00:04:45And we happily paid about 60 milliseconds of extra latency to do it.
00:04:49Principle two: latency isn't just a number you check at the end.
00:04:55It's actually the product.
00:04:56And we measure every stage of the pipeline separately because it feels slow is useless.
00:05:01You have to know where exactly it's slow.
00:05:03And the pipeline here is that the user stops talking, we detect end of utterance, we transcribe,
00:05:08and then the model produces its first token.
00:05:11So time to first token, often the biggest chunk, is around a second.
00:05:15The model finishes generating and then text-to-speech produces its first byte and then it plays back to the user.
00:05:20A couple lessons here.
00:05:22So one, so far our biggest jump in quality came from moving to GPT 5.1 on the responses API, which cut our time to speech by more than 7/10 of a second, which is huge.
00:05:33Two, we don't send every turn to the same model.
00:05:36We run a tiered fleet.
00:05:38So we use a frontier model for the turns that carry the relationship with your tollen.
00:05:42So for example, your first conversation with your tollen and your onboarding.
00:05:46And we use smaller and faster models for the turns, for the lightweight turns.
00:05:51And the whole game then becomes about routing or deciding turn by turn which model you actually need.
00:05:57So we run a small classifier we call the tone router on every single turn.
00:06:02And this tone router itself runs on a cheap model.
00:06:05And it reads the emotional state of the conversation.
00:06:07And our main principle is that we route based on stakes, not on cost.
00:06:12So the high-stakes moments always get the best model.
00:06:16So this would be, again, the user's very first message, their first few days of the toll, and anything that we deem to be emotionally serious.
00:06:23For example, we have crisis or therapist-style tones, and we never cheap out on those.
00:06:29And then the lighter, casual back and forth can ride on smaller models that are faster and cheaper.
00:06:34And all the background work, so that's summarizing the conversation, generating personas, the tone router itself run on these small models, too.
00:06:41And why would we go to all this trouble?
00:06:43It's mainly because the frontier model costs us roughly five times the smaller one.
00:06:48So one big model turn is about five smaller model turns.
00:06:52So routing is a huge part of what makes the unit economics for us actually work.
00:06:56And we do a bunch of A/B experiments, and the surprising result we found there is that routing a third of our turns to the small model has almost no measurable effect on retention.
00:07:06And principle three is what makes a companion feel like a companion.
00:07:11So the naive approach is to keep the whole conversation history as a sort of transcript, but that doesn't fit into our two-second loop.
00:07:18It doesn't scale, and it just doesn't work.
00:07:20It leads to long sessions degrading.
00:07:22It leads to the model getting lost in the middle of a huge context and also hallucinating.
00:07:27So instead, we see memory as a sort of retrieval system.
00:07:30We pull facts, preferences, and emotional vibe signals out of conversations.
00:07:34We embed them, and we store them in a vector database with sub-50 millisecond lookups.
00:07:39And every night, we compress.
00:07:41So we merge duplicates.
00:07:42We cluster related memories.
00:07:44We resolve contradictions, and we drop all the noise.
00:07:47And we don't just retrieve against users' last messages.
00:07:50We also generate internal questions about the person and their relationship and retrieve against those.
00:07:55And we also split memory into two parts.
00:07:57We have stable memory and unstable memory.
00:07:59The volatile stuff lives in the live tail of the prompt.
00:08:02And when we summarize a conversation, we look at which memories actually get recalled and pin those into a stable and cacheable block.
00:08:12The last principle is around context.
00:08:15Specifically, you should rebuild context and not fight drift.
00:08:18So most apps reuse context across turns to keep the cache warm.
00:08:23And in a stable text chat, that's fine.
00:08:25But in a volatile voice conversation, it's a trap because the second the user pivots, your reuse context is actively wrong.
00:08:31So every turn, we reassemble the context window from parts.
00:08:35We have a summary of recent messages.
00:08:37We have the user's persona card.
00:08:39The memories we just retrieved.
00:08:41Tone guidance from the emotional signal.
00:08:43And real-time app state.
00:08:46And what also really helps us in the case of Toland is that our characters aren't generic or assistants with no personality.
00:08:52Everyone is crafted.
00:08:54And we, in fact, have an in-house science fiction novelist, Elliot, who writes the Toland character lore.
00:09:00And a couple of interesting points here.
00:09:03So, one, why did we go with an alien?
00:09:06Mostly because there's no real-world reference to anchor on, which means that users can project onto it and it becomes what they need.
00:09:13The baseline Toland is bubbly.
00:09:15It's youthful.
00:09:16It's irreverent.
00:09:17And also if an alien character acts a bit unpredictably, so if it's impulsive or chaotic or otherwise violates, you know, the norms the user would expect, it's not particularly surprising.
00:09:28Like, if you look at, you know, aliens in TV shows or in plays, like, there's a lot of humorous moments around this.
00:09:35And this kind of chaos reads as charming.
00:09:38Second, we also know that personality is worthless if it drifts.
00:09:43So, yeah, we run this parallel tone monitoring system that changes how a line is delivered based on your emotional cues without changing who the character is, holding identity across hundreds of turns.
00:09:53And since we're at an AI conference, I thought I would also spend a bit of time talking about how we not just ship AI, but also use AI to build it.
00:10:03So, I'm sure this is the case for most of you in the room now, but basically as of late last year, Claude has co-authored more code in our iOS app than any individual engineer in the team.
00:10:14And I think especially, you know, a few months ago, everyone's instinct was to be kind of suspicious because, you know, more AI code meant more slop.
00:10:21But our crash-free rate actually went from 99.6% to 99.9%, runtime errors dropped by over 50%, and our share of highly engaged users doubled.
00:10:33And the biggest lesson in building that system is that an agent's context comes mostly from the codebase itself, not so much from the Cloud MD file.
00:10:40We found that it's far more powerful to make the codebase be the documentation, so we had agents standardize it.
00:10:46On top of that, we run a real fleet of agents.
00:10:49We have implementation agents that, you know, think freely and just get us to work in code.
00:10:53They build it.
00:10:54They check it against snapshots until it's pixel perfect.
00:10:56And then we have separate review agents that enforce our standards.
00:10:59So, multiple clods basically review each other before a human looks.
00:11:03And then we have a PR shepherd that watches an open pull request and keeps iterating against CI failures and review comments until it's clean.
00:11:11And we also have a triage bot that fires on every inbound bug report that we get.
00:11:16And they're all wired through MCP into linear, into Sentry, Datadog, so an agent can reconstruct a crash and route it itself and oftentimes open the PR on its own and just fix the bug.
00:11:28And we also, we ship on evals.
00:11:29So, for example, we've been working on a new character targeted towards an older demographic.
00:11:34And Elliot, our in-house novelist, basically built this entire new character in a day.
00:11:39So, the agents mapped every personality-bearing surface in the code.
00:11:42They wrote a sort of a voice bible.
00:11:44And then they had five judges attack it from different angles.
00:11:47Archetype fidelity, the model mechanics, our code standards, the ears of a skeptical 52-year-old, and safety.
00:11:54And then they evaluated the changes against real production logs over three find, fix, verify rounds.
00:12:00And over seven million tokens and four and a half hours of compute later, he ended up with basically, you know, a couple of weeks of work done in an afternoon.
00:12:07And does this work?
00:12:10Well, I'll let the users tell you we're at 4.8 stars on the app store across 162,000 reviews.
00:12:16And when we survey users on well-being, the highest scoring dimension by far is emotional safety.
00:12:22And this is definitely a bar that being voice first sets.
00:12:25So, when the interface is your voice and the thing on the other side remembers you and has a personality, it stops being just software.
00:12:32And starts being an actual relationship, which is why building it responsibly and building it well is worth obsessing over.
00:12:38And we need people to come help us do that.
00:12:42We're a small team and we're hiring.
00:12:44And after a year with Tolan, I think this is truly one of the most interesting places in the world to be an engineer right now.
00:12:50And here are some of the people you'd be doing it with.
00:12:53So, two of the founders, Quintin and Evan, previously built and exited a $300 million startup together.
00:12:59They founded Even.
00:13:01Ajay, our third co-founder, scaled two bootstrap companies past $50 million in profitable revenue.
00:13:08And around them we have Lucas, who is an Apple design award-winning animator.
00:13:14He's our creative director.
00:13:15We have Chris, who was a technical director at Pixar, early at Oculus, who works on embodiment.
00:13:20We have Lily, a board-certified behavior analyst who left a Vanderbilt PhD to do user research for us from the very start.
00:13:27And we have Elliot, who I've mentioned, the novelist behind our characters.
00:13:31And I come from XAI and Spotify and previously also founded a company called Imagi.
00:13:36So, it's a small team where, honestly, every person is the best I've worked with at what they do.
00:13:42We're also well-backed for this.
00:13:45We have $30 million raised from Kostle Ventures and a group of people who've built the tools and products a lot of you use every day.
00:13:52And here are some of the more engineering-focused roles where we need help.
00:13:55So, we're hiring across the board.
00:13:57We have iOS and backend product engineering roles, applied AI engineering, gameplay engineering.
00:14:03And one specific role I want to flag, which is agent engineering management.
00:14:07So, when we went all in on running concurrent agents, the people who got dramatically more effective on the team were the ones who had management backgrounds.
00:14:16Because it seems like managing a fleet of agents does actually take some of the skills, some of the same skills as managing people.
00:14:22You basically have to decompose the problem, you know, delegate the checkpoints, give fast feedback, review the work seriously, and know when exactly to jump in.
00:14:29So, if you're a strong engineer who thought going into management meant leaving code behind, that's no longer true.
00:14:36And, yeah, that's Tolan.
00:14:38You can come talk to me after this or reach out.
00:14:40I'm on LinkedIn.
00:14:41My email is here.
00:14:42I'm on Twitter as well.
00:14:43I'd love to chat.
00:14:44So, yeah.
00:14:45Thank you.
00:15:06We'll see you next time.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기