The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
AAI Engineer
컴퓨터/소프트웨어창업/스타트업AI/미래기술
스크립트
00:00:00Let's get started. Hi, everyone. Thank you for coming. This is the late afternoon in the last day. So I really appreciate it.
00:00:25I'm Gon, founder and CEO of Friendly AI. Today I want to talk about agentic inference. So I'll first work through what changed, why it matters, and how we rebuild the inference cloud for agents.
00:00:40Before we go deeper, let me briefly introduce Friendly AI. Friendly AI is the frontier AI inference cloud for agents.
00:00:50At scale, faster, cheaper, and more reliably. We are born from a research team at Seoul National University, and those research routes still define us.
00:01:02We are the team that invented continuous batching, the inference optimization that is now standard across the industry, and our ORCA work inspired 3LLM, a widely used open source framework.
00:01:15Today we operate globally, headquartered in San Francisco with a team in Seoul to scale frontier inference.
00:01:24As you know, 2026 is the year agents go into massive production, and it's driven by two trends coming together.
00:01:33First, agents are going exponential. AI agents are driving explosive adoption across software, operations, and knowledge work.
00:01:45Second, open rate motors have reached the frontier and make agents economical. They now rival close frontier motors in capability, which means you can run frontier quality agents on open motors with much lower token costs.
00:02:00Let me make the open rate motors are now strong enough for these types of real agent workflows.
00:02:13Here we gave the exact same task, building a tower defense game with a coding agent to two models.
00:02:19On the left is GLM 5.2, an open rate motor running on friendly AI. On the right is Anthropix, Opus 4.8.
00:02:29The important point is not that the outputs are identical. The point is that both complete the task at a level that is clearly usable.
00:02:38For many agent work flows, open rate models have crossed the quality threshold, but the economics are very different.
00:02:49For the same task, Opus 4.8 costs about $1.50. GLM 5.2 on friendly AI costs $0.27, but 5.6 times cheaper.
00:03:03So this is the promise I mentioned earlier. Open rate models give you frontier quality agents at a fraction of cost.
00:03:12But motor cost is only one part of the story. To make agents actually fast and reliable, the inference stack itself has to change.
00:03:22So let's look at what actually happens inside an agentic workload.
00:03:28So first, let's look at changes in the workload. In the past, the dominant usage was checked.
00:03:34The basic unit was a request. A person asks a question, the model answers, and the person reads it.
00:03:41Latency meant how fast did I get one response? Agents are different. The basic unit is a task.
00:03:49A task may involve many model cores, many tool cores, and it may run autonomously for a while.
00:03:58So the user does not really care about the latency of one individual request.
00:04:04The user cares about when the whole task is completed.
00:04:08That means we have to optimize for tasks, not just individual requests.
00:04:16Let's look at agentic workloads more closely. An agent really runs a session made up of tasks.
00:04:23Each task typically runs in a loop. First, it plans, which usually means an LLM call.
00:04:29Then it acts maybe by calling a tool. Then it observes the result and adds that back into the context.
00:04:38And it repeats this until the task is done.
00:04:41So we are constantly alternating between LLM inference and one or more non-LLM tool executions.
00:04:50So there is a gap between LLM calls. An agent can also create sub-agents and run them in parallel.
00:05:01Agent inputs also look very different from chat. The graph here shows the prompt and completion length distributions of our internal coding agent runs with GLM 5.2, which we use day-to-day.
00:05:15They are much longer. They grow as the task progresses since every observation gets appended back into the context.
00:05:25There is an important pattern here. Consecutive agent steps usually share a huge prefix.
00:05:32If we recompute that same prefix every time, we are burning a lot of compute on work we already did.
00:05:40So this is one of the biggest opportunities in agentic inference.
00:05:46So how token-hungry are agents?
00:05:49Now let's look at a long-horizon task example like deep research.
00:05:53We explained the spec decoding framework in VLLM using code code with GLM 5.2 on friendly AI.
00:06:02There are multiple stages and each stage is composed of sub-agents which run multiple inferences and tool cores.
00:06:12So it might run tens or even hundreds of inference steps, sometimes over minutes or hours.
00:06:19And the shared context keeps growing the whole time.
00:06:23For the user, what matters is not the latency of a single token or one core.
00:06:28What matters is when is my task completed.
00:06:35So agentic inference is not just chat with more requests.
00:06:39It's a different problem. The context grows over time.
00:06:43Tool work is interleaved between model cores.
00:06:46The number of model cores depends on the input.
00:06:49So you can't really plan around a fixed request rate plan.
00:06:55And the real metric is end-to-end test latency, not a single request latency.
00:07:02So how do we do that? Let me show you the key engineering behind it.
00:07:23Here's the engineering map for how we think about it.
00:07:27We built the stack layer by layer around agent-equal close.
00:07:33There are four big pillars I'm going to cover today.
00:07:37Prefix caching, key value, in short KB, cache management, cache-aware routing, agent-aware optimization.
00:07:48And, of course, underneath, we need model layer optimization like sparse attention for long contacts,
00:07:56techniques to reduce errors, fast corners, resilient serving, and more.
00:08:02In this work, I'm going to focus on the four pillars.
00:08:05Let's start with prefix caching.
00:08:09Since agent steps share a large prefix, we compute key value for the prefix once and cache it.
00:08:17Then on later steps, we reuse the cache key value and only process the new suffix.
00:08:23Reading from cache is much cheaper than with computing prefill,
00:08:27so this improves time-to-first token and reduces compute on every step.
00:08:33And the longer the task runs in agents, the more valuable this becomes.
00:08:41But caching only works if the KB cache actually fits and can move around efficiently.
00:08:48So we need strong KB cache management.
00:08:52We use Google memory management to pack more active contacts onto each GPU memory.
00:08:59We use KB quantization to reduce the memory footprint.
00:09:04We use hierarchical caching across GPU memory, host memory, and disk, so we can go beyond GPU limits.
00:09:13And we also use disputed caching so one prefix can be served across replicas, not just inside one instance.
00:09:26At global cluster scale, routing becomes really important.
00:09:29A naive load relancer may spread requests evenly across GPU clusters, but it can destroy cache locality.
00:09:38A cache-aware router at a global scale does something smarter.
00:09:42It sends a request to a pod that already has the right prefix cached, turning a core prefill into one cache ship.
00:09:51At the same time, it still has to balance load, so one pod doesn't become a hotspot.
00:09:58In this example, the two requests of task A go to the same pod one for cache locality.
00:10:08The next piece is agent-aware optimization.
00:10:11And this is the next frontier of agent inference.
00:10:16Today, most systems schedule each LLM core as if it were independent.
00:10:21They don't really understand that this core is part of a longer agent program.
00:10:27But if the optimizer knows the agent-level context, it can make better decisions.
00:10:33For example, preempting the right work, speculatively prefilling context for a likely next step, or making a better cache eviction decision based on agent-level context.
00:10:48So the goal is to reduce end-to-end test latency, not just make one call look fast.
00:10:58When we put all of this together, this is the payoff.
00:11:01We are using the same model, GLM 5.2, with kilocode to create a simple mobile game.
00:11:07We ran the same task with model APIs of Friendly AI and another well-known inference provider.
00:11:14As you can see, Friendly AI completes the same task end-to-end to act faster, thanks to our agent-centric cloud design.
00:11:24So what does this unlock in practice?
00:11:29A stronger production agent stack.
00:11:32Take an agent you already like.
00:11:35Now, plug-in open-weight Frontier models like GLM 5.2, Minimax, and Kimi serve on Friendly AI.
00:11:43The model gives you Frontier quality capability and better economics.
00:11:49Friendly AI gives you the speed, reliability, and end-to-end test performance needed in production.
00:11:56That combination, quality, speed, reliability, and cost is what makes agents actually useful and economical in production.
00:12:06Friendly AI is currently powering teams in production from AI-native startups to global enterprises.
00:12:15I'd like to highlight a couple here.
00:12:20Kilo is a hugely popular agentic AI coding tool serving millions of users.
00:12:27LG is a global enterprise whose businesses range from electronics to healthcare to energy.
00:12:35Very different companies, but they all need the same thing.
00:12:39Fast, reliable, cost-effective agentic inference.
00:12:45This testimonial from our client Kilo says it all.
00:12:50Over the past year, Kilo code has tested several inference providers hosting both open and closed models.
00:12:56In a split test of GLM 5 usage compared against other third-party providers and direct usage from the Model Lab G.AI,
00:13:05Friendly AI was consistently seven times faster with a significantly lower error rate.
00:13:12Today, Friendly AI is a core component of the Kilo stack.
00:13:17And you can consume this however fits your stack.
00:13:23Model API is the fastest way to start.
00:13:26Core Frontier Openweight models through our serverless API.
00:13:29Dedicated endpoints give you your own isolated deployment with guaranteed SLAs for production workloads.
00:13:36And BYOG, bring your own GPU, lets you run friendly inference on your own infrastructure.
00:13:44Same stack, three ways to deploy.
00:13:49To wrap up, there are three things to remember.
00:13:53First, Frontier Openweight models make production agents economically scalable.
00:13:59Second, agents are not just chat with more cores.
00:14:03Agentic inference requires optimizing end-to-end task latency with the challenges I mentioned.
00:14:10Third, Friendly AI is built as an inference cloud for dead world.
00:14:16Fast, reliable, cost-effective agentic inference.
00:14:23Thank you for attending my session.
00:14:25If your building agents give Frontier Openweight motors a try on Friendly AI today,
00:14:30you can get started at Friendly AI in minutes.
00:14:34And thank you.
00:14:35I'll be around after the session.
00:14:37Thank you.
00:14:38Thank you.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기