What's New in Inference Engineering — Philip Kiely, Baseten
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00I'm here to talk about what is new in inference engineering so hi I'm Philip and I'm here because
00:00:19I wrote a book this is my third year at the AI engineer World's Fair this is my favorite
00:00:24conference in the entire world it's the highlight of the calendar every single year I really get
00:00:29my start as a speaker and as an engineer here in 2024 I came back in 2025 and did a bunch of stuff
00:00:36I'm here again I love it here and I'm very thankful to the organizers for always having me I wrote this
00:00:44book called inference engineering we published it three or four months ago and I've just been
00:00:49overwhelmed by the response we've done more than I published it in on February 23rd at this point
00:00:57we've done more than 11,000 paper copies we're coming up on 30,000 digital copies and 24 million
00:01:04people around the world or 24 million Twitter accounts so we'll see how many people that actually
00:01:09is have seen something out of inference engineering and with all of this you know great reception there
00:01:17has been one question that people have been asking me why in the world would you do this like why would
00:01:23you write a book about something that's changing so fast well you know I believe that a lot of the
00:01:29principles of inference engineering at this point have been pretty solidified and there's a lot that
00:01:34we can you know learn and kind of repeat over generation and generation of model but today I'm here to talk about
00:01:42what's new in inference engineering this is the first public addendum of all new information since the book
00:01:48came out we are going to review the inference engineering principles a little bit and then we're going to talk
00:01:54about all the stuff that's happened since February 23rd of 2026 in the inference world we're going to talk
00:02:00about what happened to turbo quant talk a little bit about kv compaction we're going to spend a lot of
00:02:05time on d flash and some other new exciting things in specular decoding and then I'm going to do a
00:02:12little bit of prognosticating a little bit of forecasting of what I think is going to happen in
00:02:17inference coming up here and what I'm excited about you know hopefully being able to talk about next time
00:02:23you guys see me up here cool so let's get started so you know one thing I've been identifying now out
00:02:30of tons and tons of conversations with people about influence is a handful of shared principles and one
00:02:38of the big ones I was on this podcast the other day with Sarah we were talking about influence and
00:02:44you know there's two types of influence engineering that have really emerged there's local influence
00:02:48where the overwhelming strategy is just get it working on whatever hardware you have by squishing
00:02:56the model with quantization distillation pruning however you can you know splitting it across whatever
00:03:02GPUs you happen to have in your house and first you get it working and then you make it less dumb you take
00:03:09away whatever you know catastrophic issues all of this compression of the model has created and you try
00:03:15and get it back to that baseline intelligence running at a batch size of one and then there's there's my
00:03:20world which is the batch size and data center world where it's get it working you know just day zero get
00:03:27the build of vllm up get it working and then make it less slow do stuff like kv aware routing speculation
00:03:33disaggregation and you know with within these two worlds I think that we have a lot to learn from each other
00:03:40I am in this talk going to be focused on advances in data center oriented inference engineering because
00:03:46that's what I know but there's a lot of really cool stuff happening in the local world as well
00:03:52so in the book in inference engineering I generally assume that the weights are a finished product and I
00:03:58do think that the handoff from training to inference is an important one to keep in mind and it's a good
00:04:03way of kind of delimiting the space however what I've found more and more recently is that many
00:04:10optimizations for inference come from a dedicated training process and so the lines between training
00:04:17and inference are getting blurrier and blurrier and that's an interesting thing to keep in mind
00:04:22we're seeing this cycle where you get faster inference which gives you more data which you use to train a
00:04:28better model which gives you faster inference which gives you more data and you just kind of like keep
00:04:33doing that until you're super rich uh so with with training for inference uh we have a bunch of
00:04:39new techniques to talk about across what I like to call the big three so we're going to talk about
00:04:45some news in quantization some news in caching specifically the the kv cache mechanism and some
00:04:52news in speculation because these are you know there's a lot of all those stuff in the world of inference
00:04:57including some stuff I'm going to talk about at the end but when it comes to the practical day-to-day of
00:05:02how do I make x model faster usually these are the three techniques that people are reaching for
00:05:09so first thing I publish a book it's February I'm feeling awesome about myself I'm like wow
00:05:16everything you need to know about inference in one place and then uh we uh we had some news in the
00:05:23quantization world so just as a quick review quantization is when we use a smaller less precise number format in
00:05:32order to save ourselves on bandwidth save ourselves on compute make ttft better make tps better it's
00:05:39usually kind of hardware specific gives you cost savings but potentially degrades model quality a little
00:05:45bit and by the way if you want to hear my whole rant about quantization I did a talk at AI engineer
00:05:51Miami last month about how quantization is is not necessarily as evil as it sounds and that there
00:05:57is many things you can do to preserve quality through that process so I was feeling good about my treatment
00:06:04of quantization and then 20 million people saw turbo quant and in fact it like made the memory stock
00:06:12macro dip for for a minute just because everyone was like oh memory is going to be so much more efficient
00:06:16now like we don't need any more flash memory uh which which was wrong but anyway it was this new
00:06:23quantization approach um that was popularized in march of this year that uses polar coordinates for
00:06:29quantization and allows you to quantize the kv cache down to four bits and it was like super hot and I was
00:06:36like oh man like there's this whole thing that that I left out and like what what is this going to look like
00:06:43and so our team did a bunch of research on this um this is if you know uh shout out at waterloo
00:06:50intern on twitter um ali from our model performance team I'm not sure if he's still an intern actually
00:06:57but yeah he is from waterloo anyway so he he wrote this great piece about the math behind turbo quant
00:07:04and basically the benefit you get out of turbo quant is that you get to represent the kv cache with four
00:07:09bits instead of eight bits you save half the room and half the band and and you get effectively double
00:07:14the bandwidth when you're moving kv cache around in your system memory but the drawback is pretty big
00:07:21um for turbo quant it turns out that you need to do additional computation in the forward pass to
00:07:25account for this during decode and it cuts tps by more than half and that's just an unacceptable trade-off
00:07:32for a lot of the production use cases so we we took a good hard look at turbo quant but but are not using it
00:07:38for you know any any of these real workloads we're still on the traditional nvfp4 quantization that
00:07:44said it actually is a great technique for the local inference folks so if you are running a model
00:07:51especially a long context language model on your local computer on gpus in your basement um you have
00:07:58a very limited amount of memory that's the number one bottleneck and so anything that can free up memory
00:08:04from kv cache and allow you to put those longer sequences on there is going to be very valuable
00:08:09and the additional forward pass computation is going to be like less of a drawback um so still
00:08:16turbo quant is a fantastic research paper a really great technique that just ended up not being as
00:08:22applicable in the data center inference world as it might have first appeared um you know instead we're
00:08:28focused on you know nvfb4 with a focus on quantizing the weights versus the kv cache um you know doing
00:08:36our best to find rough edges in the quantized weights um make sure that we're not flattening out of
00:08:41probability distributions for the kv cache itself focusing instead on kv aware routing kv offloading kv sharing
00:08:49using uh you know nickel and using nvidia dynamo and other tools in order to move the kv cache around
00:08:55the system and potentially offload to cpu ordinary memory etc versus trying to use uh turbo quant to
00:09:04to compress it um and then we're also focused on quantization across modalities um so thinking about
00:09:11how can we apply the benefits of nvfp4 not only to language models but also to you know image and video
00:09:17models ali also wrote a lot of great stuff on twitter about that which you should check out
00:09:23so that said the kv cache is still very important and let's talk about it let's talk about kv compaction
00:09:29again quick review kv cache if you put in the same prompt with the same prefix uh you get to reuse the
00:09:35tokens that you calculated pre-fill on last time that makes your whole system faster and more efficient
00:09:42broadly kv cache is lossless memory there's only a couple sources of lossless memory when we think
00:09:48about our inference system we have the content of the prompt the context you have the the kv cache and
00:09:55that's going to scale linearly with the amount of data you pass in and now if you're thinking about you
00:09:59know million token sequence links uh that actually gets pretty substantial so a lot of people are thinking
00:10:05about how do you compress memory how do you compress context agent harnesses will compress context lag
00:10:11search all these techniques that we've been talking about for years are a sort of compression of a
00:10:17larger context into something that you can give to a model you can write to files all of these things
00:10:22scale sub linearly with the amount of data that you have but what if there was a middle road what if there
00:10:28was a way where you could get quite a bit of compression in the data that you remembering with near lossless
00:10:34information retention so we uh you know we have a lot of different ways that we can think of what to
00:10:41keep in the cache you know recent compaction methods have shown that we can replace the cache with a much
00:10:47shorter one we've got papers like attention matching and cartridges that have given really promising outcomes
00:10:53here with uh high compression ratios but both of these are run at inference time again one of the techniques
00:11:00i want to talk about or one of the themes i want to talk about is training for inference so in this case
00:11:05i want to introduce something called still by the base 10 research team where the synthesis on top of
00:11:12the cache where we're keeping a learned representation of the information rather than a the information
00:11:17directly or a sort of deterministic subset of it is amortized via training so charlie and mudith from
00:11:25our post training team did a fantastic chalk talk at cosa compile recently um it's up on youtube i would
00:11:31encourage you to take a look at it if you're interested in learning about kv compaction um i do not
00:11:37unfortunately have the time or the genius to explain everything up here um but the the basic mechanism is
00:11:46that still is a perceival bottleneck um that takes a fixed set of loan query vectors cross attends it
00:11:53against the full kv cache and produces a set of compact keys and values in a single forward pass
00:11:59this creates a fast differentiable compressed memory that the llm can attend to as if it was real context
00:12:05so if you're interested in kv compaction definitely check out charlie and mudith's work
00:12:09it's been a really fantastic thing to learn about so that's two of the techniques we've talked about
00:12:16quantization we've talked about caching the final one is speculation and there's been a lot of change
00:12:21here um as a review uh speculative decoding we're going to use draft tokens we're going to verify them
00:12:29during the forward pass and we're going to use that to generate more than one token per forward pass
00:12:34it helps a lot with tokens per second and it is a fully lossless optimization which is great because
00:12:40we don't have to worry about quality at all now in in the sort of history of speculation we started
00:12:46the specular decoding all of these are in the book um you have spec deck where you use a small model from
00:12:52the same family to generate draft tokens turns out small models are like not great draft token generators
00:12:58they're great small models so we invented it as an industry a bunch of new methods like medusa where
00:13:04maybe just you add dakota heads to the model and then eventually eagle three which was hey what if
00:13:09instead of taking a tiny model from the same family we actually train a billion parameter model on the
00:13:15hidden states of the target model to generate draft tokens and that actually worked really well and so
00:13:21you know as of maybe february last year of february of this year eagle three was the best method in
00:13:27speculation now we got dflash dflash is even better so it's diffusion for speculation uh dflash creates
00:13:36a sequence of draft tokens instead of a single token so the model is a diffusion language model which
00:13:43means it creates a whole sequence of tokens in the same way that a video or image generation creates a
00:13:49sequence of frames or a sequence of pixels and iterates over it rather than doing a auto regressive token
00:13:55generation dflash models might be two or four times slower to run but they're going to predict eight
00:14:01or 16 tokens at once um in that in that window while eagle is only doing one at a time so a single dflash
00:14:08forward pass is faster than the entire eagle draft phrase and predicts more tokens these tokens are able
00:14:14to cross attend to each other and generally create a higher acceptance rate because in speculation acceptance
00:14:21rate is everything so in the wild we're seeing a more than 3x improvement from dflash this is measured with
00:14:30a single b200 quen38b and we can see it versus eagle it's a substantial improvement in the tokens uh both
00:14:40the token acceptance rate and the tokens per second these dflash models are trained with a attention mask for
00:14:47bidirectional drafting um so the target model is going to provide the context um and within each block
00:14:54we're going to have a subset of clean tokens that are sampled and the attention mask is going to enforce
00:14:59causal consistency uh but it is still going to allow for bidirectional attention um where in you know a
00:15:06traditional autoregressive model you're only looking at the tokens in a single direction so that's why we're able to
00:15:13you know take advantage of this diffusion based architecture and then you know i thought i was
00:15:19done and then a couple days ago uh dspark came out now dflash we do have up and running in production
00:15:25dspark is is new research um so this one i can basically only say like hey it exists it's cool we're
00:15:31looking at it um the difference versus dflash it still has that diffusion model but it also pairs it with
00:15:38a sequential model and the idea is that we're going to improve acceptance rates um by having these two
00:15:45models work together rather than having just the iterative speculator uh just the diffusion
00:15:51speculator or just the auto aggressive speculator so dspark very exciting um but we don't have any
00:15:57production results with it yet to share um hopefully you know we'll have those for next time
00:16:03um what we do have production results though on is continuous speculator retraining so this is we're
00:16:09back to dflash here um and this is the idea that you know speculative decoding is very dependent on the
00:16:15actual content of the prompts and responses that you're looking at in your system and so if you are
00:16:21continuously retraining on those prompts and responses in your live system you can see a 20 to even 2x
00:16:29improvement in your token acceptance rates this is actually like really hard to do it takes a lot of
00:16:36storage and you have to make sure that you have permission to use the data that you're processing in
00:16:40this way it takes a ton of compute and you have to move all of this information around and if you
00:16:46change the underlying model you also have to change the speculator model but when i look forward into the
00:16:51future i do think that continuous speculation for very very large scale systems is going to be a worthwhile
00:16:58optimization so what is next in influence um the following is like personal opinion and speculation
00:17:06and public information and like if i knew anything that was actually coming out i wouldn't be able to
00:17:11talk about it um so so this is just like what i think is going to happen you know i've been through
00:17:17three hardware cycles um through the ampere release the hopper release the blackwell release um and it always
00:17:23takes time for you know when these chips get shipped to when they get installed in data centers when the
00:17:30entire software stack really is able to take advantage of their capabilities but some things that i'm
00:17:36excited about are you know with with rubin it looks like the nvfp4 performance is going to be fantastic
00:17:44so the more we can like honestly borrow techniques from local inference and get a lot of confidence running
00:17:49models in this nvfp4 data format the more we're going to be able to take advantage of the awesome
00:17:54performance of of the upcoming rubin systems i think that like disaggregation and system-wide
00:18:00communication is going to be increasingly important we're seeing really excellent early gains from pd
00:18:05disaggregation and the ability to you know move information like kv cache data around the system is
00:18:12going to be increasingly important and then like i said the theme of training for inference is going to be
00:18:17be something that continues to have a big impact in the industry moving forward so thank you all so much
00:18:25for the talk for coming to the talk um i'm on twitter i'm on linkedin and i'm giving out free books um
00:18:32you can download a pdf at the qr code or come down with me to the base 10 booth to get your free copy of
00:18:37inference engineering uh we've got a bunch there maybe enough for everyone if not we will have a career
00:18:43bring some more from the office um so yeah i'll be downstairs at the base 10 booth thank you all so
00:18:48much and have a great day
Community Posts
No posts yet. Be the first to write about this video!
Write about this video