Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax
AAI Engineer
Computing/SoftwareInternet Technology
Transcript
00:00:00Joining us on stage is the co-founder and chief science officer at Hugging Face, Thomas Wolfe.
00:00:30Hello everyone. Hello Olive. Nice to have you on stage. Thanks for having me. So I think you're on for a treat today because you just saw GLM, which is current number two on the artificial analysis. I take that out because nobody can use it. And now we have number four. So basically you will have all the top models, at least the top open source model in a row.
00:00:57And we're very lucky to have Olive, who has a pretty amazing path in life. So she came to the US, Pennsylvania. She was studying doing PhD at NYU in the lab of Yann Lequin, working on JPA. But we decided we won't talk about JPA today, right? Something for another day. And then instead of joining Hugging Face, which was in New York also at the time, she decided to go join Minimax.
00:01:26So for those who maybe don't know all the neolabs around the world, and you're forgiven because I think there's like 64 neolabs right now, Minimax is one of the top of what we call the AI dragons in China. So these are the new, there's Deep Seek, which is very well known now, Moonshot, who does Kimi, Zee, and GLM that you just saw.
00:01:51So here's the new model, and now we have Minimax. They're all extremely good, extremely talented team fighting for the first spot.
00:01:58So the latest release of Minimax was M3, just earlier in June, which was the top model at that time, top open source model.
00:02:08Very impressive. There's a lot of very interesting things about this model, so we'll quickly dive in them, and then talk a little bit about what's specific about Minimax, what's great there.
00:02:20Yeah. So maybe Olive, to start a little bit, can you give us, you know, a little bit of your view of M3, what you like about this model, how is the release?
00:02:32Yeah, M3, we released M3 earlier this month, and it is a smaller model with 400, around 400 billion total parameters, and 20 billion activated.
00:02:45But it is very capable in terms of both coding performances, and also it understands vision.
00:02:52So that's what open source models don't usually have, is that they can -- the model can only deal with coding, but it can also understand videos, images, and it has a super long context of one million.
00:03:09With our new architecture called MSA, Minimax Sparse Attention. So we really put these three things together, because we know that they are -- they will be very important in future AI applications, coding capabilities,
00:03:26agentic capabilities, longer context, and multi-model understanding. Yeah, I think that would be very interesting about the model.
00:03:35Yeah, so there's a lot to impact in this model, and it's still, I think, the only top five model, open source model, that is actually multi-model, so we need to talk about that.
00:03:45But maybe first about the long context, because there was also the first one that really had this real one million token, long context, that's actually functional.
00:03:54And you guys had also the Minimax Sparse Attention, which is this one technique to make that efficient, that you also published and share extensively,
00:04:04so can you talk a little bit about this, maybe how the project went from the attention, how to make this long context?
00:04:11Yeah, I would say the story about long context went back to either Minimax M1 and Minimax 01, where the model was actually -- was able to perform tax of 10 million token context.
00:04:25Ten million. Ten million, yes. But then it was not an agentic model, right? It was just, for example, dumping a book, it would be able to give reviews on it, stuff like that.
00:04:37So what we realized was that longer context actually unlocks a lot of capabilities, especially when interacting with users.
00:04:47And now when the agent is interacting with the whole environment and getting all the tool responses, getting multi rounds, the shorter context wouldn't be enough to perform the complex tasks.
00:05:03So for this version, we said, oh, we have to have our longer context backs.
00:05:09And so what we pursued was with our Minimax Sparse Attention, which, you know, was the architecture that was scalable and had a simple design.
00:05:23So I would say from a higher level, right? It has an index branch that, you know, selects on a higher level what is -- what matters more in the context.
00:05:36And then we have a Sparse Attention branch that calculates -- performs the calculation on the selected blocks to actually perform the tasks.
00:05:44And so, yeah, like that we really designed an elegant architecture so that we can scale the length and then scale the model size in the future with that.
00:05:57That's beautiful. I like how, for those who have been in the field for quite some time, we had a lot of work on attention, right?
00:06:03This N-square, and there was a lot of linear attention.
00:06:06Yeah.
00:06:06And then somehow all of this disappeared at some point.
00:06:09When Flash Attention came around, we discovered we just needed more efficient kernel.
00:06:13And now I like how we come back to thinking, you know, first principle, what is attention?
00:06:18How can we make that more efficient?
00:06:20So one million token is crazy, right? GPT-2 was 1,024, and everyone was like, "Oh, that's really big. We never need more."
00:06:28Where do you see this coming, like going in the future? Like Jeff Dean was pitching me the other day a trillion token attention.
00:06:35Do you think we should go to a trillion token attention?
00:06:38That's definitely something we can explore towards, right, ultra-length of the context, definitely.
00:06:45That's something that's very exciting to explore with, and something that architecture design, along with hardware, would require a lot of research onto that.
00:06:54Yeah.
00:06:55You think there's still a lot of low-hanging fruits?
00:06:57So typically today we saw OpenAI really reducing, I mean, we don't know how as a firm, but like reducing their inference build by half by probably having some more efficient processing around tensions.
00:07:08You think there is still a lot of low-hanging fruit that can be getting how we can process that?
00:07:14So one thing that's still very interesting about M3 is how cheap it is, in particular because of this part attention, or in part because it's small one, but it's also very efficient.
00:07:22Right.
00:07:23You think we can go even way further?
00:07:25Maybe how did you guys invented MinMax Pass Attention?
00:07:29Was it an agent coming up with the idea?
00:07:31Was it a human still coming up with the idea?
00:07:33Tell us a little bit about this.
00:07:35Yeah.
00:07:36So we do think there's still a lot of work that can get into architecture and inference optimization so that the model can be more efficient, especially if there are tasks that are very task sensitive, but require very strong capabilities, right?
00:07:52And for that, those kinds of tasks, we really want the model to be efficient.
00:07:56And who came up with this part?
00:07:59Actually, I think an intern from our team worked on that.
00:08:02Me too.
00:08:03We're not yet an intern.
00:08:04That doesn't usually happen in a lot of labs because I think in some labs, interns don't have access to the data, the work, and stuff.
00:08:15But yeah, we are open to anyone who would like to contribute to our model.
00:08:20So the architecture was actually designed by an intern.
00:08:23That's very good.
00:08:24Still some work for interns here.
00:08:26Good news.
00:08:27That's also a good segue to also how MinMax is working internally.
00:08:32So we were discussing before coming on stage of saying everyone can propose a project.
00:08:37Can you tell us a little bit about how you are organized, how you do research?
00:08:41I think that is very different from even in school or even in earlier, you know,
00:08:49the earlier tech companies.
00:08:51It's pretty different.
00:08:53It's that what we make sure is that we have good foundation and good infrastructure so that anyone can play with the model
00:09:04and can think of what they can improve with the model.
00:09:07And then after model releases, when they are free, right, they can play with the model.
00:09:13They can think of their own evaluations.
00:09:15They can find their own weaknesses and propose a thing that they want to improve on the model.
00:09:21And then other people who are interested in that would, you know, propose to join the project.
00:09:26And they will work on for a couple of weeks or even a couple of months.
00:09:29And when they work out, the final thing is shipped to our model.
00:09:33It is, you know, we use that in our final training and it's shipped out to the audience.
00:09:39Interesting.
00:09:39So you can have people working for a really long time on project.
00:09:42When you say a couple of months, it can be like really deep exploration of what's possible.
00:09:46Yes.
00:09:47And I would say, for example, architecture might require longer time of investigation, research,
00:09:53experiments, even redoing the evaluations for pre-training.
00:09:57Yes.
00:09:58So it might require longer time.
00:10:00That's really nice.
00:10:01Yeah.
00:10:02And I know you're also very big on evaluation.
00:10:03I agree.
00:10:04We could talk about that.
00:10:05One thing probably related to that is this unique specificity that M3 and your team has around
00:10:11multi-modality.
00:10:12So not just text, but this model can also understand image and video.
00:10:16And as I understand, but please explain better, when we read the model card on Hugging Face,
00:10:21it says the model was trained from the first step as a multi-model, not just like user one
00:10:26as after source, right?
00:10:27Yes.
00:10:28Can you tell us a little bit more about that and why you think it's important and why
00:10:33are you starting from the first step on multi-model training and not just this training?
00:10:37So we call it native multi-modality.
00:10:41And so it is somehow typical for model labs to train the multi-model, let's say vision understanding
00:10:50capabilities, after the text pre-training is done.
00:10:54They put adapters and then train that part.
00:10:57But what we found out was that that would actually harm the text performance.
00:11:02And the vision understanding performance wouldn't converge that well because the model is kind
00:11:08of converges towards the text understanding.
00:11:11And it's just not the most optimal and also not the most scalable.
00:11:17If you think about it, we want to scale the data, right?
00:11:20And also we can also, some labs train this capability from halfway through the pre-training.
00:11:27For example, continued pre-training.
00:11:29But what we found that this would be very, you know, recipe sensitive.
00:11:35It is different for, the recipe would be different for different architectures, different, you know,
00:11:41data mixtures, different learning rates.
00:11:43It's hard to control, hard to, you know, scale to your, you can't really scale your experiment results
00:11:52and conclusions to a larger model.
00:11:54And so, you know, what we thought was why not just training from the very first step?
00:12:01That comes the most natural.
00:12:03We know that a lot of labs run into problems doing that.
00:12:07The model would collapse after a couple of steps of training, you know, both text and vision understanding.
00:12:15But we managed to solve that problem.
00:12:18We did a lot of work on the IT and we did a lot of work on the data that we actually training.
00:12:26For example, we do interleaved data, what we call interleaved data.
00:12:30It's actually natural data, but we keep the images and videos in instead of masking it out.
00:12:38And we do some pretty good cleaning and masking on the data.
00:12:42And we do very good reward modeling so that we train it from the first step and scales up a lot.
00:12:50Yeah, it does not collapse.
00:12:52That's really impressive.
00:12:53Impressive.
00:12:54Should we expect much larger model in the future?
00:12:57So this one is still fairly small, right?
00:12:59It's 428 billion parameters, 23 active billion.
00:13:04Well, do you think you will go past the trillion?
00:13:08Definitely.
00:13:09Yeah, definitely in the future.
00:13:12There are many tasks that the model wouldn't be able to perform very good at with smaller parameters.
00:13:21We are definitely going more ambitious than this.
00:13:25That's great.
00:13:26Looking forward.
00:13:27And another interesting thing I always find fascinating about Minimax is how you also have this whole range of apps and products, right?
00:13:36So I remember already, so Minimax started to open source things on the Hugging Face platform in January last year.
00:13:43So that was 80 months ago.
00:13:45And we were chatting a little bit about the team to understand what you were doing.
00:13:48And I remember, so you were already having a huge usage on some of these apps.
00:13:54Can you tell us a little bit how this started, right?
00:13:57So was it basically you had a lot of apps and then you thought, we have all this data.
00:14:02Why not training a model?
00:14:03And then they build up a research team.
00:14:05How was the story there?
00:14:07Our story is model from the first day.
00:14:11So I believe that multi-modality model, a model that can understand all visions and outputs all modalities,
00:14:19was the first thing that our CEO planned on the first day, even before the company even started.
00:14:25So that was the dream of AGI.
00:14:27I think that was very, very early, even before ChatGPT came out.
00:14:30Wow.
00:14:31Yeah.
00:14:32And then apps were something that comes along.
00:14:35Because you have some model capabilities, you want people to experience it well.
00:14:40Not many people can use it with API, right?
00:14:43We can't expect everyone to experience it with API.
00:14:46So we need good user interaction, you know, interfaces, good apps, good scenarios that people can, you know, experience the model with.
00:14:57I think actually those apps covered more than 300 million people around 200 countries globally.
00:15:06And I think over a million companies as well.
00:15:09Yeah.
00:15:10This was mind-blowing when I heard about the size.
00:15:12And we don't often realize the size of this type of usage already.
00:15:17And that kind of brings me to the question around open source business model and all of that, which is the always existing question, which is right now it's nice to open source model, but you also need to have some revenue stream, right?
00:15:33And so I guess M3 is something you decided, for instance, to give for free.
00:15:38And then I think it's great for the world.
00:15:41How do you see this?
00:15:42Do you also have some specific models you use for the app?
00:15:45Do you think about, do you think in the future you'll keep, it's probably hard to say for sure, but do you think you'll keep open sourcing models?
00:15:52How is the culture around open sourcing right now?
00:15:55Personally, and also for the model research team, we always hope to open source the models.
00:16:01That is our plan because we really see how the open source community together can help the model build better.
00:16:09For example, we receive a lot of feedbacks on the model performance from the great community and we receive PRs on whatever we open source, right?
00:16:20And those are very, very valuable and comes to our later versions.
00:16:24So definitely open sourcing is great.
00:16:27That's great.
00:16:28And actually, do you have some ask for the audience, people who are using M3 or Minimax?
00:16:33Is there something you would love them to send back to you as feedback?
00:16:37Do you, for instance, do you read when people try to modify the models or play around, you know, tweaks?
00:16:43Or what is the best thing you think you can take from the community for the future models, for instance?
00:16:51I would say whatever issues that people are running into, especially with multimodality, right?
00:16:57This is the first time that we're combining it together.
00:16:59We are definitely going more ambitious on that in the future.
00:17:03It might have some flaws right now, but we are improving on that.
00:17:06So whatever that feedback that model is not doing that great, we will definitely improve that in future versions.
00:17:13And also whatever features that people want.
00:17:17Say, you know, for example, thinking effort, right?
00:17:21Some people ask for that.
00:17:23Like, everyone can ask and we will try to accomplish that in the future models.
00:17:28Yeah.
00:17:29Do you see a lot of users right now already in multimodality in terms of coding agents?
00:17:33I feel like it's a little bit underexplored.
00:17:37It is.
00:17:38It is.
00:17:39But it can actually unlock a lot of capabilities and a lot of agent applications.
00:17:47Say that, for example, you want the model to read the PowerPoints, right?
00:17:51Or to read some reports that is not very structured.
00:17:55And you want it to understand a very long video.
00:17:58Say that you've done a long playing video and then you want the model to act using some tools after understanding it.
00:18:07And it unlocks a wide variety of agent use cases.
00:18:12So like the agent could finally watch my YouTube tutorial and understand how to use my coding tools, how I describe it?
00:18:18Is it something like that?
00:18:19Huh?
00:18:20Could the agent finally watch YouTube tutorials and understand things from them?
00:18:24Yeah.
00:18:25Yeah.
00:18:26I think so.
00:18:27Do you use a lot of agent coding tools internally?
00:18:30Is it like, I mean, coding for sure, but like, is it also already in terms of research?
00:18:34Is it automated?
00:18:35Yes.
00:18:36In part?
00:18:37Or not?
00:18:38How does this work?
00:18:39Yes.
00:18:40We have our own research harnesses.
00:18:42We build our own research harnesses that automate our workflows.
00:18:46I would say a lot of our workflows are automated.
00:18:49You can see how the latest frontier models all pursues capability-less kernel optimization, right?
00:18:57Like, let the model posturing other models.
00:19:00Let the model build data, auto data, stuff like that.
00:19:05You can see how more and more models are capable of doing those, including M3.
00:19:10Actually, we were very good at those cases, longer horizons and kernel organizations.
00:19:16And so we can use that model capability, harness it together, and help with our daily routine,
00:19:24and make our iterations even faster.
00:19:26Is M3 building M4 already?
00:19:30Building M3.1.
00:19:31M3.1?
00:19:32Yes.
00:19:33Okay.
00:19:34Not yet the job.
00:19:35Already.
00:19:36I would love to finish on what you find exciting in the coming month.
00:19:41What do you think?
00:19:42It can be easier in terms of feature or things you want to see happening in AI, or more generally
00:19:49in terms of whatever really is top of your mind, I would say, is going to happen.
00:19:54A lot of things are very exciting.
00:19:58But what I recently find the most exciting would be multi-agents that I think a lot of
00:20:04AI applications are using model routing, multi-agents, that unlocks even more capabilities,
00:20:11even more complex tasks.
00:20:13And also, it tells us what the models are capable and not capable of.
00:20:19And you can, you know, do a lot of things with that.
00:20:22It's pretty exciting.
00:20:24Thanks a lot, Alive.
00:20:25Pleasure to have you.
00:20:26Thanks for having me.
00:20:27Thanks, everyone.
00:20:28Thanks, everyone.
00:20:29Thanks, everyone.
00:20:29Thank you.
00:20:29Thanks, everyone.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video