Transcript
00:00:00Yeah, I guess let's get started. I'm Armin. I'm the co-founder and CEO of Perceptron.
00:00:20We'll get a little bit into what we do, but primarily what I want to talk about today is
00:00:25kind of our research stance that we want to move away from distinctions between VLMs, VLAs,
00:00:32world models, whatever you want to call it, to something that we call embodied foundation models.
00:00:42So specifically what we do at Perceptron, kind of our north star really, is we want to be able to
00:00:48build physical AI foundations that give us the ability to perceive, understand, and interact
00:00:53with the physical world in real time. And so kind of the north star mission is really to bridge the
00:00:58physical and digital worlds. Fundamentally meaning that our goal is kind of wherever there is a device,
00:01:07an instrument, a robot, a camera, a sensor, we're essentially there providing intelligence to it.
00:01:14And specifically when we talk about this paradigm of being able to perceive, being able to reason,
00:01:19being able to act, we view this as kind of a unification of traditional multimodal modeling.
00:01:25So I came from Fair. I was there for six years and my target there was really to try to figure out how
00:01:31to scale up recipes for multimodal models. And so one of the early things that we started working on,
00:01:37and we've published a lot in this domain, was around early fusion. So this concept that you want to bring in
00:01:41all these modalities as early as you can. And the real complexity there is trying to figure out what is the
00:01:47correct way to actually properly represent all the different modalities, both on the input and on
00:01:52the output that you want to be able to represent holistically. And so VLMs have kind of became the,
00:01:58when I talk about multimodal models, you probably think of VLMs. So this is the ability to take in
00:02:03some image video and some text and be able to output some text essentially. And there's variations of this.
00:02:09There's models like the ER models, the embodied reasoning models, that are able to output maybe some
00:02:15grounding points, that are able to do spatial understanding or reasoning a little bit better.
00:02:20Then we have things like VLAs that extend kind of the output domain apart from just text and now
00:02:25actions. And these are traditionally, of course, continued to be built on by standard VLM backbones,
00:02:30although there's been some efforts to try to migrate away from,
00:02:35from VLMs to things like world models or world action models, although nothing that's been super
00:02:39fruitful just yet. And then we get into kind of more interesting and complex variants of multimodal
00:02:46models, like world models, where you essentially are outputting video from some inputs, and the inputs
00:02:51can be either image or video, or honestly image, video, and actions. And the last point that I'll talk
00:02:57about is something recent, which we call semantic world models, which is you don't really output anything,
00:03:02but you learn some type of representation that you think is useful in the future. What we kind of view
00:03:08as an embodied foundation model is actually a framing that allows you to both to do the standard
00:03:13perception, the embodied reasoning, and then the North Star target of control all within one model.
00:03:20So being able to reason across different sense of modalities on the input and being able to do
00:03:25most of what I mentioned on the output, all within one single unified model.
00:03:31And so very quickly, I'm going to talk about two challenges, and these are kind of fundamental
00:03:35research challenges that we face, and I'll talk about kind of how our company has approached this,
00:03:40and what other folks are doing in here as well.
00:03:43So the very first thing is think about purely if you're going to try to model something like video,
00:03:48like one hour of video. Depending on the representation, you might have something like
00:03:52one million visual tokens that are coming in. The truth is that there's not actually any ground truth
00:03:57that you can use effectively, right? So you can do things like, and people have done this, of course,
00:04:02like pull out the transcripts, predict the transcripts from the video, or maybe synthetically
00:04:08label some frames, ask us some questions. And it turns out that this is kind of a humongous waste,
00:04:13right? So if you think about what's going into your model, you have one million tokens going in,
00:04:16and you're essentially calculating the loss on something like, you know, 0.2 percent of all the tokens
00:04:22that are going in. And so this is very fundamentally problematic. And the truth is, is any way you
00:04:27try to figure out how to fix this, you're essentially injecting a wrong training signal.
00:04:31Either the signal is too sparse, it's too synthetic, or it's too indiscriminate. And so approaches
00:04:37beyond just synthetic enrichment have been, well, let's predict every single pixel. I mean, true,
00:04:42this is a very dense signal, but it actually very poorly allocates attention. There's, you know,
00:04:47you're comparing, you're treating a background pixel with the same degree of importance as you're
00:04:51treating a grip or tip, or the contact points, or the specific failures, or the physics.
00:04:56And so what we do kind of at Perceptron, and this is kind of some of our core IP,
00:05:01is we think about what does a natural perceptive objective look like? So specifically, how can I
00:05:06predict the percepts that we think will matter in the future in a very, very automatic way? So as an
00:05:11example, you might hard code something like, well, if you have a robotic arm, well, the tip of the grippers
00:05:16turns out to be a very useful perceptive that you can predict into the future. And folks have started
00:05:20doing this. Like the MOMO Act folks from AI2 have done this. There are other VLA's that have done
00:05:26this. But this is still a hard-coded percept. So the question is, can you figure out an automatic
00:05:30way that the model semantically is able to learn this very unique objective? And the truth is we have
00:05:37figured out a way. We're not going to share how we do it here, but this is just kind of hinting at
00:05:41how we approach the problem of sparsity. The second core problem that we spend a lot of time focusing
00:05:49on is context bloat. So if you have always-on cameras, if you have robots that don't necessarily
00:05:54wait, they don't stop, you're essentially having to reason over a very, very long amount of tokens. And so
00:06:01text is relatively dense and video is relatively sparse. And so the question is, are there architectural
00:06:08breakthroughs that are there in order for you to be able to deal with this problem natively, rather
00:06:13than just trying to, on some synthetic level, figure out how to fix this imbalance?
00:06:20And so there's a couple of things that you can do. One kind of core first principle is you need to
00:06:25start training different modalities as completely different. So you can't treat text tokens as the
00:06:31same as image tokens as the same as audio or video tokens. So one thing that you can start thinking
00:06:36about doing is focusing on spatial compression or token compression. And people do really dumb
00:06:41things, and we started off doing the dumb things, and it does work. You can start thinking about
00:06:44averaging, you know, patchwise representations across a video or an image. And you can start getting some
00:06:51interesting compression rates, you know, up to 10x. But still, this is relatively of a hack, and there's
00:06:57not, you know, well used architectural methods to actually solve this. So what we've done in the
00:07:04last couple of months, we've released what we think is our approach to dealing with varying degrees of
00:07:10sparsity, which is just let the model figure out what tokens it should look at and what tokens it
00:07:14shouldn't look like. And so we released our data sparse mixture of experts paper, which essentially
00:07:19allows you to do this. It allows the model -- well, there's a router in the model -- it allows it to
00:07:24actually predict what token I should input, what token I should skip, and it allows us to do it kind
00:07:30of for free through all the different layers. And it turns out that if you just let the model learn,
00:07:36if you're not actually hard coding any significant architectural priors, the model actually does learn.
00:07:43So if you end up visualizing the data sparse compute that our models use, you actually see that the
00:07:49models innately learn to start focusing on very high-density information or task-relevant information.
00:07:56So in the upper right, you can kind of see that the model decides to focus in on the graph, which
00:08:00actually you as a human will also kind of zoom into, because this is likely what's interesting within
00:08:05the figure. And it turns out that even task-dependent -- task-dependent allocation ends up happening as
00:08:13well. So if you look at the bottom left, if you just ask, you know, a very, very general question,
00:08:18you're kind of going to see an attention graph that is throughout the whole image. So the model just
00:08:22doesn't know what the proper way to allocate compute is. At the same time, if you ask it to do something
00:08:27like, you know, segment out all the fruit, you can see that it's going to allocate more tokens to what
00:08:31it thinks are fruit tokens. And so this is a very nice and clever trick that we use and we've published
00:08:36and other folks are starting to use around embedding priors into the architecture that are useful to
00:08:43deal with the sparsity imbalances of your modalities, but not too harsh to the point that the models aren't
00:08:49actually learning natively. And so kind of we put all this together, and you guys might have seen the
00:08:57release, but we essentially released our model which was what we considered to be the first embodied
00:09:03foundation model a couple weeks ago. And this model is essentially frontier with respect to Gemini
00:09:093.1 Pro. It's actually better than Gemini Embodied Reasoning, and it's something like 15 times cheaper.
00:09:15And it's essentially trained on this one petabyte data set that we've collected across literally
00:09:20everything. It's from internet crawls to our own custom mid-training recipes or synthetic data pipelines.
00:09:28We have this one petabyte of data across text, images, videos, trajectories, and these trajectories
00:09:34can be very, very general. It could be desktop use trajectories. It can be playing a video game trajectory.
00:09:41And it turns out that once you start doing these things, very interesting properties end up emerging.
00:09:46And so the biggest property that we kind of saw, which is kind of obvious in retrospect,
00:09:50is that you can actually start thinking of doing classical CV tasks as being an agentic task. And so
00:09:57in this case, we essentially reframed detection as an agentic task. So our model can, you know,
00:10:02write code. It can ask to zoom in into specific portions. It can change the contrast. And you can
00:10:09actually see here, this is a very hard problem. I think there's a whole Reddit subreddit of these
00:10:14problems of trying to find very hard objects and images. And our models essentially do this very
00:10:19well. But they do this in an agentic sense. So this isn't a classical, you know, detect this one
00:10:24box. This is the model actually deciding that it needs to tile things up. It needs to change the
00:10:28contrast. It proposes a box here. I think here it, yeah, increases contrast and it can find the bird.
00:10:36So again, very hard to do if, like, even for a human, and humans are very good at perceptive tasks,
00:10:42this is a relatively tough thing to do. And this all comes from just having
00:10:46natively embodied models that actually understand how to look at different modalities.
00:10:51And so following up, kind of how does this relate to the general physical AI stance around robotics?
00:10:57So I kind of stole this slide from GDM folks. And so one thing we're starting to see from kind of
00:11:04robotics agentic systems is this separation between what we call kind of embodied reasoning models or
00:11:11orchestrators and tactile policy models. So you can think of problems as like, you know,
00:11:17if I have a -- if I'm making coffee and that takes me three minutes to do, I mean, one thing I can do
00:11:21is try to force my whole, you know, VLA to try to figure out how to do this individual task. Or what I can
00:11:26use, I can have an orchestrator model that breaks up these tasks into sub-tasks. And there's a tactile
00:11:31control policy that's running on top. And there's kind of a full spectrum between, you know, full VLA
00:11:36only all the way to this kind of agentic system. But the main thing I'm trying to highlight is that
00:11:41embodied reasoning is actually a very interesting and complex problem that is yet to be solved.
00:11:46That being said, our models continue to be frontier on embodied reasoning.
00:11:50And because they're frontier, we start seeing really cool things that we haven't seen before.
00:11:55Here's a concrete example of doing very complex egocentric -- or not egocentric,
00:11:59but this is robotic data annotation. And you can kind of see here the model is jumping around,
00:12:04looking at different portions of the video, you know, clipping it, figuring out whether or not
00:12:09the captions are correct, self-verifying. And we can all do this because AR models are fast.
00:12:15They're significantly cheaper than anything else that's out there. So if you try to do this with Gemini,
00:12:20this video would probably cost you a couple of dollars, or for us it's probably in the sense.
00:12:24And these all kind of emerge from being able to have these frontier embodied reasoning capabilities
00:12:29that we just previously have not seen from other models.
00:12:37Okay, probably going to share the biggest research breakthrough that we've had, and I think we'll
00:12:41share more of this in the upcoming weeks probably on Twitter. But one thing that we found is we've
00:12:48discovered new scaling laws for embodied foundation models. So these are models, again, that you can
00:12:53jointly do control-based training, you can do trajectory training, you can do perceptive training,
00:12:59you can do embodied reasoning training. If you just figure out what the right way to mix this all
00:13:03together is and the right objectives to use, you actually start seeing very interesting levers that
00:13:08you maybe previously haven't been able to see before.
00:13:11So the concrete lever that I'll talk about is this ability to trade, this ability to trade general video
00:13:19pre-training data for tele-op data. So kind of as we know, tele-op data is very expensive. It's on the
00:13:25orders of, you know, $100 per hour of data. For $100, I can collect significantly more video pre-training data.
00:13:32And so what this graph is showing is that if you're just training pure VLA's, pure policies, there's this
00:13:39band that you have. So you do actually still have scaling loss. So you do get benefits from more and
00:13:43more tele-op data. That being said, the benefits are not as substantial as if you are really training
00:13:48these unified embodied foundation models. And so this is what kind of the bottom half of the graph is.
00:13:55And the really cool kind of lever that we get is you can essentially trade 10x less tele-op data if you
00:14:03have 10x more video pre-training data. And so far, this is kind of held for the amount of compute that
00:14:09our company has. And we'll be continuing to kind of push the fold on how far you can push these embodied
00:14:16foundation models. So here's a couple of videos of a policy that hopefully will open source one
00:14:28of the smaller models in a couple of weeks. But this is all running natively within a single model
00:14:33that is capable of doing the embodied reasoning in order to figure out the task. Actually, it's
00:14:38outputting control tokens. You can see it's a little bit jittery, but that's okay. Hopefully,
00:14:43you'll be figured out at scale. And you can actually see very complex tasks that previously, I think,
00:14:48would be really tough for pure VLAs to do. So I think if you look at the right-hand video, this is
00:14:54requiring the model to actually read the title of the book, have the knowledge about what type of book
00:14:58this is, and then properly allocate it within one of the bins. So this is actually a multi-step task
00:15:04between perception and control that is really, really tough to do if you have a kind of a pure end-to-end
00:15:09control model that is not aware of the different perceptive tasks that it needs to accomplish in order to do this task.
00:15:23Yeah. It's pretty cool. It also works zero-shot relatively well out of the box. So we're excited to get this in the
00:15:28hands of folks in a couple weeks, sometime in July.
00:15:33Yeah. Going to leave a couple of minutes for general questions, but if you guys are interested, let's connect.
00:15:41So one cool thing that we do with our company is we actually, for a limited set of partners, give access to our Mark 1 weights.
00:15:47We give access to our larger embodied foundation models weights. So email me, DM me on Twitter, whatever is easier.
00:15:58And then, yeah, I'll open up. There's a couple minutes left for questions.
00:16:02Thank you.
00:16:03Thank you.
00:16:05Thank you.
00:16:05Thank you.
00:16:09Thank you.
00:16:11Thank you.
00:16:13Thank you.
00:16:24Yeah, I think it's -- yeah, so the question is, how are we able to nail temporal and understanding to this degree?
00:16:31It's a good question. I mean, to be honest, it's not nailed. So there's still a lot of work to actually get it to a place where you can reliably deploy.
00:16:37The core thing is, how do you think about context management? So you have a relatively limited context. So I think the models here have 1 million contexts, but that's relatively easy to fit in with the high FPS video.
00:17:06And so you have to start thinking about, are there interesting things that you can do? I'll throw something out there. We used to do this, but we got past this, but, like, how do I think about, like, key frames versus delta frames?
00:17:19How can I manage my context by training these two off? And then you start thinking about, during your pre-training objective, how can I start kind of natively ingesting things that I think will be useful for the robotics tasks?
00:17:32So, for example, a very basic thing that even kind of the Gemini models used to struggle at, I think the new ones are pretty good, but being able to tell cardinalities.
00:17:40So, like, left and right is very hard to tell if you do internet-scale crawls, because no one on the internet is necessarily labeling things as, you know, this object is to the left of this object, it's below this object.
00:17:51So, really thinking about data distributions early on gives you this ability relatively quickly. And it's also, like, ER, this type of embodied reasoning was a very concrete focus with us, which is why I think we were able to kind of surpass Gemini ER with relatively less compute.
00:18:07Yeah, so probably the coolest robustness that we've seen is that for VLA models specifically, like, if you go and you take one of the Chinese ones,
00:18:35and you try to fine-tune them for a specific policy, if you just change the background of the -- I don't know, even, like, in the table, if you change the background of the table, the policy will actually fail.
00:18:47What's really interesting, if you do this type of joint perceptive and control modeling, you're much more robust to these types of errors or if the light is hitting it a slightly different way.
00:18:56And I think we primarily view this as robustness to background in a way that I think traditional models don't necessarily have.
00:19:03That being said, I'm not going to over-claim, like -- I mean, like, it's still relatively hard.
00:19:07I think if I was going to go and shine a flashlight into one of the arms, it's probably not going to work.
00:19:12But being able to jointly model these things helps a significant amount.
00:19:16We also do a lot of online augmentation, so we do actually, you know, fake, I don't know, one of the arm cameras being off, right?
00:19:27We fake, you know, sunlight coming in from a certain direction, right?
00:19:31So we do these things during training to improve robustness.
00:19:35But the big gains come from taking this early fusion paradigm and then moving into the robotics domain.
00:19:43Cool.
00:19:44.
00:19:50Oh, knowledge bases?
00:19:52It's useful for, like -- I mean, if you want to caption images, videos, it's relatively well.
00:19:57We work with robotics partners for, like, very complex, egocentric annotation like this.
00:20:02So it's not necessarily building on an ontology, but being able to do kind of very deep, structured extraction,
00:20:07I think our models are very good at.
00:20:10Yeah.
00:20:11By the way, everything that I kind of showed here is kind of public APIs, so you can go play around with it.
00:20:15The benchmarks are public.
00:20:17I think I'm out of time.
00:20:19They're cutting me off, so I can talk with folks outside.
00:20:22But thank you, guys.
00:20:27Thank you.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video