스크립트
00:00:00Hi, thanks for coming everyone. I'm the co-founder and CEO of Elorium and I'm here to talk about some
00:00:22of the issues with current models, current frontier models, this includes Claude, ChatGVT and Gemini and how they handle visual problems and this might be new to some of you who don't work in the visual space but actually there's quite a big gap between how these models handle visual reasoning and how humans deal with it and you will see that we're actually quite far away
00:00:52from any definition of AGI for visual reasoning. So here are some examples of how easy it is to find where models break down and you can find these examples yourself, it just takes a few minutes.
00:01:09In this first example, we have a chessboard hallucination and we give the models this picture and ask how many white squares are in the image and any ordinary person who doesn't hallucinate would probably not say it's 32.
00:01:2932. So 32, of course, the models say this because they see part of the chessboard and they hallucinate the complete board and as a result they give the wrong number.
00:01:40And you see this quite a lot that models rely a lot, current models rely a lot on pattern matching. That's what makes them so good at identifying plants and animals and flowers in the real world.
00:01:52But when it comes to complex questions that part hurts them. So the pattern matching is actively hurting them in this case. So what's going on in their reasoning is that this is a chessboard. Chessboards all have 32 squares. Therefore this one must have 32 white squares too.
00:02:11On the right example, I'm a big board game player, I have quite a collection. So you can see there's a board game theme going on here. And actually you can reproduce this outside if you just go to, you know, outside the talks. It's a board game area. There, there are chessboards.
00:02:30I will bet if any of you place the pieces in some kind of random position, no frontier model will be able to tell you where those pieces are located, where all those pieces are located.
00:02:42And then another example here is Catan. Here another very simple question, how many rows does the blue player have? These frontier models think extensively about this problem.
00:02:54One response I've seen is that, oh, the guy has 10 blue rows off to the side of the board. Therefore there must be five blue rows on the Catan board. But obviously that's not true. There's seven if you actually count.
00:03:11So these models, again, are great at guessing, great at pattern matching, but they are not very spatially grounded. And they just can't handle any kind of detailed questions.
00:03:23And then finally, we have this example where it actually affects robots, where here you have a robot arm manipulating this cup and cooker basically. And the state of the art models today, they miss the fact that the robot arm lifted the lid.
00:03:48And at the end, they also miss the fact that the robot is turning on the stove right now. So there's essentially context amnesia happening. And this is because these models can't maintain consistency across long videos. And they very easily lose track of what's happening.
00:04:07And a very common question I get is how do you define a visual reasoning problem versus visual understanding problem? Or you could say like visual thinking compared to visual understanding. I think a very simple way to do it is just ask yourself the same question. If you looked at an image or video, how long would it take you to answer the question? So for example, in both of these cases, you know what's happening. You know what's happening? And a very common question I get is how do you define a visual reasoning problem versus visual understanding problem versus visual understanding problem? Or you could say like visual thinking compared to visual understanding.
00:04:19I think a very simple way to do it is just ask yourself the same question. If you looked at an image or video, how long would it take you to answer the question? So for example, in both of these cases, I doubt anyone in this room will be able to give an answer if they were only allowed one second to look at the image. So one second isn't enough to do these kind of like complex questions, also called like system 2.
00:04:37Kind of thinking in Daniel Canon's book. But if I asked you, what game is this? Or similarly, what flower is this? Or if there are only three pieces on the chessboard, if I asked you how many pieces are there? Those questions, I'm sure all of you will be able to answer in less than
00:05:07a second. And similarly, all the frontier models would get that kind of question right. So that is the distinction that we make between what is understanding, what is like pattern recognition versus what is reasoning, where you actually have to look in detail at the picture and look at various things. And this is exactly where frontier models fall apart today. So as you are designing your own systems, that's something to keep in mind. Keep these visual tasks very simple.
00:05:37Otherwise you will have hallucinations and a lot of hallucinations. So this leads into evals, of course. Frontier models, there are already a bunch of multimodal reasoning evals or visual reasoning evals. Some that you might have heard of is arc AGI. This is quite often brought up to people saying, oh, the frontier models are 85% or 90% on arc AGI.
00:06:07Therefore we are 90% of the way to AGI itself. But I think these people, they haven't really looked at any of the benchmark data because if you actually look at the data, you will notice that the images are only 32 by 32 or 64 by 64 pixels.
00:06:24And I would challenge anyone to give me like a real world complex task that can be reduced to 32 by 32 pixel problem. I think you will very quickly realize almost no tasks, almost no interesting tasks can be reduced to that kind of resolution.
00:06:41Another eval that people commonly bring up is MMMU. This is the massive multi-discipline, multi-modal understanding. This is a step up from MMLU because it has images rather than just pure text science questions. This is science questions based on images, but still images are a minor part of a lot of these questions. A lot of the questions you can just answer without looking at the image or just doing some pattern recognition, just knowing roughly what the image is about.
00:07:11So what we really need is new visual reasoning benchmarks in the industry that really target the things that people care about, like geometric alignment, spatial intelligence, object permanence.
00:07:25And these are really critical for AI to be deployed in these visual use cases. And you might have noticed that still in a lot of industries that are primarily visual, which I will go into, there isn't much uptake of AI.
00:07:41Right? A lot of the AI uptake has been in the software engineering world and in the mathematician world and in like documents, document handling, et cetera. But this, there is actually a huge gap, huge opportunity that is just being looked over right now based on the interest in coding.
00:08:01And so the missing paradigm in visual AI is thinking. So we have generation models, very high quality generation models like ByteDance's C-Dance model. So we have these very high fidelity models and they look great, but they lack actual physical grounding and causal logic. So you will, you probably notice that if you ask these models to produce a picture, a video of a
00:08:06AI is thinking. So we have generation models, very high quality generation models, like
00:08:13ByteDance's C-Dance model. So we have these very high fidelity models and they look great,
00:08:20but they lack actual physical grounding and causal logic. So you will, you probably notice
00:08:26that if you ask these models to produce a picture, a video of a, like something blowing up,
00:08:33like a building falling down or these things, they look very cartoonish. They look Hollywood
00:08:38style kinds of things. And that's because they are just outputting what was in the training
00:08:43data. And a lot of disaster videos, a lot of like action kind of videos on the internet
00:08:49are just going to be from Hollywood or game engines. So they're working to reproduce that.
00:08:54And that's fundamentally a problem because that means they can be no better than those kind
00:08:58of videos. Understanding the, what we are, where we currently are is we have a lot of tools
00:09:05that can map pixels to semantic labels, like Google Lens. It's obviously great to identify
00:09:10plants and flowers and I use that all the time. There's SAM3 for segmentation, YOLO for object
00:09:17recognition detection, Mascar CNN. These are of course highly robust and they're used everywhere
00:09:22in the industry, but they're fundamentally passive. So there's no reasoning capability to them.
00:09:27So they can't answer more complex questions. And really where the frontier is, is with thinking,
00:09:33visual thinking models. These models will have active spatial and temporal intelligence.
00:09:38They can extract actual logic for planning, agentic workflows and physical execution.
00:09:45And so our approach is a fourth stage. So we are collecting and generating our own multimodal data,
00:09:57visual reasoning specific data. This kind of data we found you just can't get online.
00:10:03We have a synthetic data flywheel using evals, agents, SFT and RL to improve the model.
00:10:09We're making some, we made some advances to the architecture in terms of various different improvements
00:10:18on top of the transformer based architecture. And we're also enabling visual chain of thought reasoning.
00:10:23And this is one of the key things that humans have that no frontier model has today.
00:10:28Since the frontier models are only textual chain of thought based.
00:10:32And this is one example of a visual chain of thought. So the question is like,
00:10:36how many red hotels are built in this photo?
00:10:40Then the model realizes, oh, first we need to identify all the hotels.
00:10:45So it draws boxes around hotels and other objects.
00:10:50And then it reduces that to the red hotels.
00:10:54So it's this multi-step process happening in the visual space natively.
00:10:59So about our company, I'm the co-founder and CEO.
00:11:06I spent the last 12 years at Google Brain and DeepMind.
00:11:09I developed a lot of the foundational techniques for the model, for modern LLMs.
00:11:1411 years ago, I was the first author of the work that introduced pre-training and fine tuning.
00:11:19That's the work when combined with the transformer paper in 2017 led to the GPT series of models.
00:11:25So all the GPT papers cite our paper.
00:11:28I co-led the early MOE models, the first model that was state of the art called GLAM.
00:11:33And then more recently, I co-led the PALM II pre-training in architecture.
00:11:38And I was co-lead for the Gemini data area.
00:11:41And my co-founder, Yin Fei, he was at Apple and Google research.
00:11:45He led research for Apple's first public multimodal model, MM1.
00:11:50And he has a lot of experience in visual reasoning across language as well.
00:11:58And this is our team.
00:12:00So we're roughly 20 people now.
00:12:03We also have a chief reasoning architect, Dustin Tran.
00:12:07Previously, he was lead of post-training at XAI.
00:12:10And we've hired a world-class team across many other companies like Apple, XAI, DeepMind, Amazon, and so on.
00:12:24And in terms of the use cases that I mentioned, robotics is one primary use case.
00:12:30So robots have really critical bottlenecks performing complex real-time physical actions in these kind of dynamic environments.
00:12:39But existing vision models, you probably realize, are trained from static images and very directed videos.
00:12:47They don't have active physical interaction.
00:12:50So existing methods are over-engineered and brittle.
00:12:55And we are planning to release a model API available by the end of this year.
00:13:02And at that point, the API can be used to deliver action-relevant scene understanding into existing planning and control systems for these robots.
00:13:17Another important use case for visual reasoning is construction.
00:13:22So construction sites, they have these very complex zone-specific safety rules.
00:13:29And computer vision can't adapt fast enough to changing safety rules.
00:13:34They also can't interpret things like OSHA policy language and match that language to what's actually going on at the site.
00:13:44Or understand the spatial relationships that are important there.
00:13:47For example, like how many of these workers are wearing helmets?
00:13:51Like is the construction happening according to the plans that were defined earlier?
00:13:59And currently enforcing these rules require training separate models for different use cases.
00:14:06Because these models, as I said before, are very brittle.
00:14:09So you constantly have to do retraining.
00:14:12And our approach with the video understanding capabilities that we are building into the models allows you to ground these video streams in the safety regulations.
00:14:26And of course the safety regulations are in text.
00:14:28They're in language.
00:14:29So you have to be - the model has to manipulate both language and vision very well.
00:14:35And yeah, this will allow these models to interpret site policies using the current camera infrastructure that they have.
00:14:44And then finally, architecture and design we think is also a very promising use case here.
00:14:54This is exactly the use case where you need to be very detail-oriented.
00:14:58So back to the board game example around counting, spatial relationships.
00:15:03This shows up a lot in architecture and design.
00:15:06Like if you design a - if you design a house with four bedrooms instead of three, that homeowner is going to be very angry.
00:15:13Right?
00:15:14So obviously counting is actually important.
00:15:18And also just understanding these spatial constraints, real-world constraints is a very manual process these days.
00:15:28We spoke to a mechanical engineering company just a few weeks ago and they said to design one small part of a robot testing platform.
00:15:38It takes 100 to 200 hours of the time to design the entire testing platform.
00:15:45I believe it takes 2,000 to 3,000 hours of human time.
00:15:51Yeah.
00:15:52And they've - a lot of these places they've tried frontier models, but they just don't work for these use cases.
00:15:58They really struggle to understand visual context across these like architecture blueprints, 3D CAD CAM files.
00:16:07And so there are lots of errors there.
00:16:09And similarly we believe that this can be useful for other kinds of design as well.
00:16:15Not just architecture and engineering, but maybe like designing for the web or fashion or other things.
00:16:23And our approach is we're using multi-modal reasoning to extract this geometric logic that's important.
00:16:33We're allowing programmatic validation or simulation validation.
00:16:38Just like in code, you can run code against unit tests.
00:16:42You can also run mechanical devices through simulators that have been developed through Siemens and various other companies to see if something will work in the real world.
00:16:53So there's a lot of parallels actually between this kind of like mechanical design and coding itself.
00:16:59But mechanical design is still relatively untouched by AI.
00:17:04And yeah, CAD CAM quality control is another potential use case.
00:17:08And ultimately we believe that this is going to be a critical step to the future of mechanical design,
00:17:15where AI can make faster cars, more efficient rockets, better batteries.
00:17:22And all these things cannot be done just with code.
00:17:25People are not coding up the next iPhone or coding up the next SpaceX rocket.
00:17:30It's all fundamentally very visual.
00:17:35So you can find out more about us through our website, Lauren.ai, our Twitter page, x.com/laurienai, or our LinkedIn page.
00:17:48And yeah, happy to take any questions.
00:17:51I'll be standing around here for a little bit.
00:17:54Thanks.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기