Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta
AAI Engineer
컴퓨터/소프트웨어AI/미래기술
스크립트
00:00:00Aditya Rahm: Hello everyone, I am Aditya, and we will be talking about two main problems
00:00:17that happen on the short-form surfaces.
00:00:20We are going to talk about how do we deal with these things at a scale.
00:00:25So to get started, first thing is we will understand what are the characteristics of
00:00:28the data that we are trying to deal with, what are the two main problems that we are actually
00:00:33working on, and what are the multi-agent systems to solve those problems at a scale.
00:00:39What are the different specialized small-scale VLM that we can build to solve those individual
00:00:46agentic problem, and how do we build those agents, what are the different ways to actually
00:00:50optimize it, like to make it scalable at a very big scale, and then we will be looking
00:00:56into the evaluation in a very holistic 360 manner, not just like a position recall.
00:01:00What is out there?
00:01:01How do we understand the entire multi-agent pipeline, both from LLM level, tools, MCP, and whatnot,
00:01:08everything out there.
00:01:09And then we will be getting into the optimization techniques which are very vision-specific and
00:01:14data-specific which will allow us and give us some intelligence to tell that we really don't
00:01:20need to do this intelligent workflow for all the videos, we can figure out what are the small
00:01:25set of videos which should be the candidate for doing these things.
00:01:29And we will conclude with the takeaways.
00:01:32So the real-life data is very messy.
00:01:34It is-- we are talking about at a scale of 100 million-plus and a lot more viral content.
00:01:39There is a lot of adversarial content.
00:01:42People are trying to gain the system.
00:01:43There are a lot of multilingual text on the screens and in their videos and images.
00:01:48And the data is very dynamic.
00:01:50It keeps on changing from, like, month to month.
00:01:52Different AI tools are coming, everything out there.
00:01:54So it's really, really dynamic.
00:01:57That means there is a lot of drift issues and other things that comes with the video data.
00:02:00And then we know that there is no clear ground truth for the problem that we're trying to solve.
00:02:05So these are, like, existing problems which happens in the real-world video data sets.
00:02:10We will be-- the first problem that we will be talking about is the modality misalignment,
00:02:14where the first is intermodality, which is a very solved problem.
00:02:19You can have a clip model.
00:02:20You can have a modality on images, videos, audio, text,
00:02:25and figure out what exactly is the cosine similarity on those embedding and figure it out.
00:02:29So this is a much simpler problem.
00:02:31What we will be talking about, how we are doing intramodality issues.
00:02:37So this is considered that like a video.
00:02:39We have a big video.
00:02:40And then suddenly you are seeing something, some ad, agenda, some political things,
00:02:45or something out there which shouldn't be there in the video as when you clicked on it.
00:02:49So how do we actually look into the small segments of the video, understand this problem,
00:02:54and figure out where there is an anomaly or kind of, like, adversarial behavior out there
00:02:59which shouldn't be there in the first place.
00:03:01So this is the first problem, which requires a lot of granular understanding, vision understanding,
00:03:05and a video understanding to see what is happening at a very clip and a very frame level.
00:03:10The second problem is to understand the unoriginal content.
00:03:14In today's economy, with the AI tools available, it's really easy to actually duplicate the content.
00:03:19When someone uploads a video or something, we see that, okay, it is being copied, it is being transformed,
00:03:25and with tools it is getting much easier to transform these videos.
00:03:29So how do we actually detect this kind of unoriginal content?
00:03:33How do we figure out the source of the video?
00:03:35And this has actually caused a problem in attribution, credit, ecosystem imbalance.
00:03:41A lot of, like, user fatigue, we are seeing a lot of repetitive videos which shouldn't be there in the first place.
00:03:47So coming to the multi-agent system, the first thing is that why we are going to multi-agent,
00:03:54not a single agent or single LLM, because the problem is really complex.
00:03:57It requires a really specialized nodes and kind of like an understanding at each part of retrieval,
00:04:04content understanding, and then reasoning.
00:04:07And if you can solve a problem with one agent, one LLM, we don't need to actually do it on multi-agent systems.
00:04:14So this is where the centralized brain, how we are actually thinking, kind of like how the decomposition happens of this problem.
00:04:22The first is basically you have a video and kind of like a video ID and everything given to a reviewer agent.
00:04:28This is a centralized agent.
00:04:29This is basically the orchestrator.
00:04:31Consider that as an API gateway of doing a signal decomposition and finding what is happening around in the image.
00:04:38So what this guy do is basically this guy takes us, the Perceiver agent, and Perceiver agent is the one which is basically considered
00:04:46that a very sophisticated VLM expert with a lot of image and the video tools at disposal.
00:04:52Perceiver agent will take the ID from the reviewer agent and it fetches the video from the database.
00:05:01It decomposes into smaller parts using different tools, different like semantic embeddings, and different kind of like temporal change that happens.
00:05:08Which is a little bit beyond our scope of this talk, but it's our technique which is making sure that we are not doing it at a fixed frame rate.
00:05:17But we are finding where the temporal change happened and compressing those similar frames into one or two frames.
00:05:23And then the Perceiver agent get all the data from a clip level, the video embedding level, get all the information about a clip, embedding, tags, OCR, whatever is present out there,
00:05:34what is the natural language description, and what are the timestamps from which frame to which frame, what are those this metadata looks like.
00:05:41It gives that data to the reviewer.
00:05:43The reviewer will look into all the temporal, this JSON object provided by the Perceiver agent in a raw form, along with embeddings, along with semantic IDs, tags, OCR, and everything.
00:05:52And it does a temporal analysis.
00:05:55It looks into, okay, you know, from first frame to frame or frame number 360, or like till six seconds, it was a video talking about sports.
00:06:05And suddenly we are seeing from six seconds to 6.5 seconds of frame number this to this, we are seeing that like this is changing into some political thing.
00:06:14So just by looking into the metadata, the reviewer agent is able to understand, comprehend, and figure out what exactly is the anomaly coming in the,
00:06:22this temporal space.
00:06:23And once it do that, it figures out whether this is actually a modality misalignment, or there is some bigger issue out there on there.
00:06:30So reviewer agent will look and talk to the retriever agent, and it tells that, hey, this is the video I'm looking into.
00:06:36These are the sum of the metadata that I have already kind of like dedupe and kind of like post-processed.
00:06:41Now give me some understanding of this similar clips which are available in the corpus.
00:06:47So the retriever agent will look into all the signals provided by perceiver agent, all the metadata of a clip level and a whole video level, and index it into different databases adaptively.
00:06:57For example, the topics has to be maybe like an inverted index where we have topics and a lot of videos.
00:07:02For the embedding, it will be a vector databases which is available out there.
00:07:06And for different kind of entities which are extracted, we have a graph databases kind of like where you reviewer agent will figure out, okay, these are the multiple databases, these are the entities, and these are the metadata.
00:07:16Let me index it so that in the online instance time, I can figure out for a particular clip what are the similar clips, what are the similar entities and topics which are available to fetch and improve the recall.
00:07:28So going into individual agent, we talked about perceiver.
00:07:32It looked into the entire video, do a temporal decomposition into small clips based on the semantic embedding and a couple of algorithms, and it fine-tuned the VLM to actually emit all the real concrete
00:07:45and a very granular information about, like, each clip and the entire video.
00:07:51Because we are dealing at a very big scale, that means we cannot just go with the standard VLM which are out there.
00:07:56There has to be a compression, there has to be a cost-effective way to actually solve these models, to do it at, like, billions of frames.
00:08:05That's where we are getting into the pre-training, fine-tuning, a knowledge distillation, quantization to actually deploy and solve the very,
00:08:13very specialized VLM based on this particular problem.
00:08:16And we will be talking a little bit brief about that.
00:08:19The retriever agent is the one which we talked at a very high level.
00:08:22And then what it does in the offline processing is that all the signal which is decomposed by the perceiver agent in an offline fashion,
00:08:29consider that rather an agent, you are actually taking that library and doing an offline analysis on a big scale.
00:08:34Let's say, a ray cluster or something like that.
00:08:37And then once you have all those metadata, it is basically indexing it into different databases.
00:08:42Doing a periodic offline clustering and finding what are the similar content, what should be the embedding ID, what should be the cluster ID for each clips and entire video.
00:08:50So that is the data which is basically used by online inference to find similar videos.
00:08:56And in online manner, it's given a query or clip ID and all the metadata details.
00:09:01It's figured out what is out there in the corpus which this guy is similar to.
00:09:06What are the different things which has high similarity.
00:09:09So what it does, it looks into databases, it finds similar clips and similar authors and different metadata information.
00:09:16Re-rank those candidates and use some of the tools about, for example, the traditional model span score, like a classified score,
00:09:24to see what are the different candidates which might be spam, which might not be high quality, and those things out there.
00:09:29And once it has those top end candidates, it gives back to the reviewer.
00:09:33And reviewer is where it actually process, this is my main content signals and clip embeddings.
00:09:39These are a similar video which is given by Retriever.
00:09:42Let me think over it and whether I need to recreate and get more data from Retriever.
00:09:48So that's where reviewer have all the temporal signals from a single clip.
00:09:52It has all the information of similar clips, understanding and what is similar authors and other thing.
00:09:58It also has a real-time information about how the users are actually interacting with this video.
00:10:03What are the different kind of reports or like the comments, what is the sentiment of those comments out there, right?
00:10:13So a lot of these signals which are actually missed by the offline signals and the VLMs and other kind of agents,
00:10:18are actually also incorporated to see if there is a change happening in the sentiment.
00:10:22What is the response I'm getting in the real time?
00:10:25So there are different tools available for this to actually understand the video, not just from the content
00:10:30and from the semantic perspective, but understanding from the user interaction perspective.
00:10:36Those signals are really, really important.
00:10:38So once we have this job and everything, we build this agentic framework.
00:10:44And each of these agentic frameworks, all these three agents are actually powered by specialized
00:10:50VLMs and small-scale VLMs in a way.
00:10:54So we will be talking about pre-training first.
00:10:58Now, in most of the cases, you see that, okay, you have a pre-training.
00:11:01You fine-tune your vision transformer a little bit here and there and basically tune it for your specific purpose.
00:11:08The thing is that these VLMs, what we have outside and available, the foundational model, the front-end models,
00:11:14they are trained on a very clean, very nice data set, very web data, which is very well-tuned, cleaned, and everything.
00:11:20But data in-house for a specific purpose is not having the same data characteristics.
00:11:26It is messy.
00:11:27It is generated by users.
00:11:28It is for your specific workflow.
00:11:30That means you need to pre-train on those image tokens and language to fine-tune your vision transformer from scratch
00:11:37and see whether there is a delta in fine-tuning it and tuning it from the scratch.
00:11:42That's where the pre-training is helpful.
00:11:44It's a little expensive, but if it can get a delta, that actually works really well.
00:11:50The second is instruction fine-tuning, where we have the two problems.
00:11:54We have certain policies and certain guidelines.
00:11:57We know the content out there, signal out there, and it understands and tune it on that particular data set with a
00:12:04particular output, which is schema, like a JSON schema, to understand what is the modality misalignment or
00:12:10duplication of scores and other chain of thought reasoning provided by reviewer agent.
00:12:16So here we have a video clip.
00:12:17We have a vision encoder, which we have already pre-trained a little bit.
00:12:21Like now we are doing a little bit more training on that.
00:12:23There is a projector, which is sitting between the vision transformer and the language models,
00:12:27and it is actually basically a bridge between them.
00:12:30And then we have our output depending on what is the instruction fine-tuning data we have on this side.
00:12:35So this is the critical part of actually making your model performance go up for your domain-specific.
00:12:45So the context is basically you have a role, you have a policy, what are the tools available to
00:12:50ground that into some of the metadata, and what are the other things available out there.
00:12:54Give it entire things in a very brief manner into the context, and let it figure out what the
00:13:01structure label you have found. These labels are actually in-house label. These are the one where
00:13:06we have created like what exactly modality things are, what are the different issues we are seeing,
00:13:12what are the different chain of thought reasoning there should be there in the model,
00:13:15and how does the output looks like for a human reviewer. So this is like a very high quality
00:13:21dataset that we are fine-tuning it on for different agents, right? So once we have done with the
00:13:27pre-training just to understand the vision aspect of or the other modality aspects of the videos,
00:13:33then we go into the fine-tuning to make it understand and provide the context and the output
00:13:38in the manner that we would want to actually process the dataset on. So the next phase is the DPO phase,
00:13:44which is basically this technique is used a lot in the post-training to actually fine-tune our models
00:13:51into a specific like realm, domain or policy understanding or something. But this can also
00:13:58be used a lot in the production for actually understanding what are the samples which are
00:14:02actually getting not so good by the by your multi-agent systems and by your LLMs. So what
00:14:11can be done is that like you have this production dataset which is coming up, you have a lot of
00:14:17inference happening, you take a sub-sample of these data which is from the production, you pass it
00:14:23through LLM as a judge which is trained in-house on the human-label dataset, and then you have the
00:14:29human review queue to see what is the performance coming up on the actual system. Once you have this
00:14:36thing, if your performance is coming amazing and it is above like your whatever is the prediction threshold you
00:14:40have, 95 percent for each of the problem, it's great. But if it is not, that means there is a way,
00:14:47there has to be a way to learn on these samples where the model did not do well. That's where you
00:14:52have a human in the queue. It understands those, all the traces which is there from all the agents,
00:14:57the LLM call, MCP, and it figured out where the problem happened. Is it like a chain of thought
00:15:02reasoning? Or is it like some of the wrong tools are called, the retrieval did not work? So that entire
00:15:08validations and basically understanding of each and every hook and node both at a model intelligence
00:15:14level. Plus at a hardness level is what you figure out and say that, okay, these are the improvement
00:15:20that I need to do on these samples, which was incorrectly sampled by our system. So once you have
00:15:28this thing, you have a positive sample, negative sample, what needs to be updated and that's where you
00:15:34retrain your model to see what, how can we improve it better. And this is a continuous improvement
00:15:40where we are looking into these samples, retraiting, re improving the models and having a new data set out
00:15:46there from the production samples and trying to understand whether the drift happened or what exactly
00:15:53the model is doing. So this human in the loop is like always a continuous thing where you have a daily
00:15:58sampling from the production and trying to understand how the model and the LLM as a judge are doing.
00:16:05So because of the cost and ability to solve at this scale, we cannot go with the standard like a VLM models
00:16:14from frontier sizes because it's not scalable, it requires a lot of inference, a lot of complexity,
00:16:20and we are solving a very specific problem. Like, I really don't care if the model can solve a coding
00:16:25problem. I only care about my domain-specific problem. That is all I care. This is not exposed to the customer,
00:16:31this is an internal thing. So I would actually do an off-policy and on-policy knowledge distillation
00:16:37and do a little bit of quantization depending on like some experimentation quantization to understand
00:16:41whether the 4-bit work, braining flow, what actually works really well. Then I would have a table of
00:16:47different sizes of distillations and quantization and understand and see where exactly is my performance
00:16:54like up to the mark and where I'm gaining a lot of compute and the cost resource savings from on
00:17:00inference production by doing this optimization. So this is really critical because as a problem,
00:17:07what we have solved, it's all good. But in production, it has to be scalable. It has to be
00:17:13cost effective. It cannot just be like something which is out of the -- which is just out in the market
00:17:21because we are solving a very specific problem out here. Coming back to the evaluation, basically,
00:17:27first is the task of success. Of course, these are the binary things. The modality happen. It does not
00:17:31happen. Like there is alignment or disalignment. So the precision recall F1 are obvious matrices to
00:17:37understand what the task success is. But we want to look at a system in a very holistic manner,
00:17:42not just like the end goal, but what is happening at each nodes, how well like the retrieval system is
00:17:47working, what is the latency and recall of the system, and how well we are able to reason it. What is the chain of
00:17:53thought reasoning coming from these models at each agentic level or wherever its reasonings are
00:17:59applicable, which is basically a reviewer agent in our case? And what is the quality of that? Is it
00:18:04like overthinking? Can we reduce a budget somewhere like to make sure that like the model is actually
00:18:09doing a good job with less budget? Or can we increase it, the budget of the planning or the reasoning
00:18:16of things to have it a higher budget and make sure that the complex problem that we are solving is
00:18:22maybe having a much higher performance and accuracy. So having an adaptive reasoning budgeting is also
00:18:29quite important. And then we have the robustness. What are the edge cases we are seeing? What are the error rates
00:18:34we are having? What are the different nodes and hooks where they are happening? Is it like a tool call is not
00:18:39working really well? It's a retrieval part or LLM is not doing a good job or those kind of things. And
00:18:45then we have a system efficiency where we are not looking just at the performance of the output and
00:18:51everything, but we are looking at each and every aspect of the token cost, which LLM are calling? Can we
00:18:56optimize LLM a little bit more to actually save the cost? What is the efficiency which we are seeing and the
00:19:01latency of each agent and the entire system combined together? Then we have LLM as a judge where we see
00:19:08that like in any of the prediction system, there is always a data drift happening, especially when you
00:19:14have the user-generated content. So how our LLM as a judge, which is used for evaluation and everything,
00:19:20is doing with respect to the human queue? Is there a drift happening? Do we need to retrain our LLM as a
00:19:25judge on new data set, which is coming from labeling team? Or how do we actually do on those parts? In
00:19:30optimization, we have the spatial temporal reduction optimization, which actually look into the frame,
00:19:35which are similar and just compress them into the one aspect. This reduces the total processing of
00:19:41the videos by a huge extent. The second one is basically you have a caching, you have a viral
00:19:47content, which is basically coming up, and you do not want to have the same content out there,
00:19:52which is going through the entire pipeline. And that's where you have a high similarity score and just skip
00:19:57the multi-agent system and just make a call on that. The third one, which is really important,
00:20:01is the metadata pruning. This is where we actually shrink the space from a lot of candidates based on
00:20:07some of the metadata, for example, some topics and some creator which already have a really good record.
00:20:12That means we really don't need to process all these images, all the videos from the creator,
00:20:18which has a really good high authenticity score, which are really doing really well on this,
00:20:22where the video quality is really high, engagement is good. Those kind of things, metadata is something
00:20:27which can be used to filter out the video, which shouldn't even go to the systems. So some of these
00:20:33flags would be helpful. So the last takeaway that we basically we can take out of this room is that,
00:20:42like decomposition or T is a key, or decompose a problem as and when necessary. Adaptive optimization
00:20:50is really important for ROI and cost feasibility. Good evaluation is paramount. Everything in and out
00:20:56depends on this. This is the foundation of your entire system and entire VLMs. And prediction monitoring
00:21:02and qualitative improvement are really essential to make sure this is sustainable in long term.
00:21:08With that, I'll end it. Thank you very much for listening.
커뮤니티 글
아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!
이 영상에 대해 글쓰기