Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

EnglishالعربيةDeutschEspañolहिन्दीBahasa Indonesia한국어Português中文
AAI Engineer
Computing/SoftwareSmall Business/Startups

Transcript

00:00:00Hello and welcome to this talk on skill issue. It's actually no longer a skill issue. By the end
00:00:22of this talk you will be able to build a lot with the vision models if you are already not doing it.
00:00:27So shortly about me, I'm Merve. I've been working on computer vision since lava in the fixed times and lately I work on more agents and on-device stuff because I'm a bit fascinated and I love vision language models so much that I even wrote a book on it but I don't want developers to directly use vision language models anymore and I want every single developer to start
00:00:54building vision language vision applications end-to-end and this talk is going to give you like a good baseline to do so and the common behaviors I observe with the developers is the fact that they try to use vision language models for everything but they you will never get a real-time and when I mention real-time it's like you have a toaster
00:01:18and you get like 30-40 fps on it like whatever whether you are building like image classification instance segmentation or whatever
00:01:28and they are not super robust in terms of like if you were to train like an RFDTR which Joseph spoke in the spoke about in the first talk
00:01:39um it will always outperform your vision language model and I'm going to prove it today
00:01:47and on the right hand side you can actually see me uh doing stuff with RFDTR
00:01:52um and another thing is that they don't read the licenses everybody's like whenever I post something about object detection they always ask me about YOLO like YOLO is a good model uh but it's uh it has a GPL 3.0 license I think
00:02:07and I could swear on my life that there is like some developers that actually deploy it without knowing
00:02:12that they have to pay for it um so yeah like uh I want you to migrate to Apache 2.0 models today
00:02:23and for this I have built something called vibe vision and it's kind of inspired by this post by Maziar
00:02:30basically what he does is that he gives um Sam 3.1 model as a tool to Gemma 4 to call and I find it super
00:02:39impressive um and today I built like a toolkit where you can do this with like even more things
00:02:49so I'm kind of distilling myself um first off uh this toolkit has my favorite models as tools uh so
00:02:57that you can give it to your agent because your coding agent is a bit of a clueless computer vision
00:03:03engineer basically um and when I distill myself basically like whenever I pick a model I always check
00:03:09the following so first up the license is the biggest priority for me it has to be Apache 20 or MIT or
00:03:16something non-commercial uh secondly uh the performance has to be on par uh depending on its size or
00:03:24architectural choices so I check from the benchmarks whenever a model from a computer vision conference
00:03:31comes out and third vibes obviously and so this toolkit has a second part to it and it's a bit like a vibe
00:03:41training part which is uh the most exciting part so I will go through it first so I put myself in
00:03:47developers shoes to build like a vision application um if I have labeled images easy I can just train a model
00:03:55or like if I I can give my computer computer vision agent some tutorials uh to do that because it's all out
00:04:02there like we build transformers for it but if I have images only I have to annotate uh and then evaluate
00:04:11the annotations and then just train a model but how can you do it on scale you can actually use a vision
00:04:17language model as a labeler and the vision language models as judge and then train what you want but how
00:04:24how does this pipeline look like so basically I built this and it has like it has vlm for labeling vlm as a
00:04:35judge and then training uh it's a bit of a long horizon tasks for coding agents and it has infras many
00:04:42infrastructure support you can do this locally you can do this uh remotely so basically it runs on hugging
00:04:48face infrastructure we have jobs that allow you to do one-off batch processing or training
00:04:54we also have like a serverless routing system called inference providers where you can use multiple
00:05:00providers and you we also have like buckets to dump intermediate data on top of the data set
00:05:07repositories model repositories and so on um but what enables this work first off my favorite model rfdtr rfdtr
00:05:17segmentation i'm currently working on segmentation at the moment um we have better agents for long horizon
00:05:24tasks uh that you know you have to babysit the labeling process the training process etc
00:05:32and smaller yet more capable vision models enable you to label stuff very cheaply and also with
00:05:40transformers we went for v5 refactoring etc so like it performs better for vision models at the moment
00:05:47and this is how the pipeline actually looks like so first up i label the data set like i take an image
00:05:54data set like any image data set i labeled that image data set with q1 3.59b and then i passed the
00:06:02labeled data set to two judges first one is gemma4e4b which is like an 8b judge and then the second one is
00:06:10lfm 2.5 vl which is nearly 2b it's relatively smaller basically i checked the research and it's better to
00:06:18have like an ensemble smaller judges and on top of it i merged the judgments so i also checked the
00:06:26research around it and most of the people ask the vlm or llm to assign some score to it but those scores
00:06:34absolutely don't work especially if your models are of different size with judges um and then i pass it
00:06:40to train rfd tr medium or large so i i chatted a bit with robofull people and they encouraged to use
00:06:47that and i'm going to it it actually works i'm going to show you very shortly but how does it work so you
00:06:55take the repository and then you just ask it okay can you train it can you do the training on this data
00:07:04set on hub and then it will start like if the if the data set has labels like you can actually just
00:07:11get to training but if it doesn't have it you can just start annotating and i basically like with the
00:07:19trick is i i passed the overlaid bounding boxes on images to the judge so basically qn technically outputs
00:07:27bounding boxes as tokens i don't pass them i just overlay the bounding boxes and i pass that image
00:07:33on top of some labels and label descriptions and i say okay if this label description has bounding box
00:07:40on it then just um tell me if you approve or not and then the judge i'm i merged the judge verdicts
00:07:49over minimum agreement and not consensus which i will come to why i did that that way and these
00:07:56label descriptions are also generated by coding agents and you just approve it as a human being
00:08:03and the the models that i used in this pipeline all have apache two point of licenses except for
00:08:10lfm model which has type of license where you are kind of if you have like certain amount of revenue
00:08:19after that you pay for it but you can comfortably use it it's large and for the coding agents that are
00:08:26babysitting this pipeline i initially built with opus 4.8 and then ran the workflow with the glm 5.2
00:08:35which is doing a good job on long horizon tasks to be honest and for infra i actually work at hugging
00:08:42phase i have a lot of compute credits and i'm super impatient in life so like i use a good amount of
00:08:50hardware for experimentations but i benchmarked it and overall it takes like three four dollars if you
00:08:58want to run this entire pipeline to train models which to me is crazy initially for q1 3.5 i used
00:09:05serverless because i was like okay this is convenient and it's super cheap so i use deep infra which is
00:09:12super super cheap if you were to use together it's better if you do batch processing over jobs and then
00:09:19for the judging i used hugging face jobs which cost less and then for training again i use like an l4
00:09:27but the model is super small like rfdtr is super small and you can just use something else you can do
00:09:33it locally if you wanted to i'm just impatient i want a big batch size so yeah and i tested in two problems
00:09:42first off the road sign detection uh secondly document parsing for the road sign detection one i
00:09:48already have the labels so like i actually compared against the ground truth annotations if my pipeline
00:09:54works or not and for document parsing i actually couldn't do it because basically i use like a doc vqa
00:10:02data set and i the problem is like i want to extract the images tables i don't know signatures and stuff
00:10:08so it's like a novel task and i wanted to see if rfdtr can actually learn it so first result it works so yay
00:10:19we have like a good mean average precision over 50 and i compare it against basically i have a test set
00:10:29and i take that test set and pass through qn and then i compare against the pseudonotations
00:10:35and the ground truth annotations of that test set there is a bit of a gap but it's kind of expected
00:10:42because it's learned from qn and also rock oak is also like a good value to be frank for like such
00:10:50use case um and for the document parsing it actually generalizes which to me is crazy basically
00:10:58here the trained model output you can see that it detected the signature meanwhile the qn annotation
00:11:06of that test set missed it so i would like to say it actually generalizes very well as well
00:11:12but we owe this to like the how good rfdtr is as a backbone in a way and here you can also see how
00:11:21it captures the images technically and it's like one-on-one and while i was building this i actually
00:11:30noticed that i was clueless about building with vision agents so i have bunch of findings around it
00:11:37so first up there is a huge judging balance so for depending on the problem lfm tends to reject a
00:11:44lot that's why i couldn't take the consensus because if i were to if i were to eliminate everything that
00:11:50both lfm and gemma agreed to remove i would left with very very little number of examples which would
00:11:58leave me with very poor generalization so what i did was that okay if one of them says yes i'm going
00:12:04to take that example and nevertheless it worked well but if you have a large data set and if you care
00:12:09about the recall i suggest that you take the consensus or like just observe for document parsing the gap
00:12:17isn't as big and secondly uh the prompt generation is a bit hard so like this is the only part where as a
00:12:28human you have to approve okay the model generates the prompts for you for the judge uh and then you
00:12:35will say okay this i approve this because you need to take it still take a look at it uh take a look at
00:12:42your data set a little there's no escaping that um and thirdly and this is super interesting because your
00:12:52your coding agent despite no matter how uh good it is like you take opus 4.8 which is like a very good
00:12:59coding agent um it's clueless as a computer vision engineer as well as it misses common sense for
00:13:06instance like it was doing like horizontal flip over the traffic signs or it was doing like jitter over
00:13:14the traffic lights which will definitely corrupt your data sets and break it so i patch this later so you
00:13:21can just say okay i don't want to augment or i can augment so and your coding agent will help you with that
00:13:30and lastly the second part of this toolkit is my preferred models as tools
00:13:35so this repository covers my favorite models from depth estimation to zero shot segmentation
00:13:42and this is partially powered by first of hugging face benchmarks which we recently rolled out maybe a
00:13:48few months ago basically every single basically we have like a benchmark leaderboard and over there you
00:13:55have the open models as well as their evaluation results and you can compare different models of different
00:14:02sizes so um i keep it up to date as well but like also it's partially powered by me who likes to read the
00:14:13computer vision conference papers um so i would like to give a shout out to this model because
00:14:20not a lot of people know about this so basically some cannot do open-ended reference segmentation
00:14:26like you can do like okay segment this red car and it will do it but if you say okay the red car next
00:14:33to the orange car that is next to the blue car it will not do that and falcon perception which is a
00:14:39model by tia can actually do it and it's only like 600 million parameters with apache 2.0 license
00:14:47so this one does the zero shot segmentation for me
00:14:51and this is like a non-exhaustive list so for posing we have the sapiens family
00:14:58for the human centric tasks where you need to do human keypoint detection human
00:15:04depth estimation and so on and for zero shot detection i have moon dream 3 and mm grounding dino which
00:15:13is like a model with apache 2.0 license it's also very good it's very small compared to moon dream i give you
00:15:20the multiple models in multiple sizes depending on your hardware that you can pick like if you want
00:15:25to go fast just to pick pick the tiny alternative for ocr i actually took them from the almost cr benchmark
00:15:34in different sizes and for depth estimation i actually discovered that the large model doesn't
00:15:40have a non-commercial license and the rest of them have it so you can actually use it that one has like
00:15:45an apache 2.0 license and it also comes with supervision and tracker support they are both
00:15:51libraries from roboflow that allow you to do tracking of instances bounding boxes and so on
00:16:00and future plans so first of i i could hear you say okay this will definitely not work for the
00:16:06industry use cases because industry use cases have different parts it has like non-describable
00:16:13parts like natural language is not a good gateway to it so i think in that sense image guided detection
00:16:21could help like if you don't know about image guided detection you basically have like instance of an
00:16:27image like a huggy here as an example and then you ask the model okay detect this object in this image
00:16:36across all of the images i feel like it could actually somewhat help in the industry use cases where you
00:16:41you cannot describe it by like natural language um as well as i want to try um intersection over
00:16:52union merger sort of basically you have like labeled boxes and then judges boxes like you ask the judge to
00:17:00actually generate a box and then you take the intersection over union instead of asking judge to reject or
00:17:06accept and i'm working currently working on the segmentation support and thanks for listening if
00:17:14you want to actually learn more so basically i have a small vision repo story it has everything about
00:17:20like fine-tuning models quantizing models multi-model models everything around vision as well as
00:17:27transformers task guides we keep them up to date has many um tutorials um we also have hugging face skills
00:17:36that actually have computer vision specific skills as well as the infra skills that you can just do like
00:17:43like a one prompt training again um and this is my twitter profile and this repo is actually at github
00:17:51mervenoyan uh vision in turn i think i have um time for one question thank you so much
00:18:04yes um he's asking if i have plans to train vlms themselves like a self-improvement type of thing
00:18:16that would be super exciting but first i want to solve this thing of like developers actually training
00:18:22task-specific models and then deploying on edge and then that could come perhaps maybe one more yes
00:18:34um not really i don't think so i just use the because i wanted because a coding agent actually has the
00:18:44context i wanted it to generate the prompt maybe one more okay thank you so much
00:19:04so

Key Takeaway

Deploying heavy Vision Language Models directly into production is inefficient; developers should instead use VLMs as automated labelers and judges to train small, Apache 2.0-licensed models like RF-DETR for real-time edge deployment at under $4 per pipeline run.

Highlights

  • Vision Language Models (VLMs) cannot deliver real-time performance of 30-40 fps on edge hardware like a toaster, making specialized small models necessary for production.

  • Popular object detection models like YOLO carry GPL 3.0 licensing restrictions that require commercial fees, whereas Apache 2.0 alternatives like RF-DETR offer non-commercial freedom.

  • A self-labeling pipeline using Qwen 3.5 9B as a labeler, Gemma 4 8B and LFM 2.5 VL as judges, and RF-DETR as the target model costs only $3 to $4 to train a custom detector.

  • Falcon Perception provides open-ended reference segmentation at 600 million parameters under an Apache 2.0 license, outperforming SAM on complex multi-object spatial queries.

  • Coding agents lack computer vision domain knowledge and often inject destructive data augmentations like horizontal flips on traffic signs or color jitter on traffic lights.

Timeline

Limitations of Direct VLM Deployment and Licensing Trap

  • Vision Language Models fail to deliver real-time frame rates such as 30-40 fps on edge devices.
  • Specialized models like RF-DETR consistently outperform VLMs in object detection accuracy and robustness.
  • YOLO models carry GPL 3.0 licenses that require commercial payment, necessitating a shift to Apache 2.0 models.

Developers frequently default to deploying Vision Language Models directly for standard tasks like image classification and instance segmentation. However, VLMs cannot achieve real-time performance on edge hardware. Dedicated architectures like RF-DETR deliver superior accuracy and speed while avoiding the restrictive GPL 3.0 licensing fees associated with frameworks like YOLO.

Vibe Vision: Automated Distillation and Selection Criteria

  • Model selection prioritizes Apache 2.0 or MIT open-source licenses over proprietary alternatives.
  • Benchmark evaluations ensure performance matches architectural scale before model inclusion.
  • Automated pipelines replace manual image annotation using VLMs for labeling and validation.

Vibe Vision distills computer vision expertise into a streamlined model selection and training toolkit. Models are evaluated based on non-commercial licensing, benchmark parity relative to size, and overall usability. When only raw images exist, the framework leverages VLMs as zero-shot annotators and quality judges to eliminate manual annotation bottlenecks.

Architecture of the VLM-as-Judge Labeling Pipeline

  • Qwen 3.5 9B acts as the initial dataset annotator by generating bounding box tokens.
  • An ensemble judge system combines Gemma 4 8B and LFM 2.5 VL to evaluate overlaid bounding boxes.
  • Minimum agreement selection replaces consensus scoring to preserve dataset recall and prevent over-filtering.

The automated training pipeline feeds raw images to Qwen 3.5 9B for initial bounding box generation. The system overlays these bounding boxes directly onto the images and passes them with human-approved label descriptions to an ensemble of judges consisting of Gemma 4 8B and LFM 2.5 VL. Using minimum agreement rather than strict consensus prevents excessive filtering and preserves training sample density for downstream RF-DETR models.

Cost, Infrastructure, and Empirical Evaluation Results

  • The end-to-end labeling and training pipeline runs for $3 to $4 on cloud infrastructure.
  • The pipeline achieves a Mean Average Precision over 50 on road sign detection benchmarks.
  • Trained RF-DETR models generalize to unannotated features like signatures in document parsing.

Using serverless routing and temporary jobs on Hugging Face infrastructure keeps total compute costs between $3 and $4 per run. Validation on road sign detection confirmed strong alignment with ground-truth labels, achieving a mAP50 above 0.50. On document parsing tasks, the distilled RF-DETR model demonstrated true generalization by detecting signatures that the initial labeling model missed.

Pitfalls in Coding Agent Augmentation and Model Recommendations

  • Judges exhibit strong rejection biases that require careful threshold tuning to maintain dataset recall.
  • LLM coding agents introduce destructive data augmentations like horizontal flips on directional signs.
  • Falcon Perception handles complex open-ended reference segmentation at 600 million parameters under Apache 2.0.

Discrepancies in judge behavior can drastically reduce dataset size if consensus filtering is applied indiscriminately. Autonomous coding agents frequently lack domain-specific common sense, applying corrupting augmentations such as horizontally flipping traffic signs or color-jittering traffic lights. For specialized tasks, Falcon Perception enables open-ended spatial segmentation at 600M parameters, alongside specialized families like Sapiens for pose estimation and MM Grounding DINO for lightweight zero-shot detection.

Future Directions and Industrial Deployment Strategies

  • Image-guided detection addresses industrial use cases where natural language descriptions fail.
  • Intersection over Union box merging offers an alternative to binary pass-fail judging.
  • Task-specific edge deployment takes priority over training self-improving vision-language models.

For industrial applications with non-describable parts, image-guided detection using visual prompts provides a viable alternative to text prompts. Future pipeline iterations will explore Intersection over Union (IoU) box generation merging directly with judge predictions instead of relying on simple acceptance verdicts. The immediate focus remains enabling developers to deploy lightweight, task-specific models onto edge hardware rather than fine-tuning monolithic VLMs.

Community Posts

No posts yet. Be the first to write about this video!

Write about this video