Vibe-training pipeline builds a detector for $3-4, no VLM in production

Source · Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face
AI Engineer · Merve Noyan · Hugging Face · 2026-09-23 uploaded · 19min

한국어·English

Merve Noyan argues developers should stop deploying vision language models directly because they can't hit real-time speeds and often carry hidden licensing costs. Instead she has a coding agent orchestrate cheap open VLMs to label data, judge each other's labels, and train a small detector that beats them at inference.

The complaint — Noyan says vision language models never hit real time — you get 30-40 fps on toaster-class hardware whatever the task, and a trained RF-DETR will always outperform them.
Licensing warning — She warns that YOLO carries an AGPL 3.0 license and claims some developers deploy it commercially without realizing they owe payment, urging migration to Apache 2.0 models.
The toolkit — Inspired by a Mazar post pairing SAM 3.1 with Gemma 4 as a tool-caller, Noyan built a toolkit that hands her favorite Apache 2.0/MIT models to a coding agent she calls 'a clueless computer vision engineer.'
Pipeline design — The pipeline labels unlabeled images with Qwen 3.5 9B, passes overlaid bounding boxes to two judges — Gemma 4 E4B (8B) and LFM 2.5VL (~2B) — merges verdicts by minimum agreement rather than consensus, then trains RF-DETR medium or large.
Cost and infra — The whole run costs about $3-4 using Hugging Face jobs, inference providers like DeepInfra and Together, and an L4 GPU for training RF-DETR.
Agent stack — She built the pipeline first with Opus 4.6/4.8, then ran the workflow with GLM 5.2, which she says handles the long-horizon babysitting well.
Road sign test — On road sign detection, checked against real ground truth, the trained detector reaches a good mean average precision at 50, with an expected gap since it learned from Qwen's pseudo-labels rather than truth.
Document parsing test — On a doc-QA dataset for extracting images, tables and signatures, the trained RF-DETR caught a signature that the Qwen labeling model itself had missed.
Judge imbalance finding — LFM rejects far more examples than Gemma, so taking consensus of both judges would have left too few training examples and hurt generalization.
Human-in-loop finding — Judge prompts are generated by the coding agent but still require human approval before use.
Agent blind spots — Even Opus 4.8 as coding agent showed no computer-vision common sense, horizontally flipping traffic signs and color-jittering traffic lights during augmentation until Noyan patched the pipeline to disable that.
Models-as-tools list — The second half of the toolkit offers her preferred Apache 2.0/MIT models by task, including Falcon Perception, a 600M-parameter model from TIA that does open-ended referring segmentation SAM cannot, plus Sapiens, Moondream 3, MM Grounding DINO, and OCR and depth models curated from Hugging Face's benchmark leaderboard.
Next steps — Future plans include image-guided detection for hard-to-describe industrial objects, an IoU-based judge merger instead of accept/reject, and segmentation support she is currently building.

In their words

it will always outperform your vision language model and I'm going to prove it today.1:23

Merve Noyan slide · Skill issue: stop deploying vision language models, use them 1:23 — Merve Noyan slide · 1:23 · AI Engineer

You can actually use a vision language model as a labeler and the vision language models as judge and then train what you want.3:58

Merve Noyan slide · Skill issue: stop deploying vision language models, use them 3:58 — Merve Noyan slide · 3:58 · AI Engineer

I merge the judge verdicts over minimum agreement and not consensus which I will come to why I did that that way7:29

Merve Noyan slide · Skill issue: stop deploying vision language models, use them 7:29 — Merve Noyan slide · 7:29 · AI Engineer

LFM tends to reject a lot. That's why I couldn't take the consensus because if I were to if I were to eliminate everything that both LFM and Gemma agreed to remove, um I would left with very very little number of examples11:26

Merve Noyan slide · Skill issue: stop deploying vision language models, use them 11:26 — Merve Noyan slide · 11:26 · AI Engineer

Disclosure · Noyan works at Hugging Face and is promoting her own open-source toolkit built on Hugging Face jobs, inference providers, and models hosted on the Hugging Face Hub.

One thing to add — One thing to add — the road-sign and document-parsing results are self-reported and compared against pseudo-labels from the same labeling model in one case, so the generalization claims deserve independent replication before treating them as settled. The $3-4 cost figure also depends on Noyan's specific choice of serverless providers and GPU size, so actual costs will vary with dataset size and hardware choice.</note> </invoke>

One thing to try tonight
Try pointing a coding agent (Claude or similar) at an unlabeled image folder using Noyan's approach: have it call an open VLM like Qwen to overlay bounding boxes, then manually inspect a handful of the generated label descriptions before running any training.