Meta scans 100M+ videos with small VLMs, not frontier models

Aditya Gautam describes how Meta catches videos that switch topic mid-stream and detects unoriginal AI-copied content across more than 100 million short-form videos. The system relies on three cooperating agents built on small, specialized vision-language models rather than frontier LLMs, because the models never need to write code, only to judge messy user-generated video.

• Messy data — Gautam frames the underlying data problem: over 100 million videos, adversarial uploads gaming the system, multilingual on-screen text, monthly drift as new AI tools appear, and no clear ground truth.
• Two problems — The two problems he works on are modality misalignment (a video is about sports for six seconds, then political for half a second) and unoriginal content, which AI tools make trivial to duplicate and which causes attribution and ecosystem-credit imbalance.
• Intra vs inter — Cross-modality mismatch (image, video, audio, text) is called a solved problem via CLIP-style embeddings and cosine similarity; mismatch within one modality over time is the harder, unsolved one.
• Three agents — A reviewer agent acts as orchestrator, a perceiver agent decomposes video into clips based on temporal change rather than fixed frame rate and emits clip embeddings, tags, OCR, and timestamps, and a retriever agent indexes topics into inverted indexes, embeddings into vector databases, and entities into graph databases.
• Reviewer detail — The reviewer runs temporal analysis on the perceiver's JSON output (e.g., sports until frame ~360/6 seconds, then a shift to political content) and also folds in real-time comments, likes, dislikes, reports, and sentiment.
• Retriever detail — The retriever reranks candidates using classifier-based spam/quality scores and returns similar clips and similar authors to the reviewer for a decision.
• Small VLMs, not frontier — Every agent runs on a small specialized VLM because, as Gautam puts it, he does not care whether the model can solve a coding problem, only the domain-specific task, since it is never exposed to the customer.
• Pretraining — Because frontier models are trained on clean web data that looks nothing like messy user-generated content, the vision transformer is pretrained from scratch on in-house image tokens before fine-tuning.
• Instruction tuning — Instruction fine-tuning trains the model to output a JSON schema covering modality-misalignment labels, duplication scores, and chain-of-thought reasoning for human reviewers.
• DPO loop — A daily production sample passes through an in-house LLM-as-judge (itself trained on human-labeled data) plus a human review queue; failures are traced to the exact node — chain-of-thought reasoning, wrong tool call, or bad retrieval — before retraining.
• Distillation table — Off-policy and on-policy knowledge distillation plus quantization experiments (including float precision choices) produce a table of model sizes, chosen for compute and cost savings in production.
• Evaluation — Beyond precision, recall and F1, Gautam evaluates per-node latency and recall of retrieval, reasoning-budget quality (avoiding overthinking), token cost, robustness/error rates by node, and drift in the LLM-judge versus the human queue.
• Optimizations — Three efficiency tricks: spatiotemporal reduction that compresses similar frames, caching that skips the full multi-agent pipeline for viral content with high similarity scores, and metadata pruning that filters out videos from creators with strong authenticity and engagement records.

In their words

So the real life data is very messy. It is we are talking about at a scale of 100 million plus and a lot more viral content.1:18

Uh the thing is that these VLMs what we have outside and available the foundational model the front end models they are trained on a very clean very nice data set very web data which is very well tuned cleaned and everything but uh data inhouse for a specific purpose is not having the same data characteristics.10:51

It's little expensive but um it if if it can get a delta that that actually works really well.11:30

I really don't care if the model can solve a coding problem. I only care about my domain specific problem.16:10

Disclosure · Gautam describes systems built at his employer, Meta; no separate product or fund is being promoted.

One thing to add — One thing to add — the talk is notably light on hard evaluation numbers (no precision/recall figures, latency numbers, or cost savings percentages are given), so claims of scale and effectiveness rest on Gautam's description of the architecture rather than disclosed metrics. The 95% threshold mentioned for the DPO sampling step is the only concrete performance figure in the transcript.</note> </invoke>

One thing to try tonight
Sketch out a three-agent decomposition (perceiver, reviewer, retriever) for one messy classification problem you have, and write the JSON schema you'd want the perceiver to emit before picking any model.