Inception's diffusion LLM matches GPT-2 quality at 10x the speed

Stefano Ermon, a Stanford professor and one of the researchers behind diffusion models, argues that the autoregressive architecture behind nearly every frontier LLM is fundamentally mismatched to how GPUs work at inference time. His startup Inception is betting that diffusion-based language models, which generate many tokens in parallel instead of one at a time, can win on speed and efficiency without giving up quality.

• Origins — Ermon started as a Stanford assistant professor in 2014 working on generative models when the field was unfashionable, training small models on MNIST just to generate grainy digit images.
• Diffusion's start — With a PhD student in 2019, Ermon's lab developed score-based generative models — training a neural network to denoise images — which became the basis of modern diffusion models used in Stable Diffusion and Midjourney.
• 2024 breakthrough — In 2024, Ermon's team published a paper showing a diffusion model at the GPT-2 scale (under a billion parameters) matched autoregressive perplexity while generating text about 10x faster.
• Founding Inception — Ermon founded Inception, now about two years old with roughly 50 people, to scale this diffusion-based text and code generation approach commercially.
• Core argument — He says autoregressive inference is sequential and memory-bound, mapping poorly to GPUs, while diffusion models process many tokens in parallel, making them better suited to inference-time scaling, RL rollouts, and post-training.
• Mercury models — Inception's Mercury models are benchmarked as comparable in quality to speed-optimized models like Haiku, Flash, and OpenAI's mini/nano models, but faster, and the company built its own custom serving engine since diffusion LLMs can't run on vLLM or SGLang.
• Customer example — Voice-agent company OpenCall switched from serving autoregressive LLMs on Cerebras custom chips to Inception's diffusion models, getting comparable speed on standard Nvidia GPUs at lower cost and higher availability.
• Discrete vs continuous — Extending diffusion from continuous data (pixels) to discrete data (words, which can't be interpolated) required new research; Ermon says the resulting models still achieve GPT-2-level perplexity, indicating comparable data compression and structure discovery.
• Controllability — Ermon argues diffusion models are easier to steer than autoregressive ones because they generate coarse-to-fine, allowing mid-generation scoring against constraints or reward functions, versus waiting for a full autoregressive output.
• Data efficiency claim — He cites academic evidence that diffusion models may be more data-efficient than autoregressive ones because denoising acts as a form of data augmentation on the same training image.
• Market sizing — Using OpenRouter's task taxonomy, Ermon estimates 20-30% of workloads are latency-sensitive enough to be addressable by latency-budgeted diffusion models as a lower bound.
• Trade-offs — Inception built its own SFT, RLHF, and RL stack from scratch since no mature open-source ecosystem exists for diffusion LLM serving or kernels, keeping IP in-house but limiting community adoption.
• Academic roots — Ermon points to FlashAttention and DPO as other examples of industry-defining ideas that originated in his and colleagues' academic labs, arguing academia still enables contrarian bets industry labs won't take.

In their words

We were able to generate text like 10x faster compared to the autogressive model.5:28

You cannot generate the 10th token until you've generated everything that comes before it.8:47

Our Mercury models are on par with the haiku models, flash models, mini nano models from OpenAI if you look at benchmarks uh while being significantly faster.12:21

We are not able to share much about the the the the training uh the size of the models or the flops or all of that it's kind of like a trade secret30:36

Disclosure · Ermon is co-founder and CEO of Inception, the company whose diffusion-based Mercury models are the subject of the interview.

One thing to add — One thing to add — Ermon repeatedly concedes that key claims (diffusion models being more data-efficient, or eventually more intelligent than autoregressive ones at scale) are unproven and "we don't know," which is worth weighing against the more confident speed and cost claims he backs with specific benchmarks and a named customer. The 2024 paper result he cites was at GPT-2 scale (under a billion parameters), still far from frontier model sizes.</note> </invoke>

One thing to try tonight
Compare response latency between a diffusion-based model (Inception's Mercury via its API or playground) and a similarly-sized autoregressive model like GPT-4o-mini or Claude Haiku on the same coding or JSON-extraction prompt, timing tokens-per-second directly.