One GPU embeds 500K tokens/sec, beating managed APIs

Daniel Svonava argues small open models now match or beat frontier models on specific tasks, so the hard problem shifts from finding a model to serving fleets of dozens of them cheaply. His talk lays out why existing routers and open source serving stacks fail at this and what Superlinked built instead.

• Small model defined — Svonava defines small models as ones that fit on a single two-to-three-generation-old Nvidia GPU, making them cheap and widely available.
• Frontier flattening — On the Artificial Analysis intelligence index, frontier open models like GLM 5.2 (roughly 750 billion parameters) show diminishing returns while small models like Qwen 3 27B catch up, reaching around GPT 5.1 performance.
• Task slicing — Instead of prompting one generalized model, teams should slice workloads into tasks and pick the open model trained for each; a contract review agent Svonava describes runs nine different models.
• Specialist models win — A model trained specifically on Vietnamese receipts will outperform any generalist on Vietnamese receipt OCR, and hundreds of thousands of such fine-tuned models sit free on Hugging Face under permissive licenses.
• Bedrock critique — Svonava says open source AI in most companies just means AWS Bedrock, whose model catalog runs two to three years behind state of the art and doesn't let users own fine-tuned artifacts.
• Untuned tooling — Open source serving tools like vLLM and SGLang ship untuned for any specific model-hardware combination, turning adoption into an open-ended research project rather than a week-long engineering task.
• Router bottleneck — Top-down routers designed to spread one large model across GPUs choke on many small, fast requests because worker state is always stale, capping GPU utilization at 20-30% in Superlinked's tests with vLLM and SGLang.
• LoRA friction — Custom LoRAs and overnight fine-tunes force constant back-and-forth between AI engineers and infrastructure engineers, which Svonava calls the main killer of organizational velocity.
• Superlinked's stack — Superlinked's cluster is Apache 2.0 open source from control plane to GPU, using MessagePack instead of base64 JSON, a NATS JetStream queue handling up to a million requests per second, and a gateway that only annotates requests instead of fully parsing them.
• Queue inversion result — Switching from top-down routing to a shared queue that workers pull from and batch themselves doubled cluster throughput, Svonava says.
• Three runtimes — A Rust sidecar abstracts around 50 model adapters over three runtimes: PyTorch, Candle, and SGLang, with Candle worker images roughly 10% the size of PyTorch's 12GB image, though Candle still lags PyTorch on performance.
• Benchmark numbers — On an RTX Pro 6000, a single GPU can encode roughly half a million tokens per second into embeddings at low tens of milliseconds latency, versus hundreds of milliseconds and orders of magnitude higher cost on managed APIs like Cohere or OpenAI.
• Auto research payoff — Superlinked's auto-research loop produced a LoRA that cost 80 cents to train and improved retrieval quality on German legal text by 18%.

In their words

it's very hard to get your GPU utilization beyond 20 30% under constant load and the problem is that those batches are just not correctly sized basically because you that routing bottleneck.8:21

basically that's like the main killer in organizational uh velocity is talking right like ideally you would want the infrastructure engineers to do their job and you would want to those AI engineers to do their job and they don't have to talk to operate on the day-to-day mode.9:13

imagine you are sitting there now like hitting your text embedding tree on open AAI API instead you could be like having one GPU and push half a million tokens per second into that thing and get the vectors out right19:55

one of the outputs of that was a Laura that took 80 cents to train and it improved 18 it improved quality of retrieval on German legalist STEX as a proof of concept by 18%.23:59

Disclosure · Daniel Svonava is co-founder of Superlinked, the company whose open source inference cluster and benchmarks he presents and promotes in this talk.

One thing to add — One thing to add — the 20-30% utilization figure and the doubled-throughput claim are Svonava's own internal benchmarks against vLLM and SGLang, not independently verified numbers. The talk is also light on latency/cost specifics for the LoRA and generation benchmarks beyond the embedding case, which got the most detailed numbers.

One thing to try tonight
Benchmark a small open embedding model on a single available GPU against your current managed embeddings API for a slice of your own text, comparing latency and per-token cost directly.