TurboQuant cuts KV cache bits but halves data-center speed

Philip Kiely's book "Inference Engineering" came out February 23, but the field moved fast enough that he's already issuing a public addendum. The headline reversal: TurboQuant, the viral four-bit KV cache quantization technique that briefly dented memory stocks in March, doubles effective bandwidth but cuts tokens per second by more than half in the data center — making it great for local inference on memory-starved home GPUs and useless for production serving.

• Book stats — Published February 23, 2026; over 11,000 paper copies, nearing 30,000 digital copies, and seen by roughly 24 million Twitter accounts.
• Local vs data center — Local inference is get it working then make it less dumb (quantization, distillation, pruning at batch size one); data center inference is get it working then make it less slow (KV-aware routing, speculation, disaggregation).
• Training blu — Training blurs into inference — Faster inference generates more data, which trains better models, which enables faster inference, a cycle Kiely calls training for inference.
• TurboQuant math — Uses polar coordinates to quantize the KV cache to four bits, halving memory and doubling effective bandwidth, but the extra forward-pass computation during decode cuts TPS by more than half, unacceptable for Baseten's production workloads so they stayed on NVFP4 weight quantization.
• Where TurboQ — Where TurboQuant fits — Ideal for local inference on memory-limited home GPUs running long-context models, where freeing KV cache memory matters more than the TPS hit.
• KV compaction — Baseten's research team built Still, a learned bottleneck that cross-attends fixed query vectors against the full KV cache and emits compact keys and values in a single forward pass, amortizing compression via training rather than at inference time like Attention Matching or Cartridges.
• Speculation history — Small same-family draft models (Spec) underperformed; Medusa added decoder heads; Eagle 3 trained a billion-parameter model on the target model's hidden states and became the best method by February 2026.
• Dlash arrives — A diffusion-based drafter that predicts eight or sixteen tokens per forward pass instead of one, using bidirectional attention with causal-consistency masking; on a single B200 running Qwen 3 8B it delivered more than 3x improvement over Eagle in production.
• DSpark — Days-old research pairing a diffusion model with a sequential model to boost acceptance rates further; Baseten has no production results yet.
• Continuous s — Continuous speculator retraining — Retraining the Dlash speculator on live prompts and responses lifts token acceptance rates 20% to 2x, though it demands heavy storage, compute, and data-use permissions.
• What's next — NVFP4 performance should shine on Nvidia's upcoming Rubin systems; disaggregation and system-wide KV cache movement (PD disaggregation) are gaining traction; training-for-inference will keep growing in importance.

In their words

You know, there's two types of inference engineering that have really emerged. There's local inference where the overwhelming strategy is just get it working on whatever hardware you have by squishing the model... and first you get it working and then you make it less dumb... and then there's my world which is the batch size and data center world where it's get it working... and then make it less slow.2:28

What I've found more and more recently is that many optimizations for inference come from a dedicated training process. And so the lines between training and inference are getting blurriier and blurriier.3:52

Turns out that you need to do additional computation in the forward pass to account for this during decode and it cuts TPS by more than half and that's just an unacceptable trade-off for a lot of the production use cases.7:08

Turns out that small models are like not great draft token generators. They're great small models.12:31

Disclosure · Philip Kiely works at Baseten and is promoting his company's book "Inference Engineering," Baseten's research (Still, Dlash production results), and giving away free copies at the Baseten booth.

One thing to add — One thing to add — the talk is Baseten's internal engineering roadmap dressed as an industry update, so claims about what "works" versus what's abandoned reflect their production choices rather than a neutral survey of the field. Numbers like the 3x Dlash improvement and 20%-to-2x retraining gains come from Baseten's own benchmarks and haven't been independently verified.

One thing to try tonight
If you're running a local LLM on limited GPU memory, look up TurboQuant's four-bit KV cache quantization for your setup — it trades some tokens-per-second for roughly double the effective memory bandwidth, which can let you fit longer context windows tonight.