CoreWeave: 80-90% of Agentic Prompts Repeat, So Caching Beats Compute

Sitanshu Gupta, four months into leading inference at CoreWeave, argues that prefill — not decode — is the most expensive thing an inference stack does, and that most of that cost is avoidable. Because 80 to 90 percent of an agentic request's input matches the prior turn, his team built one platform, not a fork, to route around recomputing it.

• Background — Gupta joined CoreWeave about four months ago after running training and inference at AWS Annapurna Labs and before that at SambaNova.
• Two consumption models — CoreWeave offers serverless, paid per token via API/UI with no hardware management, and dedicated, billed per GPU-hour where the customer picks the engine and controls deployment.
• Provisioned throughput — Within serverless, a provisioned-throughput tier lets customers who know their traffic profile get carved-out capacity to avoid the noisy-neighbor problem, still billed per token.
• Four workload shapes — Agentic, chat, voice/video, and batch are the four workload shapes; agentic and chat share long-input/short-output profiles but agentic turns have no human reading time, so latency budgets are far tighter.
• Batch flexibility — Batch customers hand over 10 to 12 hours of work with loose SLAs measured in minutes or hours, letting CoreWeave schedule the same dedicated GPUs to run real-time traffic by day and drain batch queues overnight.
• Stack architecture — Requests pass through a control plane for authentication, rate limiting and usage tracking, then hit engines including VLM, SGLang, and TensorRT-LLM spread across multiple generations of Nvidia GPUs.
• Cache-aware routing — The router prioritizes KV-cache locality first and least-loaded placement second, a deliberately hard problem across heterogeneous GPUs in different zones and regions.
• Why caching matters — Because prefill is compute-bound and expensive, cached input tokens are priced far below fresh input tokens, which is why hitting the cache is central to the platform's design.
• PD disaggregation is optional — Prefill/decode disaggregation is offered as a capability, not a default, because Gupta says it isn't cheap for every use case, and dedicated customers choose whether to use it.
• Chat cache offload — Between chat turns the KV cache is offloaded to high-bandwidth storage rather than evicted, using techniques like LMCache and Mooncake, so the next question in a conversation loads faster.
• Performance levers — Beyond PD disaggregation and speculative decoding, the two biggest levers are NVFP4 quantization and custom speculators trained asynchronously on a customer's own dataset to raise acceptance length and output throughput.
• Benchmark results — CoreWeave's deployments topped the Artificial Analysis leaderboard on Kimi 2.6/2.7 and, per Open Router data using Weights & Biases branding (CoreWeave acquired Weights & Biases about a year ago), matched speeds close to Fireworks.
• Recap — Gupta's closing point: one platform serving serverless and dedicated, pay-as-you-go and provisioned throughput, with performance optimizations compounding gains across all of it.

In their words

bulk of the input sequence length about 80 to 90% depending on which company it is depending on the customers 80 to 90% of it is the same for various different requests. So there is no point in going in and recomputing the prefill or redoing the prefill for that prefill is supercomputebound very expensive that's why as much as you can hit the cache more you can save7:27

For the multi-end customers, there are separate deployments. Router over here specifically uh the the router is very important since um the router is responsible for making KV cache aware routing choices.6:57

the priority order that we typically take is uh first KV cache locality and then the least loaded fallback.9:49

prefilled decode disagregation is not uh cheap for every type of use case.8:18

Disclosure · Sitanshu Gupta leads inference at CoreWeave and is describing and promoting CoreWeave's own inference platform and benchmark results.

One thing to add — One thing to add — Gupta's cache-locality claim (80-90% input overlap between agentic turns) is presented as an industry-wide pattern but sourced only from his own customer observations, not an independent study. The benchmark comparisons to Fireworks and the Artificial Analysis leaderboard placement are self-reported screenshots from the last month of his team's work, not third-party verified over a longer window.

One thing to try tonight
Check whether your inference provider prices cached input tokens separately from fresh ones, and if you run multi-turn or agentic traffic, measure what fraction of each prompt repeats turn-to-turn to see if you're overpaying for reused context.