Meta: 7 axes, not 3, now decide where an AI request runs

Meta engineers argue inference has quietly become a distributed-systems problem bigger than the largest microservices it has ever run. The fix isn't a faster model or kernel — it's a control plane that treats GPUs, KV cache and tokens as scheduled resources, and optimizes cost per successful task instead of cost per token.

• 2008 parallel — Nishant Gupta compares today's AI infrastructure to cloud circa 2008, when value moved from virtual machines up to schedulers (Borg, Kubernetes, Mesos), service meshes and autoscalers — a shift he says AI is repeating in a few years instead of a decade.
• Demand math — Classical web serving scaled roughly linearly with users, but agentic serving scales as users × calls-per-user × tokens, going from ~1 model call per chatbot turn to 10-20 for co-pilots, 50 for research agents, and thousands for autonomous workflows.
• Request shape — LLM requests range from 50 to 100,000 tokens with distinct prefill/decode compute profiles, require continuous in-flight batching (or throughput collapses by an order of magnitude), and carry expensive per-request KV cache state that classical stateless microservices never had.
• GPU economics — GPUs are roughly 100x more expensive and 10x slower to acquire than the CPUs microservices ran on, so casual overprovisioning causes major wastage, and a mid-decode preemption can drop thousands of in-flight tokens.
• Coupling problem — Gupta's central claim: no single layer (routing, caching, scheduling, autoscaling) is new, but a routing decision now changes cache hit rate, which changes batch composition, which changes GPU utilization, which changes the autoscaler's next move — all entangled.
• Distributed transaction — A prompt passes through gateway, router, cache lookup, scheduler, and runtime (vLLM, SGLang); each hop can retry, time out or fail, and once 200 tokens have streamed to a user a GPU preemption can't simply be retried — reliability has to live in the control plane.
• 7-axis scheduling — Gupta says an inference scheduler must be aware of GPU generation (H100 vs A100 vs B200), memory headroom, KV cache state, whether weights are warm or cold, tenant priority, latency budget, and workflow context — since failing step three of five wastes what steps one and two already spent.
• 4-quadrant framework — Every optimization fits one of four moves: avoid the work (prefix/response/semantic caching), share it (continuous batching, chunked prefill, speculative decoding), move it (route to a cheaper model or region), or delay it (admission control, deadline-aware queueing).
• Cost metric — The metric to optimize is cost per successful task, not cost per token or per request, since that's what actually tracks user-facing value.
• Cascading failure — Gupta traces the classic failure loop — GPU degrades, latency rises, clients retry, queue depth grows, healthy GPUs saturate — complicated in agentic serving because a cold pool must rewarm KV cache before absorbing traffic shed from a hot pool.
• Naman's metrics — Naman Ahuja lists core observability metrics: time to first token, utilization ratio, success per dollar, and end-to-end request latency, framing the latency/cost/throughput tradeoff as analogous to the CAP theorem.
• Control plane thesis — Ahuja argues routing, batching, caching, scheduling and reliability are converging into one logical inference control plane, just as VMs converged into Kubernetes, with models, GPUs, KV cache and tokens all becoming scheduled resources.
• Operating lessons — Ahuja's takeaways: infrastructure bottlenecks show up before model bottlenecks in production, adding GPUs hides rather than solves scheduling problems, and control loops beat manual processes — the next phase after better models and faster inference is orchestration.

In their words

LLM serving requires continuous inflight batching how otherwise the throughput collapses by an order of magnitude3:26

for model inferencing we require to run on GPUs which are 100 times more expensive which are 10 times slower to acquire3:55

the bottleneck is not just the model, it's the orchestration itself4:36

it's not important to optimize just cost per token or cost per request. We have to optimize cost per successful task because this is what actually users care about12:04

Disclosure · Both speakers work on training and inference infrastructure at Meta and are describing systems and frameworks built there; the talk functions as a showcase of Meta's internal approach rather than a vendor pitch, but readers should note the institutional lens.

One thing to add — One thing to add — the talk is long on framework (four quadrants, seven axes, CAP-theorem analogy) and short on hard numbers from Meta's own production systems, so treat the specific multipliers (100x GPU cost, order-of-magnitude throughput collapse) as illustrative rather than measured figures. The most exportable idea for any team running LLM inference is the shift from cost-per-token to cost-per-successful-task as the north-star metric.</note> </invoke>

One thing to try tonight
Pick one production LLM endpoint you operate and compute its cost per successful completed task (not per token or per request, counting failures/retries as wasted spend) for the last day of logs to see how far it diverges from your current cost-per-token dashboard.