OpenAI ditched a feedback-loop router that caused traffic to bounce

OpenAI's inference load balancer used to set routing weights through a proportional-controller feedback loop that self-balanced engines but oscillated traffic and defied explanation. The team replaced it with a control plane/data plane split where an optimizer minimizes expected end-to-end latency under hard capacity constraints, rather than looping signals into a score.

• IRB basics — The inference load balancer (IRB) sits between front-end CPU clusters and GPU engine clusters, selecting an engine and proxying each request, with signals like time-to-first-token, time-between-tokens, and KV cache locality all factoring in.
• Old system — Early routing used weighted consistent hashing where weights came from a periodic feedback loop: engines reported signals, a controller smoothed them into a performance score, compared it against the fleet average, and adjusted weight up or down, conceptually a proportional (P) controller.
• Old system's flaws — The feedback loop made routing decisions hard to explain, tuning one property moved another, and shifting traffic off a hot engine cooled it, causing the controller to route traffic back and creating oscillations that disrupted KV cache utilization.
• New architecture — The redesign uses a control plane with a global view of all CPU clusters and GPU engines to compute globally optimized routing weights, and a data plane in each CPU cluster that answers routing questions synchronously from a locally cached snapshot, so no request waits on the control plane.
• Three paths — The system runs three paths: the synchronous inference request path (fast, local to the data plane), the engine signal path feeding both planes, and the routing weight path where the control plane publishes weights the data plane pulls asynchronously.
• Nearest isn't enough — In region one, cluster A sends 90 RPS to a 100 RPS engine and nearest-only works fine; in region two, cluster B sends 120 RPS to a 100 RPS engine B, overloading it, while in region three an 80 RPS engine C is only used at 40 RPS, so sending B's excess 20 RPS to the farther engine C beats waiting on overloaded engine B.
• Optimizer inputs — The optimizer takes four inputs — request volume per CPU cluster, network latency to each engine, available engine capacity and health, and TTFT/TBOT latency profiles — and outputs routing weights per cluster-engine pair.
• Optimizer goal — The goal is to minimize expected end-to-end latency (network distance plus engine-side latency) across all routed traffic, subject to hard constraints: route all demand, stay within effective engine capacity, and keep weights non-negative.
• Penalties — When an engine is an outlier, the system reduces its routing weight, giving it a chance to self-recover or flagging it for human intervention or hardware replacement.
• Retry storms — Retries can worsen failures near system tip-over points, so the team implemented dynamic caps or budgets on retries that tighten under heavy utilization and loosen in normal times.
• Load shedding — As a last resort when capacity can't meet demand, the system proactively sheds a portion of traffic so it degrades gracefully instead of failing entirely.

In their words

So they were generated by a periodic feedback loop. The inference engines as mentioned earlier uh reports all kind of the signals we care about and the controller will periodically smooth out those signals and compute a performance score.4:07

and no this P controller will not help you care a Linux process but instead it's a classic control theory technique4:53

the feedback loop sometimes creates bad oscillations because when you shift an engine away some traffic the engine turns a bit cooler and this signal get fit to the controller. The controller now thinks that hey this engine can take a lot more traffic. then the some traffic going to be shifted back and forth between a few engines and disrupting the KV cache utilization.6:35

Multiple CPU cluster route traffic to the same engines independently which could overload that engine while leave other engines underutilized.8:26

Disclosure · Both speakers work on OpenAI's inference team describing OpenAI's own production system, so the talk promotes OpenAI's internal engineering choices.

One thing to add — One thing to add — the talk is a useful case study in why self-balancing control-loop systems can be operationally worse than explicit optimization even when they require less manual tuning, because unexplainable behavior and oscillation become their own maintenance burden. The concrete RPS example (90/100, 120/100, 40/80) is a clean illustration worth stealing for explaining locality-vs-load tradeoffs to other engineers.</note> </invoke>

One thing to try tonight
Sketch your own service's routing logic tonight and check whether it optimizes for nearest or lowest-latency-only versus accounting for downstream queueing at the destination, the way OpenAI's example showed a farther, less-loaded engine beating an overloaded nearby one.