Homa cuts AI cluster tail latency 13x by ditching TCP

John Ousterhout argues that AI networking's old assumption — that throughput is all that matters — no longer holds as inference and agentic workloads generate small, latency-sensitive coordination messages. He presents Homa, a clean-slate Stanford transport protocol, as a replacement for TCP and RDMA that he claims cuts tail latency by an order of magnitude.

• Old workloads — Ousterhout says AI traffic historically consisted of gigabyte-scale transfers like weight gradients, where throughput was the only metric that mattered and TCP/RDMA (specifically RoCE) performed well.
• Shift to small messages — He says inference and agentic workloads now generate frequent small exchanges — KV cache lookups, barrier synchronization — where round-trip latency, not throughput, is what matters.
• Tail latency stalls GPUs — He describes multi-node workloads where all GPUs must finish a small synchronization step before the next compute phase, so as compute phases shrink to millisecond scale, millisecond-level sync delays waste a significant fraction of GPU resources.
• Audience poll — Ousterhout polled the audience on whether small-message latency was already limiting their application throughput and reported more hands raised than he expected.
• Incast congestion — He explains that incast — multiple senders converging on one destination — causes packets to queue at the top-of-rack switch's egress port, where short messages get stuck behind long ones, and buffer overflow triggers drops, timeouts, and retransmissions.
• Sender-side control fails — TCP and RDMA rely on senders to detect congestion via dropped packets or ECN marking relayed back from switches, a process he says takes multiple round trips and causes oscillation between overshooting and undershooting rates.
• 20 years unsolved — He notes this sender-driven congestion control problem has been studied in research for more than 20 years with many published papers but still no protocol that performs well.
• Byte-stream limits — TCP and RDMA treat data as an undifferentiated byte stream, so they can't identify message boundaries, can't prioritize short messages, and suffer head-of-line blocking when short messages queue behind large ones.
• Homa origins — Homa began as Benam Montazeri's PhD dissertation at Stanford, and Ousterhout says the results were promising enough that he built a Linux kernel module, now on GitHub and in the process of being upstreamed.
• Message-based design — Homa's core unit is an RPC (request plus response message), it tracks message lengths throughout the transport, and uses SRPT (shortest remaining processing time first) so shorter messages can bypass longer ones.
• Receiver-driven control — Unlike TCP/RDMA, Homa controls congestion from the receiver: senders transmit only initial 'unscheduled' packets, and receivers pace subsequent 'scheduled' packets via grant packets, delaying or prioritizing them to favor shorter messages.
• Priority queues — Homa uses the roughly eight priority queues available in modern switches, dynamically assigning short messages to higher-priority queues so they bypass congested queues holding long messages.
• Benchmark results — In Ousterhout's benchmark across message sizes from 50 bytes to a megabyte, Homa's P99 tail latency for short messages was under 100 microseconds versus over 1 millisecond for TCP — about 13 times faster — while Homa was also nearly twice as fast as TCP on the longest messages.

In their words

So with TCP it's more than a millisecond tail latency. Home is less than 100 microsconds about 13 times faster.16:36

Even on the longest messages, Hom is almost a factor of two better than TCP.16:56

So I'm curious. I'd like to just do a a quick audience poll here. Is there anybody here where you have reason to believe that the latency of small messages is impacting the overall throughput of your applications?5:21

You need all of those exchanges to complete before you can go on to the next phase of computation.4:43

Disclosure · Ousterhout is the creator of Homa and describes it as his current full-time project (he says he semi-retired from Stanford to work on it); he explicitly invites attendees to contact him for help adopting it.

One thing to add — One thing to add — Ousterhout presents only his own benchmark and offers no independent third-party validation of the 13x figure or comparisons against newer TCP congestion-control variants, so the numbers should be read as a proponent's best-case demonstration rather than a neutral evaluation. It's also worth noting Homa is still pre-upstream in the Linux kernel, meaning production adoption requires nontrivial infrastructure work.</note> </invoke>

One thing to try tonight
Check whether your own AI inference or agentic pipeline logs P99 latency for small coordination messages (KV cache lookups, barrier syncs) separately from bulk transfer throughput, since that split determines whether a protocol like Homa would even help you.