Why 20% Overlap in Human Calls Breaks Every Voice AI Today

Neil Zeghidour argues that every real-time voice model shipping today, however low-latency, is still a walkie-talkie: it listens or it speaks, never both. He traces four generations of voice tech to show a recurring trade-off — every gain in naturalness has cost intelligence — and proposes splitting the two into separate models rather than waiting for one model to do both.

• Gradium background — Zeghidour is co-founder and CEO of Gradium, a Paris startup spun from the non-profit lab Kyutai, which released Moshi (2024, first full-duplex speech-to-speech model), Hibiki (first real-time speech-to-speech translation system), and an on-device TTS model.
• Siri, 2011 — Siri's 2011 demo ran on a pipeline of speech-to-text plus hand-built natural language understanding with no LLM, mapping transcripts to app actions in a closed-ended dialogue.
• Open-ended chat era — OpenAI's original voice mode (not Advanced Voice Mode) replaced that logic with an LLM, enabling open-ended conversation like a bedtime story about a hedgehog named Larry, but with no agentic capability and latency of several seconds.
• Cascaded voice agents — A live-coded drive-thru bot demo shows a cascaded STT-LLM-TTS agent taking a real order (a classic chicken sandwich and mac and cheese, total of eight dollars), with tool calls and planning but latency and emotion loss from routing through text.
• Speech-to-speech arrives — OpenAI's Advanced Voice Mode collapsed STT-LLM-TTS into one model, cutting latency close to human levels and picking up non-linguistic cues, but Zeghidour says its intelligence is still much more limited than cascaded systems.
• Backchanneling demo — A live demo shows a speech-to-speech model breaking down when the user simply says 'yeah' and 'mhm' the way humans backchannel, because it segments conversation into full turns rather than allowing overlap.
• The audio cost problem — An eight-word, three-second sentence becomes 72,000 timesteps at 24 kHz; since transformer attention cost scales with the square of sequence length, a 10,000-times-longer sequence is roughly 100 million times more expensive to process on raw audio.
• Neural codecs — Neural codecs (audio tokenizers) compress raw audio into a dense token-like representation an LLM can train on, the same approach used in ElevenLabs-style speech-to-speech models, though those still model one sequence and remain half duplex.
• Multi-stream models — Kyutai's multi-stream language models, first shown live in Moshi in August 2024, model two token streams at once so both sides can be active, silent, or overlapping simultaneously — technology Zeghidour says is now used by Thinking Machines' interaction models and likely OpenAI's upcoming bidirectional model.
• Naturalness vs intelligence — Zeghidour states plainly that adding speech modality to a text model's fixed capacity always subtracts from its intelligence, so full-duplex models like Moshi were noticeably 'more stupid' than cascaded systems.
• Two paths forward — Path one scales a single frontier speech-to-speech model end-to-end, which Zeghidour guesses is OpenAI's approach; path two, introduced in Gradium's paper and reused by Thinking Machines, splits a small full-duplex interface model from a background text LLM handling reasoning and tool calls.
• Why hybrid wins — Zeghidour argues the hybrid approach is cheaper than running chit-chat through giant multimodal mixture-of-experts models and preserves the ability to swap backend LLMs, noting Advanced Voice Mode stayed on GPT-4o for a long time even as newer text models shipped.
• Call to action — Zeghidour closes by pointing listeners to gradium.ai and saying the company is hiring research scientists and engineers.

In their words

If you're having a relative on the phone, there is up to 20% of the time where you are both speaking at the same time.9:38

So, the speech-to-speech models are fundamentally still limited compared to the textual models.7:44

Fundamentally, this is uh an issue that can not be resolved when you're using turn-taking.9:12

it was even more stupid than speech-to-speech models that were already less intelligent than cascaded systems.13:32

Disclosure · Zeghidour is CEO of Gradium, the company whose models and research (Moshi, Hibiki, multi-stream models) he showcases throughout the talk, and he directs attendees to gradium.ai and its job openings at the end.

One thing to add — One thing to add — the talk's most testable claim, that adding audio to a text model necessarily subtracts intelligence because capacity is fixed, is asserted rather than quantified with benchmark numbers, so it's worth treating as Gradium's working thesis rather than an established result. It's also notable that Zeghidour openly guesses at competitors' architectures (OpenAI, Thinking Machines) rather than citing confirmed technical disclosures.</note> </invoke>

One thing to try tonight
Call a live speech-to-speech voice assistant (like ChatGPT Advanced Voice Mode) and deliberately say "mhm" or "yeah" while it's mid-sentence to see whether it stops dead — the same backchanneling test Zeghidour ran live.