Google DeepMind: one model now handles 70+ languages live, no pipeline

Google DeepMind's speech-to-speech leads argue the decades-old pipeline of feature extraction, acoustic modeling, and rescoring is now obsolete, replaced by a single model trained jointly on audio, video, and text. That single model, they say, produces behaviors — like leaving "mid-century" untranslated in a Spanish sentence — that nobody explicitly programmed.

• Old pipeline — Before roughly 2018, speech-to-text required a chain of hand-built components: feature extraction, acoustic modeling, pronunciation modeling, language modeling, and a rescoring pass.
• End-to-end limits — Even post-2018 end-to-end ASR models only mapped speech to a text transcription; they couldn't report tone, emotion, or speed, or respond and translate, without extra engineering.
• Native multimodality — Gemini models are trained on multimodal interleaved examples in pretraining, such as summarizing a bedtime story from a mixed video-and-audio input sequence, unifying ASR, TTS, captioning, and agentic tasks in one token embedding space.
• Live translation — The live model supports streaming translation across 70+ languages, with quality the team says matches offline systems despite translating in real time, demonstrated live on Google Meet across English, Spanish, Italian and Chinese speakers.
• Three-way tension — Valeria Wu Fon frames the design goal as a Venn diagram of conversational (low latency), intelligent (task completion, instruction following), and multimodal (video, screen shares, PDFs in and out), where pushing one knob, like raising thinking budget to boost intelligence scores, degrades time to first audio and naturalness.
• Non-English focus — Wu Fon says the majority of Gemini users are non-English speakers, so the team prioritizes capabilities working across all customer languages, not just en-US.
• Search Live demo — In a Spanish-language demo about a mid-century sofa, the model streams video and audio in, calls a tool to pull up search cards, localizes to Spain Spanish, and leaves the English term mid-century untranslated because that's how Spanish speakers actually use it.
• Roadside assistance — A demo voice agent named Ace, from Auto Transform Insurance Roadside Assistance, looks up a caller by registration plate BD21 XYA and postcode SN48ZX instead of a policy number, showing alphanumeric accuracy and proactive audio that ignores background noise like a passing car instead of cutting itself off.
• Avatars pilot — The team launched a pilot supporting customizable real-time avatars at Cloud Next, from hyperrealistic humans to cartoons, with low-latency multilingual lip-syncing powered by the same speech-to-speech model.
• Finance demo — A closing demo shows an assistant tracking a user's college fund, greeting a second person named Lisa, then switching into Spanish to discuss volatile tech and communications sector performance, combining tool-calling, multimodality, and fluid code-switching.
• Closing claim — Wu Fon closes with the line that AGI will not be typed, it will be spoken, framing seamless switching between translation, action, and brainstorming as the team's goal.

In their words

They're only doing kind of one thing, which is speech to a transcription of that speech. They're not responding, they're not translating.2:15

And it's translating all of them to your language as they talk.4:31

we're finding is the translation quality for this like streaming real-time translation is about as good as you would get with offline systems4:34

once you move one of the knobs, it's very easy for the other knobs to kind of like mess up7:37

Disclosure · Both speakers work at Google DeepMind on the Gemini speech-to-speech model and are presenting their own team's research and product demos.

One thing to add — One thing to add — the talk is light on hard benchmark numbers (only "70+ languages" is quantified), so most of the "no rule was written for that" claims about behaviors like the mid-century example rest on the speakers' own characterization rather than shown evaluation data. It's also worth noting the tension they admit but don't resolve: raising thinking budget improves intelligence scores but breaks latency, and they offer no metric for how close current models get to solving that trade-off.

One thing to try tonight
Try prompting a multimodal voice assistant, such as Gemini Live, in a non-English language with a mixed-language technical term, for example asking about a mid-century chair in Spanish or French, and see whether it keeps the loanword untranslated the way the demo describes.