Voice agent data collection fixed with a 30%-to-95% reframe

Voice agents demo well and then fall apart in production, and Plivo's Venky B says the culprit list is predictable: latency budgets that ignore how LLM thinking has to be turned off, transcription that quietly mangles proper nouns and code-switched languages, and data collection treated as freeform transcription instead of typed fields. His fix for the last one alone reportedly moved accuracy from roughly 30 percent to the mid-90s with no fine-tuning.

• Company context — Plivo has run telephony APIs since 2011, now carries over a billion voice calls a month, has a 90-person team, and holds $50 million in the bank from profits rather than external VC funding.
• Latency targets — Teams aim for under 550 milliseconds time-to-first-audio, most land between 750 and 1200 ms, and beyond 1.2 seconds users start hanging up.
• Thinking is off — Because voice agents need hundreds-of-milliseconds turnaround, almost all of the last year's LLM intelligence gains from reasoning are unusable in production voice pipelines.
• Frontier vs dedicated — Frontier models like OpenAI, Claude, and Gemini run a P50 time-to-first-token around 450-500ms but spike to 1.2-1.3s at P90-P95; Groq and Cerebras need dedicated capacity booked out roughly 12 months in advance.
• Open-source picks — Plivo self-hosts Qwen 3.5 and Gemma 4, targeting under 300ms; Gemma 4 is about 2.5-3x more token-efficient per word for multilingual use, while Qwen suffices for English only.
• Model sizing — A 3-4B mixture-of-experts model gets about 90 percent of the way out of the box, but fine-tuning MoE models is hard and can break them, so domain-specific tuning should start at 8B-12B models.
• Transcription brittleness — State-of-the-art ASR hits 4-6 percent word error rate on clean eval sets but climbs into double digits on noisy real calls, especially with proper nouns, phone numbers, and code-switched languages like Hindi written in Latin script.
• ASR fixes — Recommended fixes are dynamic keyword boosting scoped to specific call states, LLM post-processing of transcripts to correct errors like a stray letter in a phone number, and neural transliteration before sending clean text downstream.
• Data collection reframe — Treating fields as typed data such as phone number and datetime, with validators and per-field unit-test evals rather than end-to-end transcript parsing, took accuracy from about 30 percent to 95-97 percent without fine-tuning.
• Relative values — Ambiguous inputs like next week Wednesday 8 need to be resolved against a known current date inside a constrained datetime field via tool calling, not left to free-text interpretation.
• TTS normalization — LLM output should never go straight to TTS: strip emojis and markdown, use custom pronunciation dictionaries for names and brands, slow speech to 0.7-0.8x for spelled-out entities, and normalize emails, currency, and dates in-house.
• Personal test case — Venky uses his own last name, Balasubramanian, and the company name Plivo, often mispronounced as pivo or pleo, as a working pronunciation check for any TTS engine.
• Skipped topics — Turn detection, barge-in, and back-channeling were flagged as solvable in standard speech pipelines without needing speech-to-speech models, but were left for offline discussion due to time.

In their words

This is where I think 50 to 60% of AI agents mess up pretty badly.17:49

we have seen accuracy grow up from grow from 30% to like 95% from a data collection standpoint when you start thinking in that manner.18:25

almost always your the the LLM or the agent that's talking has to have thinking turned Right. So all the advancements we've had in the LLM layer in the last one year like none of that even apply here now.6:54

the state-of-the-art transcription engines out out in the market u you know sort of get you to four to 6% word error rate right and this is on known eval sets on real world noisy calls with you know sort of uh accents like people having different sort of accents uh domain vocabulary and so on and so forth like those usually end up in the double digits from a word erate perspective right13:39

Disclosure · Venky B is founder and CEO of Plivo, and the talk promotes Plivo's own voice agent platform, models, and infrastructure choices.

One thing to add — One thing to add — the 30-to-95 percent accuracy jump is a striking number but comes from Plivo's own account with no independent benchmark or eval set disclosed, so it should be read as a vendor's case study rather than a verified industry result. The talk is also useful as a checklist precisely because it names concrete failure patterns like stray digits, code-switched scripts, and mispronounced proper nouns rather than general advice.

One thing to try tonight
Pick one field your voice or chat agent currently collects via free-text, such as a phone number or date, define it as a typed field with explicit validation rules, and write a single unit test that checks only that field's extraction accuracy against ten sample utterances.