Roku engineer cuts voice-assistant pain 40% without touching accuracy

Amit Desai argues that voice AI teams obsess over one knob, model accuracy, while ignoring a second, independent knob: what the system does when it's unsure. Holding accuracy fixed at 79 percent, he shows that choosing the right confidence thresholds for stopping, confirming, or acting cuts a "user cost" metric by nearly half.

• Background — Desai worked on voice AI at Alexa, Roku, and his own startups before this talk.
• The premise — accuracy and system decision are two orthogonal knobs for improving voice assistant satisfaction.
• Baseline example — a hypothetical smart speaker called 'Alexa' plays music with 79 percent accuracy: 790 of 1,000 requests get the right song, 210 the wrong one.
• First fix — adding a 'stop' behavior (saying 'sorry, I didn't get that') requires a confidence threshold t, and intuition picks 65 percent, which is not optimal.
• OUCH heuristic — Desai assigns unit costs: a wrong song costs 10 seconds of user pain, a stop costs 4 seconds, and total cost is named the Outcome User Cost Heuristic (OUCH).
• Optimization result — minimizing OUCH shows the optimal threshold is 43 percent, not 65 percent, cutting cost from 2.1 to 1.27 ouch points per turn.
• Third behavior — adding 'confirm' (system says 'play Kiss by Prince?') splits the range into two thresholds, t1 and t2, for stop, confirm, and act.
• Confirm costs — confirming and getting a 'yes' costs 2 seconds; confirming and getting corrected costs 6 seconds.
• Second optimization — the AI-computed optimal thresholds are 41 and 49 percent, dropping cost to 1.464, later refined to 1.26 with further tuning.
• Real systems caveat — production systems would use a real-time learned decision model rather than an offline fixed threshold, but the principle holds.
• TV extension — on a TV AI assistant, multimodal interfaces (showing choices like 'ABC News Live' visually) change the cost values because confirmation via remote control is less painful than by speech.
• Embodied AI stakes — Desai warns that as AI assistants take physical or digital actions (e.g., a robot throwing out a watch, sending an email), the cost of wrong 'acts' grows, raising the stakes for this second knob.

In their words

there are two ways to improve customer or user satisfaction of a voice AI assistant and that is by increasing accuracy which people know about I mean technically accuracy and the other is a different knob that we have that we are not using adequately2:13

Any tea we choose is producing bad outcomes.7:13

we might say something like okay I don't know let's do 65%. It seems you know gut feeling like okay it's kind of confident that's probably when we should speak.6:52

it turns out 43% is in fact the optimal optimal point of t. This minimizes the cost function.11:31

Disclosure · Desai works on voice AI at Roku; the talk implicitly promotes his professional approach and framework but does not pitch a specific product or course for sale.

One thing to add — One thing to add — Desai's numbers (10 seconds for a wrong song, 4 for a stop, 2 for a confirmed yes) are self-described heuristics, not measured user data, so the "optimal" thresholds he derives are only as good as those guesses. Still, the core structural point — that confidence thresholds should be chosen by minimizing a cost function rather than picked by gut feel — doesn't depend on getting those specific numbers right.

One thing to try tonight
Take any AI assistant feature you're building that acts on a confidence score, write down a rough cost-in-seconds for each bad outcome type, and plot total cost against threshold value instead of picking a threshold by intuition.