In short
- Four infrastructure decisions quietly decide whether a voice AI launch survives contact with real customers
- Stitching together your own carrier, transcription, voice engine and language model means owning every latency and guardrail problem yourself
- A single AI layer can be talked into a bad answer, a second independent layer needs to check every response before it’s spoken
- A transcript shows what was said, not why the AI decided it, and that gap is where genuine improvement happens
- Voice AI has to start forming a response before the caller finishes speaking, the same way people do in a real conversation
Most conversations about why voice AI projects fail focus on the business case: the wrong goal, the wrong starting point, no way to measure success. Fewer focus on what happens in the weeks before the first call ever reaches a customer, when a handful of infrastructure decisions get made quickly, often by whoever’s available, and quietly decide whether the whole system holds up. Across enough of these deployments, the same four mistakes show up more than any others, and the first one tends to cause the other three.
Building your own voice AI stack means owning the carrier, the transcription, the voice engine, and the language model, plus every failure between them
Every voice AI system needs the same shortlist of components:
- A carrier, to actually carry the call
- A speech-to-text engine, to transcribe what’s being said
- A voice engine, to generate the spoken response
- A language model, to decide what that response should be
Each of these is genuinely excellent as a standalone tool, which makes stitching them together look like an integration exercise rather than a specialism in its own right. The moment you do it yourself, you own:
- Every piece of latency running through that chain, on a channel where a two-second gap already feels like a hang-up-worthy delay
- Guardrailing
- Observability
- Carrying the voice service itself end to end, the unglamorous telephony-grade reliability work that has nothing to do with AI and everything to do with why a call on a train still drops out
Simon Woodward, Natterbox’s Director of AI Solutions, has a line for this that gets repeated back to him more than most:
“It’s the equivalent of saying I’ll just go on holiday, order all the parts, and build an aeroplane. I’m not a qualified pilot, and my plane would probably be rubbish.”
There’s a reason specialists exist for each part of this stack, and assembling your own version from parts skips past that expertise entirely. This is mistake one, and it rarely stays contained. The next three tend to arrive as direct consequences of building it yourself rather than choosing a platform that’s already absorbed them.
| Responsibility | DIY stack | A platform built for this |
|---|---|---|
| Latency across every component | Yours to manage | Engineered in before the first call |
| Guardrails against a talked-into-it answer | Yours to build | An independent AI layer, built in |
| Observability into why the AI decided something | Yours to instrument | Native, on every call |
| Carrier-grade call reliability | Yours to guarantee | Backed by 16+ years of carrier-grade telephony |
A single AI layer will talk itself into a bad answer, it needs an independent layer checking it before the customer hears it
A well-written prompt behaves like a guardrail right up until someone tests it properly. The same model handling a conversation can be talked into offering a discount it shouldn’t, sharing information it shouldn’t, or simply drifting off-script, especially by a determined caller. Text-based chatbots getting talked around this way is well documented. On a voice channel, the same risk exists and matters just as much, arguably more, because there’s no screenshot afterwards proving what was actually said.
The fix is a second, independent language model sitting above the conversational one, adjudicating every response before it’s delivered rather than just generating it. Simon puts it simply:
“It’s essentially like having a manager on the line, whispering to the employee: hey, don’t say that, you can’t offer this discount.”
A separate model has no investment in defending the first one’s decisions, which is exactly why a determined caller can’t talk it around the same way twice. It’s also part of why >93% of AI Agent calls get routed correctly across the Natterbox platform, before anything reaches a human queue.
Most call logs show what was said. The ones that improve an AI system show why it decided that.
A transcript tells you what a customer heard. It doesn’t tell you:
- What tools the AI called
- What alternatives it considered
- Why it landed on the answer it gave
For refining prompts and catching drift over time, that distinction is nearly everything. In practice, this looks like the same AI layer reviewing a finished call and flagging something like a missed opportunity to lock in a delivery date, alongside an example of what a stronger response would have looked like. That’s the difference between a system that simply ran and one that’s actively getting better call over call. It also compounds at scale: reviewing 100% of calls this way, rather than the small sample a human QA team might get through, means patterns surface in weeks that would otherwise take months, if they surfaced at all.
Voice AI that waits for a caller to finish speaking before responding is already too slow
Nobody waits for the other person to finish a sentence before starting to think about their reply. If someone starts asking how you are, you’re already forming “fine thanks, you?” before they’ve finished asking it. Voice AI has to work the same way: streaming a response and starting to formulate it while the caller is still speaking, rather than processing for two or three seconds after they stop, which feels glacial on a call in a way it never would over text.
The complication is what happens when the AI predicts how a sentence will end, starts forming a response accordingly, and the caller finishes it differently. This is exactly where the independent layer from the second mistake earns its keep again: taking one last look at the response against what was actually said, and correcting it before anything reaches the caller if the prediction was wrong. Treating voice like a slower version of text chat is the underlying mistake. The fix is architectural, faster models alone won’t solve it.
Danone picked voice because it’s the hardest channel to get right, and built these safeguards in before going live
Each of these mistakes looks reasonable in isolation:
- Stitching tools together looks efficient
- Trusting one well-prompted model looks sufficient
- A transcript looks like enough of a record
- Waiting for a full sentence looks like good manners
Put together, they’re why so many voice AI pilots work beautifully in a demo and fall over in production.
Danone’s Regional CIO for Northern Europe, Artem Gordiyenko, picked voice as an early AI priority for exactly the reason most businesses put it off: it’s hard to get right. The safeguards covered above, an independent layer checking every response, full observability into why the AI decided something, latency engineered in rather than patched on afterwards, are what let that bet pay off. Two months after Danone’s AI agent went live in one of the more safety-critical settings voice AI operates in anywhere, it had handled more than 32,000 patient calls. Artem’s own verdict on the result: “the quality of what we see coming out of AI is clearly higher than we see from the people.” That’s a CIO running a regulated, healthcare-adjacent contact centre, about as demanding a bar as this kind of infrastructure decision gets tested against.
This piece draws on Simon Woodward’s session on deploying customer-facing voice AI, including the live product demo and audience Q&A. The complete session is available to watch on demand: Watch the webinar replay.
For more results like Danone’s, see our customer stories.