Why Voice Agents Fail in Real Conversations

A caller says: “Can we do Tuesday... um... actually, Wednesday? After three, if you have it.”

A person hears one request. A voice agent receives a stream of audio frames, a pause, a partial transcript that says Tuesday, a second fragment that says Wednesday, and possibly its own voice leaking back through the phone line. Each of those can produce a different failure: the agent answers during the pause, books Tuesday, ignores “after three”, or stops talking because it heard itself.

Part 1 covered how to choose the architecture. This part follows the conversation itself. It connects the symptom a caller notices to the mechanism behind it and the evidence needed to diagnose it. The booking scenario is illustrative.

Part 2 of Voice Agents in Production. Pauses, backchannels, corrections, names, numbers, noise and long calls, from symptom to mechanism.

Conversation runs on a clock

In a streaming cascade, the wait after a caller finishes can include endpoint detection, final recognition, model generation, tool execution, speech synthesis and playback buffers. Some stages overlap, so adding their full durations can overstate the delay. A trace should show which work actually blocked the first audible response.

Measure two delays separately: end of caller speech to first audible response, and end of caller speech to the first useful answer. “Let me check” improves the first while the availability lookup continues. Record typical turns and the slow tail separately, including turns with retries and corrections. A quick acknowledgement can hide a slow answer.

This distinction becomes visible when the caller changes the request during a lookup. The runtime has to keep listening while the tool runs. If Wednesday replaces Tuesday, the next answer must use the new request even if the Tuesday lookup finishes first. Repeating “one moment” without tracking that change adds speech without making progress.

Pauses: thinking or finished?

Symptom: the agent answers while the caller is still looking for a date, or cuts into the middle of a phone number.

Mechanism: silence is ambiguous. A pause can mean “I have finished”, “I am thinking”, or “I am reading digits from a card”. A silence-based detector waits for a fixed duration. Short durations make the agent fast and rude; long durations make it polite and slow. Every product picks a point on that curve.

Detectors use different evidence. OpenAI's Realtime API offers semantic VAD, which estimates whether the utterance is complete, with an eagerness setting. LiveKit now documents both audio and text turn detectors, with the audio detector as its recommended default. Deepgram's Flux integrates recognition with end-of-turn events. A transcript-based detector, an audio-based detector and a silence threshold need different tests; product names alone do not tell you which mode an application is using.

Some stacks start work before the turn is confirmed. LiveKit documents preemptive generation from a final transcript, with cancellation when that transcript changes. Deepgram reports that Flux's eager event can arrive 150 to 250 ms before the confirmed end of turn, at the cost of 50% to 70% more LLM calls for the threshold range in its example. Those are vendor-reported settings and results, not a guarantee for another workload.

Speculation needs a boundary: prepare an answer or a discardable read, then validate it before use. Do not let an early turn hypothesis commit a booking. Track canceled work as well as latency saved.

First check: collect calls where the agent spoke during a pause, and label what the caller was doing: hesitating, spelling, reading a number, or actually finished. The mix tells you whether to tune thresholds, change the detector, or change the question so callers pause less.

Backchannels and interruptions

Symptom: the agent stops mid-sentence every time the caller says “mm-hmm”, or it keeps talking over a caller who is trying to correct it.

Mechanism: both are overlapping speech, with opposite meanings. An “mm-hmm” during an explanation may mean continue. “Wait, no” usually signals a correction. The same short acknowledgement after a confirmation question can be an answer, so timing and task context matter. Treating every overlap as an interruption makes the agent stop for nothing; ignoring overlap makes it miss corrections.

Interruption handling has two parts that are easy to merge by mistake. The first is acoustic: is this the caller, or the agent's own audio returning through a speakerphone or a line with poor echo cancellation? LiveKit distinguishes echo cancellation from background noise and voice suppression. Its adaptive interruption mode uses audio to distinguish genuine interruptions from backchannels; check support in the selected pipeline and SDK. The second is semantic: does this overlap carry a new intent?

After a real interruption, the agent's memory must match what the caller heard. As Part 1 noted, a sentence generated by the model is not necessarily a sentence the caller heard. LiveKit trims interrupted turns in the conversation history to the played portion. OpenAI's Realtime guide describes WebSocket clients stopping local playback and sending conversation.item.truncate with an audio cutoff. WebRTC and SIP use server-managed output buffering. The cutoff tracks playback position; it is not proof that a person understood every word. If that bookkeeping is missing, the model may believe it already offered “Tuesday at 10” when the caller only heard “Tuesday”.

First check: compare generated output, the playback cutoff and the history supplied on the next turn. The history must not imply that unplayed content reached the caller. With phone audio, distinguish media handed to the transport from audio observed at the receiving endpoint.

An illustrative correction. The backchannel keeps the turn with the agent; the correction stops playback and changes the task state. Timings are not measurements.

Test line echo separately

The discussion under Part 1 raised a useful omission in our test set. Anil Mathew described an agent stopping when its own voice returned through the phone line. We had listed caller-side noise without explicitly separating line echo. The following paired tests address that distinction; the report is a reader's account, not a measured result from VorkLab.

In the first test, return a delayed copy of the agent's output while the caller stays silent. The agent should continue, without inventing a user turn. In the second, keep that echo and add a genuine “no, Wednesday” over the playback. The agent should yield and process the correction. Repeat both at different echo levels and delays using the actual telephony path where possible.

Suppressing false stops must not suppress the caller. Raising a speech threshold can make the first test look better while making the second worse. Echo cancellation uses a reference of the outgoing audio to help remove its return. Noise suppression and speaker isolation solve different problems; inspect which processing stages the deployment actually enables.

Also test what happens after a false stop. Does the agent resume the unfinished phrase, restart it, ask a sensible question, or remain silent? LiveKit exposes false-interruption timeout and resume controls. Their presence does not establish that recovery sounds natural on your line. Listen to the resumed audio and check for skipped or duplicated words.

Use the same echo conditions in both tests. Check false stops and missed corrections separately.

Corrections have to reach the task state

Symptom: the caller says “actually, Wednesday”, the agent says “Sure, Wednesday”, and the confirmation message says Tuesday.

Mechanism: a correction has to update the task, and the transcript is only the first place it appears. The date may already be stored in a slot, used in a running availability lookup, or passed to a later step. Each of those copies needs to change or be rejected as stale.

Research benchmarks show this is still hard for models. Full-Duplex-Bench v2, which runs multi-turn spoken conversations with an automated examiner, reports that the tested systems struggle with corrections and lose track of what is being discussed. Outside voice, “LLMs Get Lost in Multi-Turn Conversation” found an average drop of about 39% when information arrives across several turns compared with a single complete prompt, and models tend not to recover after an early wrong assumption.

Application state needs an explicit correction path. Keep the current request as structured task state with a version. When the caller corrects a value, update the state, increment the version, and reject older read results before they influence the next response or write. Keep unaffected constraints, such as “after three”, unless the caller changes those too.

There are two different races to test. A Tuesday availability result arriving after the correction is stale information. A Tuesday booking already submitted before the correction is a business operation whose outcome must be checked. Discarding its response does not undo it. The agent must establish what happened before claiming that Wednesday is booked. Part 3 will cover confirmation, write validation and reconciliation in detail.

First check: for every correction in your test set, verify the final backend record, the spoken confirmation and any follow-up message. All three must carry the corrected value.

Names, numbers and dates

Symptom: the conversation sounds fine and the booking lands under the wrong surname or the wrong phone number.

Mechanism: word error rate averages over all words. “The” and a surname count the same. An unfamiliar surname has less linguistic context than an ordinary phrase. Phone audio, pronunciation and spelling can make it harder to distinguish candidate values. AssemblyAI's own benchmark reports that every model it tested had an entity error rate more than double its word error rate. That is vendor-run, but the direction matches the mechanism: one wrong character can invalidate an identifier.

A booking may require several values to be correct together. Measure the final record directly: multiplying per-field averages assumes independence and ignores readbacks and repairs.

Several mitigations work together:

• Vocabulary hints. Most STT providers support boosting expected terms. Deepgram's keyterm prompting is one example. Over-boosting can turn similar-sounding words into the boosted term, so test both directions.
• Match against a closed list. If the caller is choosing a clinic, a doctor or an existing appointment, compare the recognized text with the real candidates from the backend. Ask when several candidates fit; a nearest match is not a verified identity.
• Readback. Read critical values back in a normalized form and ask for confirmation. For spelled values, chunk them. Readback adds a turn, so reserve it for values that change the outcome.
• Another channel. Keypad input works well for digits. An SMS link can collect an email address that is painful to spell aloud.
• Backend verification. Check that the patient record, booking reference or account exists before the conversation continues.

First check: measure accuracy per field type: dates, times, names, phone numbers. A single WER figure will hide the field that breaks bookings.

Saying it back correctly

Symptom: the agent understood the time and still says “fifteen hundred” or mispronounces the doctor's name.

Mechanism: TTS models infer pronunciation from spelling and context. Numbers, dates, currencies, phone numbers, abbreviations and names in another language are common trouble spots. ElevenLabs documents model-dependent readings of numbers and supports normalization instructions for text supplied to TTS. Its current guide says normalization is enabled by default, which still leaves differences between model outputs. Check the pronunciation controls for the exact model and language you use.

For a cascade, a deterministic formatting step before TTS can make critical values testable: convert “15:00” to “three p.m.” in the caller's locale, read phone numbers in groups, and look up names in a per-deployment lexicon. Keep a small regression set of phrases and listen to it after every voice or model change, using transcription as an additional check. Another recognizer can repeat or conceal the original error, so it does not replace listening. A native audio model may not expose a separate TTS input; in that case, test the spoken output directly and use its supported pronunciation controls.

Noise and other voices

Symptom: random interruptions, phantom words in the transcript, or an agent that answers the TV.

Mechanism: a television, a second person in the room, or a car interior all contain speech-like sound. VAD detects speech activity; ASR transcribes speech. Neither function alone establishes which speaker should control the conversation, so background voices can produce plausible words and turn events.

Vendors offer speaker isolation and background voice cancellation, and turn detectors are now trained with noise. One result is worth knowing before turning on a denoiser: the paper “When De-noising Hurts”, which studied medical speech recognition, found that MetricGAN-plus-voicebank preprocessing made recognition worse in all 40 tested combinations of four recognizers and ten audio conditions. This result concerns that denoiser and dataset; it does not establish that every noise-reduction system is harmful. Enhancement tuned for human listeners can remove detail recognizers depend on.

First check: run the same set of noisy recordings with and without each processing step and compare entity accuracy and false interruptions. Decide from your own recordings.

Long calls and the voice gap

Symptom: the agent follows instructions well for three minutes, then forgets a constraint, repeats a question or skips a confirmation.

Mechanism: long calls accumulate context, interruptions and partial corrections. Speech also makes the task harder than the same task in text. Sierra's τ-Voice benchmark compares voice agents with text agents on the same customer service tasks. In the March 2026 paper, the best text configuration completed 85% of tasks, while voice agents completed 31% to 51% with clean audio and 26% to 38% with realistic noise and accents. Most failures were attributed to agent behavior, not to the task itself.

These are results for the paper's models, tasks and simulator, not a current leaderboard or a predicted success rate for a booking service. They justify testing the spoken workflow separately from its text equivalent.

For long calls, retain a compact record of the active request, unresolved questions and confirmed values. A summary can omit a correction, so compare it with that record before it becomes the next prompt's context. Test a late change to an early constraint, including after any context compaction. If the agent cannot resolve an ambiguity, the handoff should carry the current request and the status of any pending operation.

From symptom to mechanism

When a call goes wrong, start from the symptom and look for the mechanism before changing a model:

• Talks over the caller during a pause: turn detection threshold or detector choice. Look at what the caller was doing in the pause.
• Stops without a correction: interruption policy, echo or noise. Compare the input with outgoing playback and inspect recovery.
• Repeats an old value after a correction: stale task state or a late tool result. Compare the request version on each tool result.
• Wrong name, number or date in the record: entity recognition. Measure per-field accuracy and add matching, readback or another channel.
• Understood correctly, said incorrectly: TTS normalization or pronunciation. Check the text that was sent to TTS.
• Answers background speech: VAD and speaker isolation. A/B test the processing on recorded audio.
• Drifts late in long calls: context growth and unscoped prompts. Check whether the failure depends on call length.

Symptom to mechanism. The first check is usually a comparison between two traces of the same moment.

For any failed call, line up three things at the same timestamp: what the caller heard, what the model saw next, and what the backend recorded.

Add these calls to the test set

Part 1 ended with a small set of calls to compare architectures. For conversation quality, extend it with these scenarios:

• A caller who pauses for two seconds in the middle of a date.
• A caller who says “mm-hmm” twice while the agent lists options.
• A correction from Tuesday to Wednesday during the agent's answer.
• A surname that the default STT model gets wrong, spelled once.
• A phone number read in groups with pauses.
• The same booking with a television in the background.
• Delayed line echo while the caller stays silent.
• The same echo with a genuine correction over it.
• A false stop followed by silence, checking whether speech resumes correctly.
• A slow Tuesday lookup that returns after the Wednesday correction, plus a separate run where a Tuesday write was already submitted.
• A ten-minute call that ends with a booking change.

For each one, record the expected conversational behavior and backend outcome before running it. Inspect what reached the caller and when, the history supplied to the model, and the final booking record. A completed booking does not excuse an unexplained silence; a fluent answer does not establish a correct booking.

Part 3 moves from the conversation to the code behind it: how the decision layer holds state, checks permissions, asks for confirmation and makes writes safe to retry.

Build one replayable conversation investigation

Suppose a fictional caller says “Tuesday... actually, Wednesday” while an availability lookup is running. The final booking uses Tuesday. Before changing a prompt, preserve the evidence needed to locate the failure.

Create a case with a stable identifier and align three records: the audio the caller actually heard, the conversation history supplied to the model, and the events recorded by the booking service. Use a common time reference or document how the clocks relate. Otherwise, a misleading timestamp order can suggest that a correction arrived after a write when the services were simply using different clocks.

The evidence should let the team answer a sequence of concrete questions:

1. Did the caller's correction reach the input stream?
2. Was Wednesday present in the input used for the next decision?
3. Was the Tuesday lookup marked as obsolete, or accepted as current?
4. Which request version accompanied the proposed booking?
5. Did the booking service reject stale work at the write boundary?
6. What confirmation reached the caller?

The first failing boundary narrows the investigation. If the correction never reached recognition, inspect audio and input processing. If the correct date appeared in model history but an old write committed, inspect request state and backend validation. A more forceful prompt would not repair a missing backend check.

This investigation is illustrative. It is designed to distinguish possible causes, not establish which component failed in a real system without a trace.

Write acceptance criteria before recording the result

A test description such as “handles interruptions” leaves too much room for interpretation. Specify what must stop, what must continue and what the final business record must contain.

For the date correction, an acceptance record could say: obsolete speech stops; history reflects the played portion; the current request becomes Wednesday; the old lookup cannot become a new booking; and the final confirmation matches the stored reservation. If a Tuesday write was already submitted, the system must establish its outcome and follow the agreed correction path rather than pretending it never happened.

For a backchannel, the expected behavior may be different: a brief “mm-hmm” should not necessarily interrupt a useful explanation. For a caller spelling a name, silence may be part of the input rather than the end of the turn. Define these expectations for the product and audience instead of using a single silence threshold as the definition of a good conversation.

A sample worksheet row can therefore use:

• Scenario: date correction during an outstanding availability read.
• Expected audio behavior: stop the obsolete response and acknowledge the corrected date.
• Expected backend outcome: no new Tuesday booking; any confirmed booking uses Wednesday.
• Evidence: caller-audio timestamp, model-history event and booking-operation identifier.
• Result: pass, fail or inconclusive, with the missing evidence named.

“Inconclusive” is useful. If playback evidence is missing, do not silently treat generated text as proof of what the caller heard.

Change one suspected cause and keep paired tests

If echo appears to trigger false interruptions, compare two controlled calls with the same playback signal. In one, the caller stays silent. In the other, the caller makes a genuine correction over the echo. Reducing false interruptions is only an improvement if the real correction still gets through.

The same principle applies to noise filtering. A setting may suppress background television but also damage a quiet caller's surname. Keep both examples in the regression set. Record the input processing configuration, recognition settings, model, voice and transport so that the result can be reproduced.

Change one suspected cause at a time when practical. If the prompt, detector, recognition model and noise processing all change together, a better overall result tells you little about which change helped or which introduced a new failure.

For a small diagnostic set, inspect individual calls as well as summaries. Separate false interruptions, missed corrections, incorrect actions and unexplained silence. A single “success rate” can hide a tradeoff between them.

Make the regression set safe to reuse

Use synthetic or appropriately permitted recordings for repeatable tests. Keep credentials, personal caller details and production identifiers out of shared examples. If a real incident informs a test, retain only the information needed to reproduce the behavior under the team's data-handling rules.

Run state-changing scenarios against an isolated test backend. The test should verify the resulting record and detect duplicates, rather than rely on the agent's spoken claim that an action succeeded. Keep audio quality and task correctness as separate checks: a natural conversation can still produce the wrong booking, and a correct booking can still come with an unacceptable caller experience.