Choosing a Voice Agent Architecture

A caller asks to move an appointment to Tuesday. The agent starts checking availability. Then the caller interrupts: “Actually, make that Wednesday.”

The voice may stop immediately. The availability request may still be running. The next response may use the old date. If the workflow has moved on to booking, a write may already have reached the backend.

This illustrative scenario is the kind of test I would bring to an architecture discussion, before committing to a stack.

An architecture review needs two separate decisions: how speech moves through the system, and which parts the team will operate. A speech-to-speech model can be integrated through a managed platform or a custom agent runtime. A cascaded pipeline can do either as well.

Part 1 of Voice Agents in Production. A practical guide to speech pipelines, managed platforms, programmable frameworks, and operating responsibilities.

Define the job before choosing the stack

A tutor needs fluid dialogue and useful feedback. A booking assistant needs accurate details and a valid reservation. A support agent may need authentication, several business systems, and a reliable transfer path. These lead to different requirements even when all three products speak.

Write down the successful outcome, the supported channel, the actions the agent may perform, and the cases it must hand off. Add expected call peaks, languages, and data-handling constraints. For our appointment assistant, success means the intended booking exists in the authoritative system and the caller hears an accurate confirmation.

This also limits the first implementation. One bounded workflow may need a configured voice runtime and an existing booking API. Retrieval, another database, or multiple specialist agents need a requirement that justifies them.

Start with the whole conversation

A system diagram should account for five responsibilities:

1. Connection: receiving and sending audio through a phone call, browser, or app.
2. Conversation control: deciding when a turn ends, when to respond, and how to handle interruptions.
3. Speech and language: interpreting the caller and producing an audible response.
4. Application state and actions: tracking the current request, checking permissions, and interacting with business systems.
5. Operations: tracing failures, recovering sessions where possible, handing off to a person, and checking outcomes.

These are responsibilities, not a requirement to deploy five services. A platform may handle several together. The architecture still needs to say who owns each one.

In the appointment example, the connection carries the correction, conversation control handles the interruption, and application state needs to replace Tuesday with Wednesday. The booking service must then act on the right request.

The runtime is an event coordinator: a partial transcript, a tool result, a new user turn, and a playback-complete event can arrive while other work is running. Generated speech and heard speech can differ. A useful trace needs to connect the model output to what actually reached the caller.

Three ways to process speech

LiveKit's pipeline documentation provides a useful vocabulary for the audio path.

A streaming cascade: STT → text LLM → TTS. Its explicit boundaries make failures inspectable: recognized text, a proposed tool call, the backend result, and text submitted for synthesis. Existing text-based retrieval and business workflows can often be reused.

The stages can overlap. Recognition can emit partial text, and TTS can start on a suitable chunk before the full answer is generated. The integration work is deciding when an input is stable enough to act on and when output is ready to speak. Partial transcripts may change; buffered speech may become obsolete after a correction. Provider replacement also requires checking event semantics, audio formats, and cancellation behavior.

Native speech-to-speech: audio in → audio out. The model can use information in the incoming audio that plain transcription may discard. That makes it a useful candidate for tutoring, role-play, or other products where timing and vocal expression are central. Which cues it actually understands is a question for evaluation.

Inspect the API's transcripts, tool interface, output controls, and interruption events. Native audio does not establish a universal latency advantage: turn detection, network placement, model processing, and business tools all contribute to the delay. A slow booking lookup can dominate either architecture.

A half-cascade: audio understanding → text → TTS. An audio-input model produces text for a separate synthesis service. Ultravox is a concrete example of the audio-input, streaming-text-output pattern. This preserves a textual boundary before speech and allows an independent voice choice.

It is useful to investigate when audio input and a particular synthesis system are both requirements. Support for audio input alone does not prove understanding of emotion or prosody. The output mode and the capabilities of the chosen model still need checking.

Simplified audio paths, following LiveKit's pipeline taxonomy. Tools, application state, and transport are omitted.

For an appointment assistant, I would start the comparison with a cascade when inspecting extracted details and controlling response wording are priorities. I would also test a speech-to-speech candidate if conversational timing or vocal cues materially affect the experience.

A transcript can contain the wrong date, and correct text can still be pronounced incorrectly. Each architecture needs a test that reaches the final application state.

Streaming does not settle turn-taking

Streaming allows work to begin before an entire input or output is complete. It does not, by itself, tell the agent whether a pause means “I have finished” or “I am thinking.”

Voice activity detection, speech-recognition endpointing, and semantic turn detection provide different signals: whether speech is present, where a transcription segment ends, and whether the speaker has finished the thought. Interruption handling adds another question: should a short “mm-hmm” stop the assistant, or is the caller simply acknowledging it?

LiveKit documents these controls separately and adjusts interrupted conversation history to reflect speech the user actually heard. That matters because an agent should not assume the caller received a sentence that was still waiting in a playback buffer.

Transport can change who performs that bookkeeping. OpenAI's Realtime documentation distinguishes server-managed output buffering for WebRTC and SIP from WebSocket clients that must manage playback and truncation themselves.

A concrete question for stack evaluation is: after an interruption, what tells the model how much of its answer was actually played?

Full duplex needs a precise definition. A connection carrying audio both ways, a runtime stopping playback on interruption, and a model processing incoming speech while generating outgoing speech are three different capabilities. Moshi's full-duplex research addresses the model-level problem. It does not make every bidirectional voice API equivalent.

Test a hesitation, an acknowledgement, and a correction while a tool is running. These reveal whether the system has the behavior your product needs.

Decide what the team will operate

Managed platforms expose specific boundaries of control. Retell's custom LLM interface is a useful example: your server generates responses over a per-call WebSocket, while Retell handles telephony, transcription, turn-taking, and synthesis. If the caller continues speaking, Retell can discard a requested response and issue a new response_id. Your response server therefore has to handle work that is no longer relevant.

Custom mode also changes the available tooling. Retell documents limitations on built-in testing, and custom tool calls need explicit reporting to appear in the transcript. Test the actual integration mode before relying on a platform feature list.

Vapi's data-flow documentation describes another boundary: custom model and speech integrations can coexist with proprietary orchestration running on Vapi infrastructure. Bringing your own model does not move the whole service into your environment. Using a provider's TTS API and using its complete agent platform are different decisions as well.

Programmable frameworks put application behavior in your code. LiveKit Agents fits an application built around LiveKit rooms, participants, and agent sessions. Pipecat is a Python framework whose processors exchange audio, text, and control frames. Its pipeline abstraction makes processing stages explicit.

Choose between them by implementing the difficult part of your workflow: interruption behavior, a custom processor, tool coordination, or a channel integration. Both approaches still require decisions about transport, model services, and hosting. Open-source application code does not imply self-hosted inference.

A custom stack can remain modest: a phone or browser connection, an agent runtime, hosted models, existing business APIs, and traces linking a call to its result. Several roles may be provided by one service. The team owns its application behavior, integration maintenance, and whichever deployment responsibilities it takes on.

A direct voice-model API is another starting point. If its transports, session events, and tool interface fit one focused workflow, a separate framework may add little initially. The application still needs secure access, task state, integration code, and recovery. Add a framework when those responsibilities benefit from its abstractions.

Self-hosting introduces another set of decisions: where media runs, where agent workers run, and where model inference happens. These choices can be mixed. For each layer, record where it runs, who operates it, and what happens when it fails. LiveKit, for example, documents self-hosted agent workers using its cloud media transport and observability.

An ownership checklist. Select an operating arrangement for each layer, then check the dependencies between them.

For self-hosted workers, LiveKit calls out capacity planning, autoscaling, and draining active sessions during shutdown. Those are concrete operating responsibilities to include in a build-versus-buy discussion.

For our appointment assistant, a reasonable design to evaluate would be a managed conversation layer connected to a booking service controlled by the application team. Another would use custom agent workers with cloud media and hosted models. The same business rules could sit behind either arrangement.

Self-hosting the worker does not keep audio local if it sends that audio to hosted inference. Map the complete data path, including recordings and logs. Hosting inference separately adds model compatibility, hardware capacity, serving, upgrades, and performance under load to the team's responsibilities.

For a bounded booking workflow, start by testing whether a managed platform exposes the required operations and failure evidence. For a product with custom conversation processing, include a programmable runtime. For a voice-first learning product, include native audio.

Let the tools shape the architecture

Before choosing a model, map the calls needed to complete the task. For each tool, record its inputs, latency distribution, dependencies, and whether it reads data or changes it. Check what happens on timeout and whether its work can be canceled safely.

For a tool-dependent answer, trace the time spent deciding what to call, executing the backend work, and turning the result into speech. Several dependent calls can add several model round trips. Independent reads may run in parallel when permissions and backend limits allow it. A known sequence can live in application code behind one tool, avoiding a model decision at every step.

A faster conversational model only reduces the model's share of the wait. If availability lookup takes several seconds, the runtime needs a way to acknowledge the request, keep listening, and handle a correction while that work continues. An acknowledgement improves feedback; it does not make the result arrive sooner. Measure typical and slow-tail latency, including p95, under realistic load.

This affects model placement. A conversational model might handle a bounded lookup directly. A workflow requiring substantial analysis could delegate that work to a separate model through a tool while the voice runtime manages the interaction. That design needs explicit task status and a rule for accepting or discarding a late result. The extra model also adds latency and context transfer, so test whether its improvement in task accuracy justifies the split.

Evaluate the candidate on the actual tool schemas: choosing the right operation, extracting arguments from speech, asking for missing details, and using returned data correctly. A valid JSON object can still contain the wrong appointment date. Include total model-plus-tool delay and recovery behavior in the comparison.

For the booking assistant, I would test availability lookup separately from the commit operation and keep authorization and write validation in the booking service. Select the speech path and runtime together with this tool workflow. A platform that sounds good in a conversation demo may still lack the control needed for a long-running or state-changing operation.

Give business actions their own boundary

Return to the caller changing Tuesday to Wednesday. Three kinds of state now matter:

• Session state: listening, generating, or playing buffered audio.
• Task state: the current date is Wednesday; a confirmation may still be pending.
• Business state: the booking service has committed a change, rejected it, or has an outcome the caller does not yet know.

A late Tuesday lookup belongs to an earlier request. Correlating results with the request version lets the application reject stale information before it affects the next action. A transcript provides context; the booking system provides the authoritative outcome.

Stopping playback prevents more of the old answer from being heard. Canceling generation prevents more of that answer from being produced. Neither operation establishes whether a booking request has already changed the database.

LiveKit's tool documentation makes this distinction explicit: interruption does not automatically cancel tool work. Applications need to define cancellation behavior, including what to do when an external operation has partly completed.

For this example, I would keep availability checks separate from booking commits, associate a proposed action with the current request, and enforce a request-version check in the booking service at the write boundary. If a write times out, the next step should establish its outcome before a retry can create another booking.

Stopping the voice does not establish the outcome of a business action.

The Decision Layer article will cover confirmation rules, idempotency, and recovery. This design needs an explicit place to implement them.

Design for simultaneous calls and recovery

Monthly minutes cannot tell you how much capacity a busy hour needs. For a stable illustrative workload, two calls entering active conversation per second and an average active duration of 180 seconds imply 360 active calls on average: 2 × 180. Use that average to check steady load; peak capacity also needs burst and failure assumptions.

Capacity must exist across the path: media connections, session workers, speech services, model quotas, and business APIs. More worker replicas cannot remove a provider's concurrency limit. Test bursts and degraded dependencies separately from steady load.

Calls are long-lived work. During a planned rollout, stop assigning new sessions to retiring workers and let active sessions finish. LiveKit's deployment documentation describes draining for this purpose. A process crash is different: a replacement worker does not establish the outcome of an in-flight business operation. Define the caller's fallback and reconcile any uncertain operation.

Keep required validation on the live path. Move nonessential summaries or analytics out of that path when possible. Place the runtime with the full network journey in mind: a nearby worker can still wait on distant inference or business services.

Compare the same cost boundary

At the time of checking in October 2026, Vapi lists a $0.05/minute hosting charge with model usage passed through separately. LiveKit lists $0.01 per hosted agent session minute beyond the allowances on its paid self-service plans, with other charges such as inference and telephony separate. These are different billing boundaries, so the two numbers do not compare the price of a complete working agent.

Build one workload estimate covering media and telephone usage, inference, runtime hosting, reserved capacity, recordings, traces, and human recovery. Include engineering and operational overhead if the comparison is meant to guide ownership. Avoid counting bundled items twice.

A useful outcome metric is cost per resolved request: total cost for a defined request cohort divided by verified successful outcomes. State whether human-assisted resolutions and their costs are included. A cheaper minute may lose its advantage through longer calls, repeat contacts, or rework. Part 5 will examine that calculation in detail.

Compare candidates on the same call

Before expanding a prototype, I would run each candidate through the same small set of scenarios:

• A straightforward request with no corrections.
• A correction while the assistant is speaking.
• A long pause in the middle of a date or name.
• A slow availability lookup followed by a changed request.
• A booking timeout followed by a retry.
• A handoff when the assistant cannot complete the task.

In the correction test, success means the final booking uses Wednesday, no stale Tuesday action commits, and the spoken confirmation matches the backend.

For each run, inspect what the caller heard, the sequence of state changes, and the result in the booking system. Measure response delay from a defined event, such as the end of the caller's turn, to the first audible response. Track the delay to a useful answer separately from an immediate “Let me check.”

Capture a small decision record: the required behavior, the selected speech path, who operates each layer, the evidence from the test calls, and the condition that would trigger a review.

For example: “We use a managed conversation runtime with our existing booking service. We will reconsider the runtime if a required interaction cannot be implemented, essential failure evidence is unavailable, or measured economics justify taking on more operational work.”

Continue with Part 2 for pauses, backchannels, names, noise, and recovery after an interruption.

Turn the comparison into an architecture decision record

A useful decision record is short enough to revisit when the system changes. It should connect a choice to evidence, rather than list technologies that happen to be available.

For a fictional appointment assistant, the record could begin with this requirement: a caller may correct the date while a lookup is running; the eventual booking must use the latest confirmed request. That requirement crosses the speech layer, runtime and booking service. Comparing voice quality alone cannot establish whether a candidate satisfies it.

Use one row per candidate in the downloadable worksheet. A sample entry might read:

• Speech path: streaming speech recognition, a text model and separate synthesis.
• Runtime: managed conversation handling, with application tools owned by the engineering team.
• Business backend: the existing booking service, which validates permissions and the current request version before accepting a write.
• Required evidence: a trace connecting the correction, the cancelled or superseded lookup, the proposed write and the final reservation.
• Review trigger: the runtime cannot expose enough information to explain a stale response, or a required interaction cannot be implemented reliably.

This is an illustrative candidate, not a recommendation for every booking product. A native audio model can be evaluated against the same behavioral requirements. The important comparison is whether each complete configuration meets the workflow's needs, including the integrations and recovery path.

Separate a slow read from an uncertain write

Two failures can sound identical to a caller: the agent says “one moment” and then goes quiet. Their recovery paths should differ.

If an availability lookup is slow, the system can often abandon that read and make a fresh lookup for the current request. An old result must not overwrite a newer correction. The runtime needs a way to associate each result with the request that produced it.

If a booking write times out, the system may not know whether the reservation was created. Blindly repeating the write can create a duplicate. The application should use a stable operation identifier and a backend mechanism for looking up the outcome or deduplicating the operation. Whether the backend actually supports that contract is an architecture question to settle before launch.

For the comparison test, deliberately delay each operation in a controlled environment. Record what the caller hears, what the runtime believes happened and what the booking service stored. A candidate passes only when those three views can be reconciled. “The tool timed out” is not an adequate final account of a potentially completed booking.

Keep authorization and business validation in the service that owns the booking. Prompt instructions can guide the model, but the write boundary still needs to reject an action that is no longer valid.

Compare the operating burden with the same test workload

A managed runtime can reduce work in one layer while leaving integration, evaluation and incident handling with your team. A custom runtime can provide more control while adding responsibilities for deployment, scaling and recovery. The worksheet names owners explicitly so that these responsibilities do not disappear inside a vendor comparison.

Run candidates against the same representative call set, with the same backend and comparable model configuration where practical. Record the actual setup, since a result for one combination of recognition, model, voice and transport does not automatically apply to another.

Include normal calls, corrections, slow tools, failed writes and transfers to a person. For each candidate, note:

1. Whether the intended task completed correctly.
2. How long the caller waited for useful audio.
3. Whether the trace was sufficient to explain a failure.
4. Which failures required engineering intervention.
5. What usage was billed, and which operating costs remain outside that bill.

Do not turn a small trial into a universal vendor ranking. Use it to eliminate configurations that cannot meet your requirements and identify what still needs testing. Preserve failed runs as part of the decision record, not just the successful demonstration.

Revisit the choice when the workflow changes

Adding outbound calls, another language or a new state-changing tool can change the architecture requirements. A configuration that works for answering questions may need a different recovery contract once it can make reservations or take payments.

Define review triggers around behavior and operating evidence. Examples include repeated ambiguous write outcomes, missing traces, a growing manual-recovery workload or a channel the current runtime does not support. Re-evaluate the affected boundary first; a new requirement does not automatically justify replacing the whole system.