Build your first AI voice agent: 3 step-by-step examples

If you've tried to build a voice agent in the last year, you already know the hard part isn't the idea — it's the plumbing. You have to wire up speech-to-text, feed it into an LLM, convert the response back to speech, and somehow make all of that feel like a real conversation with sub-second latency, clean turn-taking, and graceful handling of interruptions. Get one piece wrong and the whole thing feels robotic.

Here's the good news: there are now three clear ways to build an AI voice agent, and the fastest one didn't exist when most tutorials were written. You can use an all-in-one API like AssemblyAI's Voice Agent API, which bundles speech-to-text, an LLM, and text-to-speech behind a single WebSocket. You can use an orchestrator like Vapi, LiveKit, or Pipecat that stitches best-of-breed components together for you. Or you can build a fully custom pipeline from scratch, choosing every model yourself.

This guide walks through all three, with working examples for each. I'll be honest about the tradeoffs — because the right choice depends on how much control you actually need and how fast you want a talking agent in front of users. Let's start with what every voice agent is made of.

The core components of any voice agent

No matter which path you pick, an AI voice agent is the same four-part loop under the hood:

• Speech-to-text (STT). Streaming transcription that turns the caller's audio into text in real time. This is the foundation — if the transcript is wrong, the LLM reasons over garbage. Accuracy on messy, real-world audio (names, emails, phone numbers, order IDs) matters more here than in any other application.
• A large language model (LLM). The brain that interprets intent, follows your system prompt, calls tools or functions, and decides what to say next.
• Text-to-speech (TTS). Converts the model's response back into natural-sounding audio.
• Turn detection and interruption handling. The part everyone underestimates. Your agent needs to know when the user has finished speaking, when to jump in, and how to stop talking the instant someone interrupts it. Bad turn-taking is the number one reason a demo feels off, and it's consistently flagged as the hardest engineering problem in voice.

Every approach below solves these four problems — the difference is how much of the wiring you do yourself. If you want a deeper primer on the category, our overview of AI voice agents covers the architecture in more detail.

The three ways to build an AI voice agent

Before we get into code, here's the decision at a glance. The taxonomy is simple: how much of the stack do you want to own?

Teams with specific latency, privacy, or on-prem requirements

Most teams building their first agent overestimate how much control they need. So we'll start with the path that gets you talking to a working agent the fastest, then work outward toward more control.

Option 1: The all-in-one path — AssemblyAI's Voice Agent API

This is the path I'd recommend to almost anyone building their first voice agent, and it's the one the older versions of this guide couldn't include because it hadn't shipped yet. The Voice Agent API collapses the entire STT-LLM-TTS loop into a single WebSocket connection. You open one connection, stream audio in, and get audio back. That's it — no separate providers to reconcile, no glue code to buffer and re-time three different streams.

Here's what that buys you in practice:

• One connection instead of three. Transcription, reasoning, and speech generation all live behind a single WebSocket. There's no SDK to install and no orchestration layer to babysit — it's invisible infrastructure, so you spend your time on your agent's behavior, not on plumbing.
• Flat, predictable pricing. It's $4.50/hour, flat. No token metering, no per-component line items, no surprise bill after a traffic spike. You can model your unit economics on day one.
• Roughly 1 second end-to-end latency. That's the full round trip — user stops talking, agent starts responding — which is what actually determines whether a conversation feels natural.
• Built on Universal-3.5 Pro Realtime. The STT foundation is our flagship realtime model, and it's the reason this path doesn't sacrifice accuracy for convenience (more on the benchmark below).
• Session resumption within 30 seconds. If a connection drops, the session can pick back up without losing conversational context — the difference between a dropped call and a recovered one.
• 6 languages at launch (English, Spanish, French, German, Italian, and Portuguese), and it's been generally available since April 14, 2026.

There's one more thing that changes how it feels to build with. The Voice Agent API is designed to be assembled by coding agents — it works cleanly with Claude Code. Because the whole thing is one documented WebSocket with no SDK, you can describe the agent you want in plain language, let a coding agent scaffold it, and have a working prototype in well under 30 minutes. Teams that onboard this way have seen activation lift meaningfully — the fewer moving parts, the fewer places to get stuck.

The accuracy story is what makes this more than a convenience play. On the Pipecat open STT benchmark — real voice-agent conversations, where lower word error rate is better — Universal-3.5 Pro Realtime posts a 6.99% pooled WER, ahead of ElevenLabs Scribe v2 at 9.76%, Google Chirp3 at 9.04%, and Deepgram Flux at 15.58%. The gap is even wider on the things voice agents actually trip over: entity error rate lands at 15.31% versus 39.70% for Scribe v2. That's the difference between an agent that hears "order number A-1-5-J" correctly and one that guesses. You can see the full methodology on our benchmarks page.

To get started, spin up a key and follow the build guide, or read the launch announcement for the full feature rundown. The docs have a copy-paste quickstart. This is the "Example 0" I'd reach for first — and for most receptionist, appointment-setting, lead-qualification, and food-ordering agents, it's also the last stop.

Option 2: The orchestrator path — Vapi, LiveKit, and Pipecat

Sometimes you want managed transport and turn-taking but insist on choosing each model yourself — a specific LLM for reasoning, a particular voice for TTS, and best-in-class STT feeding both. That's what orchestrators are for. They handle the real-time media plumbing while letting you swap components at each stage. AssemblyAI plugs into these as the speech-to-text layer, and honestly, running our STT inside an orchestrator is a completely valid production setup — the all-in-one and orchestrator paths coexist.

Example: Vapi + AssemblyAI

Vapi is a popular orchestrator that manages the call lifecycle, turn detection, and provider routing for you. You configure it declaratively: point the STT step at AssemblyAI's streaming speech-to-text, choose your LLM, and pick a TTS voice.

What you'll need: a Vapi account, an AssemblyAI API key, an LLM provider key, and a TTS voice. Core components: Vapi handles transport and turn-taking; AssemblyAI handles transcription; your chosen LLM and TTS handle reasoning and speech. The win here is that you keep AssemblyAI's real-time accuracy on names and numbers while letting Vapi own the orchestration.

Example: LiveKit + AssemblyAI

LiveKit gives you more control over the media layer, which is useful if you're building something with custom audio routing, multi-party calls, or your own front end. You wire AssemblyAI's streaming transcription into LiveKit's agent framework as the STT plugin and bring your own LLM and TTS. Retell, LiveKit, and Fireflies were all early adopters of Universal-3.5 Pro Realtime, so you're in good company running our models inside a framework.

What you'll need: a LiveKit project, an AssemblyAI API key, and your LLM plus TTS providers. Core components: LiveKit owns the real-time transport and agent loop; AssemblyAI supplies the transcript; you own reasoning and voice. If you want to understand where these stacks start to strain as you scale, we wrote about exactly that in where voice agent stacks start showing their limits.

The tradeoff with orchestrators is real: you get flexibility, but you're now responsible for reconciling latency across providers, managing multiple bills, and debugging across component boundaries when something feels laggy. That flexibility is worth it when you genuinely need a specific component — and overkill when you don't.

Option 3: The fully custom path — self-hosted models end to end

If you have hard requirements around on-prem deployment, data residency, or a very specific latency budget, you can build the entire pipeline yourself and own every hop.

Example: DeepSeek R1 (local via Ollama) + AssemblyAI STT + ElevenLabs TTS

This setup runs the LLM locally with Ollama — DeepSeek R1 in this case — for full control over the reasoning layer, uses AssemblyAI for streaming speech-to-text, and ElevenLabs for text-to-speech. You write the orchestration code yourself: capturing microphone audio, streaming it to STT, detecting end-of-turn, prompting the local model, and piping its response into TTS.

What you'll need: Ollama running DeepSeek R1 locally, an AssemblyAI API key, an ElevenLabs key, and comfort managing async audio buffers. Core components: you own all four — STT, LLM, TTS, and turn detection — and every buffer and network call between them.

Worth noting on accuracy: because the custom path names each component explicitly, your STT choice is doing a lot of work. On real voice-agent audio, Universal-3.5 Pro Realtime's 6.99% WER and low entity error rate are what keep the transcript clean enough for a local LLM to reason over. The rest of this path is engineering effort — which is the point. You take on days-to-weeks of work in exchange for total control. For most teams, that trade only pays off with a genuine on-prem or compliance constraint.

Which path should you choose?

Here's how I'd decide, quickly:

• Building your first agent, or want it in production fast? Start with the Voice Agent API. One WebSocket, flat $4.50/hour, ~1 second latency, and you can prototype with Claude Code in under 30 minutes. This is the right call for the vast majority of data-collection agents — receptionists, appointment setters, lead qualifiers, order takers.
• Need to pick a specific LLM or TTS voice but want managed transport? Use an orchestrator like Vapi, LiveKit, or Pipecat with AssemblyAI as your STT layer.
• Have on-prem, residency, or bespoke latency requirements? Build the custom pipeline and own every component.

A good rule of thumb: pick the shallowest layer that covers your requirements. Every layer of control you add is a layer of infrastructure you now maintain. If you want a data-backed look at what actually separates good agents from frustrating ones, our 2026 insights report is worth a read before you commit to an architecture.

A note on where this is heading

Here's the shift most teams haven't fully internalized yet: the hard part of voice agents is moving away from wiring and toward behavior. A year ago, "building a voice agent" mostly meant integration work — getting three services to talk to each other without adding half a second of latency at every seam. That work is increasingly a solved problem you can buy. What's left is the part that was always the real product: the prompt design, the tool calls, the way your agent recovers when a caller mumbles an address or interrupts mid-sentence.

That's why I'd push almost everyone toward the all-in-one path first, even teams that assume they'll need more control. Not because the other paths are wrong — they're excellent when you have a real reason to reach for them — but because starting with managed infrastructure lets you spend your first week on your agent's actual behavior instead of on plumbing you'll eventually want to hand off anyway. Build the conversation first. You can always drop down a layer when a concrete requirement forces the question. Most teams find it never does.

Frequently asked questions

What's the easiest way to build an AI voice agent?

The all-in-one path. AssemblyAI's Voice Agent API bundles speech-to-text, an LLM, and text-to-speech behind a single WebSocket, so you don't wire three providers together. There's no SDK, pricing is a flat $4.50/hour, and you can stand up a working agent — often with a coding agent like Claude Code — in under 30 minutes.

How much does it cost to build a voice agent?

With the Voice Agent API it's a flat $4.50/hour of conversation, with no separate token or per-component metering, which makes cost easy to model. Orchestrator and fully custom paths bill each component (STT, LLM, TTS) separately, so total cost varies with your provider choices and traffic. See pricing for current details.

Do I need to know how to code?

You'll need some development ability, but far less than you'd expect. The Voice Agent API is a single documented WebSocket with no SDK, which means you can describe the agent you want to a coding agent and have it scaffold the integration. Orchestrator and custom paths require more hands-on engineering, especially around turn detection and latency.

Voice Agent API vs. Vapi vs. LiveKit — which should I use?

Use the Voice Agent API when you want a production-grade agent fast with managed infrastructure and predictable cost. Use an orchestrator like Vapi or LiveKit when you need to choose specific LLM or TTS providers but want managed transport and turn-taking. AssemblyAI runs as the STT layer inside those orchestrators too, so the paths aren't mutually exclusive.

How accurate is the speech recognition for voice agents?

On the Pipecat open STT benchmark of real voice-agent conversations, Universal-3.5 Pro Realtime posts a 6.99% pooled word error rate — ahead of ElevenLabs Scribe v2 (9.76%), Google Chirp3 (9.04%), and Deepgram Flux (15.58%) — with a much lower entity error rate on names, phone numbers, and order IDs. Full methodology is on the benchmarks page.

Which languages does the Voice Agent API support?

At launch it supports six languages: English, Spanish, French, German, Italian, and Portuguese. It's been generally available since April 14, 2026, and includes session resumption within 30 seconds so a dropped connection doesn't lose conversational context.