How I built a voice agent without writing (or understanding) any code

I'm not a developer. I can read code well enough to know roughly what it's doing, and I can tell when something looks wrong, but I can't write it from scratch. Half the error messages might as well be in Latin. I have never once set up a server.

And yet I built and deployed a fully functional AI voice agent that talks 2026 F1 regulations with fans in real time, complete with a custom persona, a private knowledge base, and a web interface people can actually use. This is how I did it, and how you can do the same thing for your own topic without writing any code yourself.

Can you really build a voice agent with no code?

Short answer: yes, if you let a coding agent write the code for you. Here's the whole path in three steps:

1. Build the brain in Claude Cowork. Create a knowledge base (a clean markdown file of everything your agent should know) and a system prompt (the persona and rules) through conversation. No terminal required.
2. Hand it to Claude Code with the Voice Agent API docs. Describe the app you want. Claude Code writes, wires, and debugs the integration for you.
3. Deploy to Vercel. Push to GitHub, connect Vercel, and your agent is live at a URL you can share.

What you need: an AssemblyAI account, Claude Cowork, and Claude Code. You should be comfortable opening a terminal and pasting a command, but you don't need to understand or write the code. What it costs: the Voice Agent API is a flat $4.50 per hour of conversation that covers speech-to-text, the LLM, and text-to-speech together, and there's a free tier so you can build and test without a credit card.

That's the summary. The rest of this post is the honest, detailed version, including the parts that broke.

What we're building

A voice agent is a program that listens to you through your microphone, understands what you said, thinks about it (optionally searching a knowledge base), and speaks a response back through your speaker. The hard parts, speech recognition, language understanding, and voice synthesis, are all handled by the Voice Agent API. Your job is to tell the agent who it is, what it knows, and how it should behave.

My agent is Pit Lane Pete: a retired F1 pit crew mechanic with 22 years of tire changes behind him, now unwillingly detained in a podcast booth to explain the 2026 regulations to people who've never had to bolt on a front wing at 300kph.

Yours can be anything. A support rep for your product. A docent for a museum exhibit. A study partner for medical board exams. A receptionist that books appointments. The architecture is identical every time, and it's the same architecture teams use to ship real AI voice agents in production.

Phase 1: Build the brain in Claude Cowork

Before touching any code, I built two things in Claude Cowork, Anthropic's desktop AI tool for non-developers: the knowledge base and the system prompt. Cowork lets you work with files, do research, and create documents through conversation, so none of this needs a terminal.

The knowledge base

A knowledge base is the source of truth your agent searches when users ask questions. For Pete, this was 847 pages of 2026 F1 regulations distilled into a single structured markdown file. For your project, it might be your product documentation, a company FAQ, a manual, or any domain reference you want the agent to draw from.

To build yours, open Cowork and describe what you need: "I need a knowledge base for a voice agent about [your topic]. Here are the source documents." Then attach your PDFs, paste your text, or ask Claude to research and structure it. The output should be a clean markdown file with headings, short sections, and plain language. The agent doesn't read it like a human, it searches it for relevant keywords, so clarity beats prose.

The system prompt

The system prompt is the instruction sheet your agent reads before every conversation. It defines the persona, the rules, the tone, and the constraints. AssemblyAI ships pre-built starter prompts you can use directly, and they're production-tested for common cases like support, scheduling, and general Q&A. If you're not sure where to start, grab one and modify the persona section.

I wrote my own with Cowork's help. The conversation was basically: "I'm building a voice agent with this knowledge base. Write me a system prompt that..." followed by a detailed description of Pete's character.

Phase 2: Build the app with Claude Code

Once the knowledge base and system prompt existed, I moved into Claude Code, Anthropic's AI coding tool that runs in your terminal and can write, edit, and debug code on your behalf through conversation. I gave it my prompt.md and knowledge-base.md files and described what I wanted: a simple, clean interface covering the three actions a user needs, start a conversation, talk, and end it. That was the brief.

This is the part worth sitting with, because it's the whole reason the no-code path works. The Voice Agent API was designed for coding agents. There's no proprietary SDK to learn. The entire integration is one WebSocket connection carrying standard JSON, so a coding agent can wire it up from the docs in a single pass. A developer I know built a working agent in under 30 minutes this way, and his summary was blunt: one WebSocket connection, standard JSON, no SDK.

What to think about before you hand it off

Before Claude Code writes a line, spend two minutes on the shape of your ideal agent. Not technically, just practically. Who uses it, and how? Do you share it with customers, keep it for your team, or run it privately? Does it live on a webpage or run as a tool? Is it always on, or do you start it when you need it?

Jot that picture down and put it in your first message. Claude's job is to translate that context into the right approach, and it'll ask follow-up questions when it needs to. Think less "instructions for a contractor" and more "describing a problem to a patient consultant who happens to know how to build everything."

The one thing worth naming upfront is that your AssemblyAI API key is like a password, it unlocks your account and bills your usage. However you describe your vision, mention that you want the key kept private and protected. Claude will figure out the right way to do that for your setup. You don't need to know what that looks like. Just name the concern.

Hosting

Claude and I deployed to Vercel. It's free for small projects, handles serverless functions well, and deploys straight from a GitHub push. The API route format is simple:

// api/token.js
export default async function handler(req, res) {
  const response = await fetch('https://api.assemblyai.com/v1/realtime/token', {
    method: 'POST',
    headers: {
      'Authorization': process.env.ASSEMBLYAI_API_KEY,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ expires_in: 300, max_session_duration: 3600 })
  });
  const data = await response.json();
  res.json({ token: data.token });
}

A vercel.json at the root routes the base URL to your HTML file:

{ "rewrites": [{ "source": "/", "destination": "/index.html" }] }

One honest caveat: field names and token routes can drift as the API evolves, so always confirm the current endpoint and payload in the documentation before you ship. More on that below.

No-code builder or coding agent: which path is yours?

If you searched for a no code voice agent, you probably ran into drag-and-drop platforms first, the Voiceflows and Synthflows of the world. They're genuinely good at what they do, and for a linear phone tree or a simple FAQ bot with zero engineering appetite, a visual builder can be the right call.

But there's a ceiling. The moment you want custom logic, a specific interface, your own hosting, or behavior the builder doesn't expose, you're stuck inside someone else's UI. The coding-agent path I'm describing gives you a real application you own, deployed on your own infrastructure, with none of the templates to learn, because Claude Code writes the code and the Voice Agent API hides the hard voice infrastructure behind one connection. You get the "no code" experience of a builder with the flexibility of a custom app. That's the trade I'd make almost every time.

Phase 3: Gotchas and things to expect

How should the conversation end?

A session doesn't automatically know when it's done. If nobody speaks for a while, should it hang up, wait, or say something? These are design decisions, not technical ones. A support agent shouldn't disconnect after 10 seconds of silence, the user might be looking something up. A quick-demo agent on a marketing page probably shouldn't run forever if someone walks away.

Tell Claude Code what you want: "If the user hasn't spoken for 30 seconds, say goodbye and end the session," or "Stay open until the user explicitly ends it." You can also tune how sensitive the agent is to silence and background noise. Describe the behavior; Claude handles the settings.

The agent triggers on background noise

If your agent keeps responding to keyboard clatter, a TV, or a background conversation, that's a sensitivity setting you can dial down. If it cuts you off mid-sentence, the silence threshold is too short. These are turn detection problems, and they're genuinely one of the hardest parts of voice agent development. You don't have to find the settings yourself. Just describe it: "It keeps triggering when I'm not speaking, make it less sensitive," or "It interrupts before I finish." Claude knows what to adjust.

Something broke. How do I fix it?

Bugs will happen, and the fix is the same every time: copy the error message exactly as it appears, paste it into Claude Code, and say "I'm getting this error, fix it." Don't interpret it, don't Google the code, don't edit the code yourself. The error almost always contains what Claude needs, and the more of it you include, the faster it resolves.

Two things caught me specifically. The docs had a couple of field names wrong, and Claude corrected them by reading the error responses. And a URL changed silently at some point, which produced a confusing authentication error that had nothing to do with authentication. Both times, pasting the error into Claude and asking it to fix the problem was the right move.

The docs won't always be right

API documentation is written by humans and doesn't always keep up with the product. If something that looks correct isn't working, trust the error message over the docs. Claude Code is good at reading error responses and translating them into fixes, and it's often faster to let it try, read the error, and correct course than to study the docs first.

Your API key is a password, treat it like one

Before you share your URL, make sure your key isn't exposed. If someone finds it, they can rack up usage billed to you. Ask Claude Code to block requests from anywhere other than your own site, then verify it yourself. It's a five-minute addition that's easy to skip and worth not skipping.

Phase 4: Security before you ship

Before I made the URL public, I found a real hole: anyone who knew the URL could call /api/token directly and get a valid session token billed to my account. Each token allows up to an hour of usage. The fix is an origin allowlist:

const ALLOWED_HOSTS = [
  "your-domain.vercel.app",
  "www.yourdomain.com",
  "localhost",
  "127.0.0.1",
];
const origin = req.headers.origin || req.headers.referer || "";
if (!ALLOWED_HOSTS.some((host) => origin.includes(host))) {
  return res.status(403).json({ error: "Forbidden" });
}

Browsers send Origin headers automatically on cross-origin requests, so your site works normally while direct API calls and cross-site embedding are blocked. Verify it: curl https://your-app.vercel.app/api/token should return {"error":"Forbidden"}.

Try it and talk to it

If you want to build your own, the Voice Agent API documentation is where to start, with starter prompts you can grab and modify. Bring your knowledge base, describe your persona, and let Claude Code handle the scaffolding. If you'd rather see the shape of a good build first, the build guide walks through the same path end to end.

Under the hood, the Voice Agent API is built on Universal-3.5 Pro Realtime for speech-to-text, with LLM reasoning and text-to-speech handled in the same pipeline, one WebSocket connection at a flat $4.50 per hour, with roughly one second of end-to-end latency and session resumption if a connection drops within 30 seconds. That speech model posts a 6.99% word error rate on the Pipecat open STT benchmark of real voice-agent conversations, ahead of the alternatives, which matters more than it sounds when your agent has to catch phone numbers, order IDs, and names on the first try. When you're choosing a speech-to-text foundation for a voice agent, that combination of accuracy and simplicity is the whole game.

You're not the only one building this way

While I was building Pete, half the people I work with were doing the same thing. The AssemblyAI team ran an internal build sprint, everyone building their own agent from scratch, shipping it, and putting it in a showcase. The results were genuinely varied: medical scribes generating clinical notes in real time, a nail-salon receptionist handling bookings, national-park trip planners, and a tea recommender that interviews you about your taste.

None of these were built by people with years of voice AI experience. They were built by people who had an idea, described it to Claude, and figured it out as they went. Even noisy environments and turn detection tuning became solvable once the coding agent had the right docs.

Here's the insight I didn't expect walking in: the hard part was never the code. Claude Code did the heavy lifting on the technical work, and the Voice Agent API erased the infrastructure entirely. What actually took iteration was Pete, getting him to sound dry, blunt, and deeply resigned took more revisions than any bug. The system prompt changed more than the code did. So if you're building an agent for your business, the lesson is counterintuitive: spend at least as much time on the persona and the knowledge base as you do on the deployment, because the model already handles the parts you were afraid of. You don't need to understand the code to build something real. You need to know what you want it to do.

Frequently asked questions

Do I need to know how to code to build a voice agent with AssemblyAI?

No. The Voice Agent API handles speech recognition, LLM reasoning, and text-to-speech in a single managed pipeline, and there's no SDK to learn. You describe what you want to a coding agent like Claude Code, and it writes the integration. You need to be comfortable opening a terminal and running a command, but you don't need to understand or write the code yourself.

How long does it take to build and deploy a voice agent from scratch?

The full build, knowledge base, system prompt, working app, and deployment, took me a few days, most of it iterating on the persona. The actual technical setup, getting a working agent talking, happened in an afternoon. A developer starting from the docs can get a first agent running in under 30 minutes.

What does the Voice Agent API cost?

A flat $4.50 per hour of conversation, and that rate covers speech-to-text (built on Universal-3.5 Pro Realtime), LLM reasoning, and text-to-speech together. There are no per-token surcharges or hidden fees for individual pipeline components, and there's a free tier so you can build and test without a credit card.

What coding agent should I use to build a voice agent?

Claude Code, Cursor, GitHub Copilot, and Windsurf all work. Claude Code is particularly well-suited because of how it handles multi-file project generation and iterative debugging through conversation. The choice matters less than the setup: whichever agent you use, give it the AssemblyAI docs so it generates current, working code.

How do I stop the agent from cutting me off or responding to background noise?

These are turn detection and voice activity detection settings you can tune. You don't need to find them yourself. Describe the problem to your coding agent ("it keeps interrupting me" or "it triggers on keyboard noise") and it will adjust the silence thresholds and sensitivity for you.

Is a no-code builder or a coding agent better for building a voice agent?

A visual no-code builder is fine for a simple, linear bot with no engineering appetite. The coding-agent path gives you a real application you own, deployed on your own infrastructure, with custom logic and interface, and you still don't write the code yourself. For anything you want to grow or customize, the coding-agent path with the Voice Agent API holds up better.