What is speaker diarization and how does it work? (Complete 2026 Guide)

Speaker diarization is the process of partitioning an audio recording by speaker identity — figuring out who spoke when, and attaching a consistent label like Speaker A or Speaker B to every stretch of speech. It doesn't need to know anyone's name. It just needs to tell the voices apart and keep them straight across the whole file.

If you've ever read an auto-generated transcript of a meeting and couldn't tell who said what, you've felt the absence of good diarization. Add it, and a flat wall of text turns into a readable conversation. (Quick aside on the word itself: "diarization" is pronounced die-uh-rih-ZAY-shun, and to diarize simply means to split audio into speaker-labeled segments.)

How does speaker diarization work? The short answer

Here's the TL;DR before we go deep. A diarization system takes an audio stream and runs it through four moves: it detects where speech exists and drops the silence, segments that speech into short homogeneous chunks that each belong to a single voice, embeds each chunk into a numerical voiceprint that captures its acoustic fingerprint, and then clusters those voiceprints so that segments from the same person get the same label. The output is a timeline of turns — start time, end time, speaker label — that you can lay over a transcript so every word lands with the right person. That's the whole idea. Everything below is detail.

Why speaker diarization matters

Transcription answers "what was said." Diarization answers "who said it." You need both before a transcript becomes something you can actually work with.

Look at the difference on a two-person call. Without diarization, you get an undifferentiated block:

"Hi, thanks for calling support. My internet's been down since this morning. I'm sorry to hear that, let me pull up your account. Sure, it's under Jordan Lee."

With diarization, the same audio reads like a conversation:

• Speaker A: Hi, thanks for calling support.
• Speaker B: My internet's been down since this morning.
• Speaker A: I'm sorry to hear that, let me pull up your account.
• Speaker B: Sure, it's under Jordan Lee.

That structure is what makes downstream work possible — measuring talk-time ratios in a sales call, isolating the patient's words from the clinician's, attributing action items to the right person in a meeting summary, or scoring an agent's turns in a contact center. Once speech is attributed, everything built on top of it, from conversation intelligence to compliance review, gets sharper.

The four steps in detail

1. Voice activity detection

First the system decides which parts of the audio are actually speech and which are silence, music, or background noise. This is voice activity detection, and it matters more than it sounds — feed non-speech into later stages and you'll get phantom speakers and smeared boundaries. Good VAD is the foundation everything else stands on.

2. Segmentation

Next the speech gets cut into short segments, each ideally containing just one speaker. The tricky part is the boundaries. People interrupt each other, finish each other's sentences, and talk over one another. Modern systems place speaker-change points precisely so a single segment doesn't accidentally straddle two voices.

3. Embedding

Each segment is converted into a speaker embedding — a vector, often a few hundred dimensions, that encodes the acoustic characteristics of the voice: pitch, timbre, cadence, and the subtle qualities that make one person sound different from another. Two segments from the same speaker land close together in this vector space; two different speakers land far apart. These embeddings are what make the same person recognizable across a long recording.

4. Clustering

Finally the embeddings are grouped. Clustering assigns every segment to a speaker so that all of one person's turns share a label. This is also where the system estimates how many speakers are in the audio, which is often unknown going in. When you can tell the system to expect a specific number of speakers — through a parameter like speakers_expected — clustering gets easier and more accurate, because you've removed the hardest guess. If you can't, the model infers the count on its own.

Getting the speaker count right is one of the quiet hard problems in diarization. Guess too few and you merge two people into one label; guess too many and you split one person across several. For a deeper look at the recognition layer underneath all of this, see our explainer on what automatic speech recognition is.

Three architectural approaches

There's more than one way to wire these steps together, and the approach shapes the tradeoffs you inherit.

• Modular (pipeline) systems run VAD, segmentation, embedding, and clustering as separate stages. Each stage is easy to inspect and swap, which makes debugging straightforward — but errors compound, so a bad segmentation boundary poisons everything downstream.
• End-to-end neural systems train a single model to map raw audio directly to speaker-labeled output. They handle overlapping speech more gracefully because they optimize the whole task at once, but they're hungrier for training data and harder to interpret when something goes wrong.
• Hybrid systems blend the two — neural components for the parts that benefit from end-to-end learning, modular structure for control and transparency. In practice this is where most production-grade diarization lives, because it captures most of the accuracy of end-to-end models while staying debuggable.

How diarization is measured: DER vs cpWER

You can't improve what you can't measure, and here's where a lot of vendor marketing gets misleading. The traditional metric is Diarization Error Rate (DER), which measures the fraction of audio time attributed to the wrong speaker. It's useful, and it's the number most tools quote — but it has a real blind spot: DER scores time, not words. A system can post an impressive DER while still hanging the wrong words on the wrong speaker, which is exactly the failure your users notice in the transcript.

That's why we lead with concatenated minimum-permutation Word Error Rate (cpWER). cpWER evaluates the diarized transcript the way a reader experiences it: it ties speaker labels to the actual words and penalizes attribution mistakes at the word level. In production, cpWER plus speaker-count accuracy reflects customer value far better than DER alone. So when a competitor waves a strong DER number, ask what the cpWER looks like — the gap between the two is often where the real quality lives.

On our internal diarization benchmark, Universal-3.5 Pro posts an average cpWER of 30.17 — ahead of Deepgram Nova-3 EN at 37.92, ElevenLabs Scribe v2 at 35.26, and Gladia at 36.87 (lower is better). You can see the full methodology on our benchmarks page. If you want to go deeper on why single-number accuracy scores can mislead, our post on why Word Error Rate is broken covers the same trap from the transcription side.

Diarization on recordings vs. live streams

Diarizing a finished recording and diarizing a live stream are genuinely different problems. With a recording, the model can see the whole file at once, cluster across the entire timeline, and make globally consistent decisions. Real-time diarization has to commit to a label the instant someone speaks, before it knows what comes next.

This is where an old myth needs to die. For years, guidance around diarization warned that you needed long chunks of speech per speaker — think 15 to 30 seconds — before a system could reliably tell voices apart. That floor is outdated. Modern models like Universal-3.5 Pro reliably identify speakers from segments as short as 250 milliseconds. A quick "yeah" or "no problem" is enough to attribute. The short-utterance problem that used to define diarization is largely solved.

For streaming, we handle the commit-early tradeoff with a revision step. Universal-3.5 Pro Realtime emits live speaker labels turn by turn as the conversation unfolds, then issues a single revision within about 0.5 seconds of the stream ending to clean up any early calls it can now improve with full context. It supports up to 10 speakers in a live session. You turn it on with speaker_labels: true and connect to wss://streaming.assemblyai.com/v3/ws; the max_speakers parameter caps the count when you know it in advance. We break down the mechanics in our posts on streaming speaker diarization and the major streaming diarization upgrade.

One recent async improvement worth calling out: [Speaker]-based segmentation is now the default for single-channel English, which sharpens speaker-change detection and makes speaker labeling roughly 6x faster than the previous approach.

Where speaker diarization gets used

Diarization shows up anywhere multiple people share a microphone. A few of the most common places:

Metaview, which builds AI notetaking for recruiters, saw roughly a 47% drop in low-confidence tokens after moving to Universal-3.5 Pro async — a reminder that better underlying transcription and diarization directly reduce the cleanup work downstream teams have to do.

Build it yourself or call an API?

If you want full control and don't mind the maintenance, open-source toolkits like pyannote.audio and NVIDIA NeMo let you assemble and tune every stage of the pipeline yourself. That flexibility is real, and so is the cost: you own the models, the infrastructure, the scaling, and the accuracy tuning. Our roundup of the top free speech-to-text APIs and open-source engines is a good starting point if that's your path.

An API takes the pipeline off your plate — you send audio, you get back a diarized transcript, and someone else keeps the models current. This post is deliberately about how diarization works rather than which vendor to pick; if you're comparing accuracy, pricing, and features across providers, that's a different question, and we cover it in our dedicated guide to the top speaker diarization libraries and APIs. Either way, understanding the four steps above will make you a sharper evaluator.

Where diarization is heading

Here's the shift I'd point to that most explainers miss: diarization is quietly changing from a post-processing afterthought into a real-time primitive. For most of its history, diarization was something you ran on a file after the fact. But once you can attribute a 250-millisecond utterance and correct yourself within half a second of a stream ending, diarization stops being a report you read later and starts being live context an application can act on while the conversation is still happening. That's what makes responsive voice agents, real-time coaching, and live compliance possible — and it's why the metric you optimize for (cpWER, not just DER) increasingly determines whether those experiences feel trustworthy. The teams that treat diarization as live infrastructure, not a batch job, are the ones building the next generation of voice products.

Frequently asked questions

What is the difference between speaker diarization and speaker recognition?

Diarization answers "who spoke when" without knowing anyone's identity — it just separates and labels distinct voices as Speaker A, Speaker B, and so on. Speaker recognition (or verification) goes a step further and matches a voice to a known identity. Diarization tells the voices apart; recognition names them.

What is speaker role diarization?

Speaker role diarization assigns functional roles rather than anonymous labels — for example, "agent" and "customer" in a support call, or "clinician" and "patient" in a medical visit. It builds on standard diarization by mapping each detected speaker to the role they're playing in the conversation, which is often more useful downstream than a generic Speaker A/B label.

How short an utterance can diarization handle?

Modern models identify speakers from segments as short as 250 milliseconds. The old guidance that you needed 15 to 30 seconds of speech per speaker is outdated — brief interjections like "yeah" or "got it" are now enough to attribute correctly.

Can speaker diarization work in real time?

Yes. Universal-3.5 Pro Realtime produces live speaker labels as a conversation unfolds and then issues a single revision within about 0.5 seconds of the stream ending to refine any early calls. It supports up to 10 speakers per live session and is enabled with speaker_labels: true.

How many languages does diarization support?

Universal-3.5 Pro handles diarization with native code-switching across 18 languages, so it can track speakers even when they switch languages mid-conversation. For the broadest language coverage on recorded audio, Universal-2 remains available with 99+ languages as a legacy option.

How is diarization accuracy measured?

The traditional metric is Diarization Error Rate (DER), which scores the fraction of audio time attributed to the wrong speaker. We prefer concatenated minimum-permutation Word Error Rate (cpWER), because it ties speaker labels to the actual words in the transcript and better reflects the quality a user experiences.