Top 8 speaker diarization libraries and APIs in 2026

Speaker diarization answers one deceptively simple question: who spoke when? Get it right and every downstream feature—per-speaker sentiment, agent-vs-customer analytics, speaker-attributed summaries—just works. Get it wrong and all of that breaks, no matter how good your transcription is.

That's why diarization is usually one of the first things teams building conversation intelligence, voice agents, and meeting products evaluate. So I want to give you a straight comparison of the eight diarization solutions worth knowing in 2026, how to actually measure their quality, and which ones handle real-time streaming versus batch-only. I run Voice AI at AssemblyAI, so I'll be direct about where we lead and where I'd point you elsewhere.

One framing note up front. Most diarization benchmarks you'll see quote Diarization Error Rate (DER). DER is a fine academic metric, but it measures diarization in isolation from the transcript. In production what you care about is whether the right speaker label lands on the right words—which is what cpWER measures. Keep that distinction in mind, because it changes how the leaderboard looks.

What is speaker diarization?

Speaker diarization segments and labels an audio stream by speaker, so a transcript reads "Speaker A… Speaker B…" instead of one undifferentiated wall of text. In the field of automatic speech recognition (ASR), it does two jobs: it detects how many distinct speakers are in a file, and it attributes each segment of speech to the correct one.

Modern models do both with near human-level accuracy, and—this is the big shift of the last couple of years—they now run in real time on a single live audio stream, not just on finished recordings. That's what makes live captioning, contact center coaching, and voice agents possible without multichannel audio.

Async vs. streaming: the axis most teams overlook

Before the list, decide which mode you're building for, because it narrows your options fast.

Batch (async) diarization waits for the complete recording, then labels it. It can look at the whole file at once, so it's the most accurate option and the right choice for post-call analytics, meeting summaries, and media transcription.

Streaming diarization assigns labels live, as people talk. It can't look ahead—it commits to a label with only the audio it's heard so far—so the first few turns are the least stable. But if speaker identity has to drive behavior during the conversation, like a voice agent distinguishing the customer from a human agent, streaming is the only option. Here's how the leading solutions stack up on both.

The cpWER numbers come from our internal diarization benchmark, run across a mix of real-world datasets. Let's go solution by solution.

1. AssemblyAI

AssemblyAI is a Voice AI infrastructure platform built around highly accurate speech-to-text, with a full suite of Speech Understanding models for sentiment analysis, topic detection, summarization, and entity detection layered on top.

Our flagship async model, Universal-3.5 Pro (released July 7, 2026), ships the most accurate diarization we've ever put out. It's optimized directly for cpWER, and here's the head-to-head:

Because cpWER ties speaker labels to the actual transcribed words, that gap reflects what your users will feel—not just how cleanly the audio was segmented in the abstract. For even more precision, Speaker Identification replaces generic labels like "Speaker A" with real names or roles ("Customer," "Agent") inferred from the conversation.

Diarization runs in both async and real-time streaming. Streaming diarization runs on the same WebSocket you already use for transcription—wss://streaming.assemblyai.com/v3/ws—on Universal-3.5 Pro Realtime, the model that also powers our Voice Agent API. That makes it a fit for voice agents and live call intelligence, not just post-call analysis. Metaview, a meeting-intelligence platform, saw roughly a 47% drop in low-confidence tokens after moving to Universal-3.5 Pro async—a good proxy for how much cleaner the downstream speaker-attributed data gets.

Key strengths:

• Most accurate diarization AssemblyAI has shipped, reported as cpWER (30.17 avg) rather than DER alone
• Async and real-time streaming diarization on one API key
• Enhanced handling of similar voices and short utterances
• Async supports up to 20 expected speakers via speakers_expected; streaming supports up to 10 via a max_speakers hint
• Native code-switching across 18 languages on Universal-3.5 Pro, with Universal-2 available for 99+ language coverage
• Pricing: $0.15/hr for Universal-2, $0.21/hr for Universal-3.5 Pro async, and $0.45/hr base for Universal-3.5 Pro Realtime

One recent under-the-hood improvement worth flagging: for single-channel English, [Speaker]-based segmentation is now the default, using the model's own turn markers as segmentation boundaries, and speaker labeling got about 6× faster (a 60-minute file dropped from roughly 9.8s to 1.5s). Universal-3 Pro and Universal-3 Pro Streaming, the previous flagships, are still available as pinnable snapshots if you need version stability.

2. Deepgram

Deepgram's diarization emphasizes processing speed and language-agnostic operation, integrated with its Nova-3 model, and supports both batch and streaming.

Speed-first positioning suits high-throughput pipelines. On diarization cpWER, though, Nova-3 EN lands at 37.92 versus Universal-3.5 Pro's 30.17—and cleaner turn detection feeds cleaner inputs to the diarizer, so accuracy on real-time speaker attribution tracks with transcription quality. On the Pipecat open STT benchmark (real voice-agent conversations), Universal-3.5 Pro Realtime posted 6.99% WER versus Deepgram Flux's 15.58%.

3. ElevenLabs Scribe

Scribe is worth calling out separately because it's an ASR-first provider whose diarization is genuinely strong—Scribe v2 comes in at 35.26 cpWER, ahead of Deepgram and Gladia in our runs. If you're already in the ElevenLabs ecosystem, it's a reasonable option for async, speaker-labeled transcription. On live voice-agent audio it trails, though: 9.76% WER on the Pipecat benchmark versus 6.99% for Universal-3.5 Pro Realtime, and a 39.70% entity error rate versus our 15.31%.

4. Gladia

Gladia combines Whisper transcription with PyAnnote diarization, which gives teams already using Whisper a path to add diarization without stitching together multiple services. It offers an enhanced diarization mode for challenging audio, configurable speaker hints, and streaming support. Diarization cpWER sits at 36.87 in our benchmark.

5. Speechmatics

Speechmatics offers diarization through its Flow platform with both cloud and on-premise deployment, plus real-time support. Its strengths are punctuation-based accuracy corrections, configurable maximum speakers (2–20), and 30+ language support. Deployment flexibility is the real draw here—if you have strict on-premise or compliance requirements, it's a strong candidate.

6. PyAnnote

PyAnnote is the widely used open-source diarization toolkit, now on version 3.1, and it's the foundation under several commercial services (including Gladia). It reports roughly 10% DER with optimized configs on standard benchmarks and runs at a low real-time factor on GPU.

A caveat on those numbers: they're pure DER, on diarization in isolation from transcription. pyannote.ai also publishes strong streaming DER on datasets like DIHARD III, and that's a real result—but DER doesn't tell you whether the right label landed on the right words in your transcript. PyAnnote is best for research teams with ML expertise who want a customizable, self-hosted solution and are willing to fine-tune.

7. NVIDIA NeMo

NeMo's Sortformer is an end-to-end diarization approach using an 18-layer Transformer that treats diarization as a unified problem rather than a multi-stage pipeline. It supports both oracle and system VAD, integrates with ASR, and is GPU-optimized. It's aimed at researchers and teams building custom multi-speaker ASR with GPU resources on hand.

8. Kaldi and SpeechBrain

Both are open-source research toolkits. Kaldi is the long-standing, highly configurable speech recognition framework with X-Vector and PLDA backends for diarization—maximum control, steep learning curve. SpeechBrain is the PyTorch-based option with 200+ recipes, modular design, and an active community. Both are best for academic research and prototyping rather than production, where the engineering and infrastructure overhead of self-hosting starts to dominate.

Real-time (streaming) speaker diarization

For a long time, reliable diarization meant batch processing, and the only solid path to real-time speaker separation was multichannel audio with each speaker on a dedicated channel. That's no longer the case.

Streaming diarization now identifies who's speaking on a single live stream. With AssemblyAI you enable it by adding speaker_labels: true to a streaming WebSocket connection, plus an optional max_speakers hint (up to 10) when you know the count. Labels are emitted live as people talk, and the system issues a single revision within roughly 0.5 seconds of the stream ending—so late-arriving context can correct an early call without your UI flickering through constant relabeling mid-conversation.

A few practical notes for streaming:

• Turn detection drives latency. The biggest delay isn't the diarization—it's waiting for someone to finish talking. Neural turn detection that uses both acoustic and linguistic signals beats voice-activity detection alone, which misfires on mid-sentence pauses.
• Short utterances are hard. Brief responses like "yeah" with under a second of audio may come back UNKNOWN; handle that gracefully in your client.
• Two speakers is the sweet spot. Accuracy is highest with two speakers and degrades as the count climbs.
• Multichannel is still great when your infrastructure already separates speakers onto channels (common in telephony)—perfect separation, no diarization overhead.

One honest caveat: I'd avoid blanket "we lead streaming diarization" claims, because a competitor can point to a DER-only streaming benchmark and tell a different story. The defensible position is that cpWER and speaker-count accuracy reflect production value better than isolated DER—and that's where we focus.

How to evaluate speaker diarization quality

Most published benchmarks report a single number: Diarization Error Rate (DER), which combines missed speech, false alarms, and speaker confusion. Lower is better, and production systems often target DER below 10%.

Here's the catch, and it's the most important thing in this guide: DER alone doesn't reflect what most teams care about. DER measures diarization in isolation from transcription. In production what matters is whether the right speaker label lands on the right words, whether timestamps are precise, whether overlapping speech is attributed correctly, and whether the system gets the number of speakers right. A system can post a great DER and still produce transcripts that feel wrong to your users.

That's why we report cpWER—which ties labels to the actual transcribed words—and speaker count error rate alongside DER. Be skeptical of standalone, DER-only benchmarks from diarization-only tools: a strong DER on a research dataset doesn't guarantee accurate, word-aligned labels on your audio.

For streaming, two more metrics matter:

• Latency — the delay between speech and a labeled transcript. 200ms is fine for a voice agent; two seconds isn't.
• Label stability — how often labels change or correct themselves. Constant flipping makes a UI look broken even when the final transcript is right.

Whatever you track, test on audio that matches your real use case—turn-taking speed, quality variation, short utterances, overlapping speech—rather than relying on published benchmark numbers alone.

How to choose a speaker diarization solution

For production applications:

• Highest word-aligned accuracy → AssemblyAI (30.17 cpWER, most accurate diarization we've shipped)
• Real-time voice agents and live coaching → a provider with mature streaming diarization (AssemblyAI, Deepgram, Speechmatics)
• Whisper ecosystem → Gladia's integrated Whisper + diarization
• On-premise or compliance-driven deployment → Speechmatics

For research and development, the open-source options—NeMo's Sortformer, PyAnnote 3.1, Kaldi, SpeechBrain—give you model control at the cost of engineering time, GPU infrastructure, and ongoing maintenance. Many teams start on an API to validate the use case, then evaluate self-hosting once they understand their scale.

Frequently asked questions

What is the best speaker diarization API in 2026?

For word-aligned accuracy, AssemblyAI's Universal-3.5 Pro leads our benchmark at 30.17 average cpWER, ahead of ElevenLabs Scribe v2 (35.26), Gladia (36.87), and Deepgram Nova-3 EN (37.92). "Best" depends on your needs, though—Speechmatics wins on on-premise flexibility, and open-source PyAnnote or NeMo win when you need full model control.

What's the best real-time speaker diarization API for voice agents?

You want mature streaming diarization plus strong live-audio transcription. AssemblyAI runs streaming diarization on the same WebSocket as transcription (wss://streaming.assemblyai.com/v3/ws) on Universal-3.5 Pro Realtime, which posted 6.99% WER on the Pipecat voice-agent benchmark—versus 9.76% for ElevenLabs Scribe v2 and 15.58% for Deepgram Flux.

What's the difference between DER and cpWER?

DER measures diarization in isolation—missed speech, false alarms, and speaker confusion as a percentage of audio time. cpWER measures whether the right speaker label landed on the right transcribed words. cpWER is the metric that reflects what your users actually experience, which is why we lead with it.

How many speakers can speaker diarization detect?

It depends on the provider and mode. AssemblyAI supports up to 20 expected speakers for async diarization (via speakers_expected) and up to 10 for streaming (via max_speakers). Accuracy is highest with two speakers and decreases as the count rises, since there's less audio per person to build a reliable voice profile.

Are open-source diarization libraries good enough for production?

They can be, but there's a real cost. PyAnnote, NeMo, Kaldi, and SpeechBrain give you full control and no per-usage fees, but you take on deployment, GPU infrastructure, scaling, and the work of tracking new research. APIs ship faster and update automatically. A common pattern is to start on an API, then evaluate self-hosting once you understand your scale.

Does speaker diarization work in real time?

Yes. Streaming diarization labels a single live audio stream as people talk, with a single revision within about 0.5 seconds of the stream ending to correct early calls. It's the right choice for voice agents, live captioning, and real-time coaching. For pre-recorded files, async diarization remains the most accurate option.