DER vs. cpWER: why the standard diarization metric ranks systems backwards
If your product tells users who said what (meeting notes, sales coaching, CRM logging), the metric you want to optimize is cpWER. This post shows, on real recordings you can listen to, why Diarization Error Rate (DER) is the wrong metric to optimize for that job, and why measuring speaker-labeled transcripts with DER is tricky in the first place.
Start with two real 30-second clips: a design meeting from the AMI corpus and a dinner party from DiPCo,
each with a system output printed underneath, timestamps included. (A note on provenance: the “system
outputs” in this post are manually constructed for illustration, so that the metric—not any model—is the
only variable; the exception is the max_gap study later, which uses real production output.)
Listen to both, read the transcripts, and decide which output you'd rather ship. Then look at what DER
says.
Example 1: the meeting-planning clip (AMI)
Transcript perfect, every word on the right speaker; utterances grouped into readable paragraphs
Example 2: the dinner-party clip (DiPCo)
A guest vanishes and the storyteller “agrees with himself” (timestamps fine)
DER ranks the broken output 36 points above the flawless one. cpWER ranks them the way anyone who listens would. The rest of this post walks through both clips, shows the same effect on a full production transcript, and explains which metrics actually track what your users experience.
A flawless transcript with a “broken” DER
AMI corpus, meeting EN2002d (four colleagues wrapping up: “should we meet up Monday?”, “we'll probably have to talk to Steve at some point”), 30 seconds starting at 20:20. The system output below is perfect: every word right, every word on the right speaker, word timestamps correct. There is no laughter, no crosstalk trickery, nothing exotic in this clip. The system made exactly one formatting choice, the same one every meeting product makes: it groups each speaker's consecutive utterances into readable paragraphs instead of splitting them at every pause.
AMI EN2002d · 20:20–20:50 · headset mix · listen for A's and D's natural pauses mid-turn
Reference (AMI segment labels)
System output: same words, same speakers, paragraphs
0.0%
cpWER: all 111 words correct, all attributed to the right speaker
51.5%
DER: almost entirely “false alarm” charged for the silent pauses inside merged paragraphs
0.3%
DER for the identical transcript split at every 0.25s pause; formatting is the whole difference
Side note: the transcript text itself reads rough: lowercase, no punctuation, verbatim disfluencies (“just sh like w”). That is AMI's human ground-truth annotation style, which we reuse verbatim so that the words are perfect by definition and the metrics are the only variable. It is not the output of our Universal-3.5 Pro model family, which produces formatted, punctuated transcripts.
What happened? The AMI reference labels speech tightly: speaker A's turn is annotated as three short segments—“should we meet up monday”, “or”, “should we”—with silent gaps between them. The system's paragraph for A covers the same words but spans the gaps, and DER bills every second of bridged silence as false-alarm speech. Merge pauses up to 3 seconds (normal paragraphing) and the penalty reaches 51.5%. Split at every quarter-second pause instead and the identical transcript scores 0.3%.
Neither construction is wrong. One of them just happens to match the pause conventions the corpus annotators used, and the other doesn't. And this choice is unavoidably yours as the evaluator: speech-to-text APIs (ours and most others') return words with speaker labels, not diarization segments. Whoever runs a DER benchmark must build segments out of the words, and as the bars above show, that decision alone moves DER across the entire quality scale. A DER comparison between two providers is, to a first approximation, a comparison of two segment-formatting conventions.
The transcript is identical and perfect in all three rows: cpWER says 0.0% every time. DER says anything between 0.3% and 51.5%, depending on a formatting decision that has nothing to do with who said what.
An obviously broken transcript with a “good” DER
So far DER has punished output that deserved a perfect score. The trap has a second jaw: it also forgives output that deserves a terrible one. DiPCo corpus, session S02 (a dinner party), 30 seconds starting at 22:40. P08 tells a story about a dog falling out of a pickup truck; P07 tops it with a bear riding in a Mustang; the others react: “a bear?!”, “a grizzly, yeah”. This time the system's timestamps are fine (within ±0.15s throughout). The mistakes are in the names: every short reaction is credited to whoever held the floor, and P06, who speaks only once, is merged into P07 entirely.
DiPCo S02 · 22:40–23:10 · far-field mic · the “a bear?!” round starts ~15s in
Reference (human annotation)
System output
15.1%
DER: a genuinely competitive score on far-field dinner-party audio
30.7%
cpWER: nearly a third of all words credited to the wrong person
3 of 4
speakers found; P06 does not exist in this output
Read the output on the right: P05 apparently interrupts P08's dog story to say “yeah… a bear… a bear” to himself, P07 answers his own bear story with “a grizzly, yeah… okay”, and P06 has left the dinner party: his one contribution is now P07's. A meeting-notes product built on this output gets the participant list wrong and almost every reaction misattributed.
DER barely notices: 15.1%, far better than Example 1's 51.5%. The reason is structural. DER is a time-weighted metric, and these errors are short in seconds: a half-second “yeah” here, a one-second “a bear” there, a guest who talks for only 2.4 of 30 seconds. But they are exactly the seconds a who-said-what product exists for: the agreements, the reactions, the quiet participant. cpWER, which weights by words and charges a misattributed word on both the losing and the gaining speaker, reads 30.7%, and a simple speaker count check flags the missing guest.
A caveat on absolute numbers: this is a deliberately reaction-dense 30 seconds, chosen to make the failure audible. On full-length far-field benchmarks (hours of dinner-party and meeting audio like DiPCo), cpWERs in the 30s are typical for every vendor. What a metric owes you there is the right ranking between systems, and that is exactly where DER falls down.
DER prefers this output to Example 1's by 36 points. Anyone who listens to the two clips prefers Example 1's without hesitation. When a metric and your ears disagree this reliably, the metric is measuring the wrong thing.
The same diarization scores 48% or 15%: the only thing that changed is max_gap
Both examples above use outputs we constructed, so that the metric, not any particular model, is the
only variable. This section repeats the point with nothing constructed at all. DER rebuilds a predicted
timeline by stitching consecutive same-speaker words into segments, bridging gaps only up to
max_gap (a common default: 0.25s). We scored our production model's transcript of one full
AMI meeting: TS3003c, 3,799 words, ~4 speakers; collar 0.25s, overlap skipped. At the default, that
convention counts almost every within-turn pause as missed speech, inflating DER to 47.8%. Relax the
bridge and DER collapses into a 13–15% basin, then overshoots again toward ∞. Nothing about the model's
output changes anywhere in this section.
47.8%
DER @ max_gap 0.25, the current convention; 44.7 points of it are missed detection
14.5%
DER @ max_gap 3.0: −33.3 points; same audio, same labels
19.0%
DER @ max_gap ∞: overshoot; false alarm balloons to 14.1 points
DER decomposed as max_gap grows
Missed detection dominates the error and collapses as the bridge widens; false alarm creeps up as the bridge starts claiming real silence. Their sum, DER, is a U: minimized near 3–5s, worse at both 0.25 and ∞. Speaker confusion stays ~1% throughout, so the speaker assignment was right the whole time; the metric was scoring timeline granularity.
Why: a correct single-speaker stretch, shredded into misses
A 20–50s window of the same meeting. In the reference, speaker MTD009PM talks almost continuously. The
prediction's word timestamps are correct but tight: natural 0.7–2s pauses sit between phrases. At
max_gap 0.25 each pause splits the turn into 13 fragments, and the silence inside real
speech is scored as missed detection. At 3.0 the turn is made whole again; only where the bridge
crosses genuine reference silence does it flip to false alarm.
the 20–50s window · the speaker is talking almost the whole time · the playhead tracks the chart
The full sweep
The per-segment count shows the mechanism: 1,015 fragments at 0.25s collapse to 158 at ∞.
| max_gap | DER | miss | false alarm | confusion | pred segments |
|---|---|---|---|---|---|
| 0.25 | 47.8 | 44.7 | 2.4 | 0.7 | 1015 |
| 0.5 | 42.7 | 39.6 | 2.4 | 0.7 | 748 |
| 1.0 | 32.3 | 29.1 | 2.5 | 0.7 | 478 |
| 2.0 | 19.0 | 14.2 | 3.9 | 0.9 | 258 |
| 3.0 | 14.5 | 8.4 | 5.0 | 1.1 | 206 |
| 5.0 | 13.0 | 4.2 | 7.6 | 1.2 | 172 |
| ∞ | 19.0 | 3.6 | 14.1 | 1.3 | 158 |
Recomputed locally from cached benchmark responses via pyannote.metrics DiarizationErrorRate;
max_gap applied when reconstructing predicted speaker segments. Reference = AMI ground
truth (MTD009PM is the dominant speaker in the window). Confusion = DER − miss − false alarm.
- DER ≠ diarization quality here. At the current 0.25s convention, 44.7 of 47.8 points are missed detection and only ~1 point is speaker confusion. The model put the right speakers in the right order; it was penalized for emitting tight word timestamps.
-
max_gapis a tunable dial, not a fix. Widening it trades miss for false alarm. The 3–5s basin recovers the within-turn pauses without over-claiming; ∞ merges across genuine silence and swings DER back up to 19.0%. The number can be moved ~33 points by convention alone. - Implication for comparisons. Systems with tight per-word timestamps, like ours, are
the ones this convention penalizes most, and helps most when relaxed. A fair cross-provider DER has to
fix
max_gap(and collar, and overlap) before ranking anything.
One more thing to check on AMI: is laughter in the reference?
Everything so far applies to DER on any dataset. AMI, the dataset most DER benchmarks reach for first, has one more scoring option worth checking. AMI's annotators labeled more than words: laughter, coughs, and breaths are annotated per speaker as vocal sounds, a category separate from the transcript text. Most scoring setups exclude them from the reference, and that is the sensible default—but some reference builds keep them in as speech, and a DER computed that way quietly grades a task few products need: attributing laughter to the right person.
Here is what that does in practice. In this clip, speaker C makes a joke—“that's what programming is all about, I'm afraid”—and the room cracks up: B laughs for a full seven seconds while D talks through it; 11 of the 30 seconds contain annotated laughter.
AMI EN2002b · 25:15–25:45 · headset mix · the group laugh starts ~17s in
Reference (AMI annotation, vocal sounds included)
System output
1.1%
cpWER: one “yeah” drowned by the laughter; all 87 other words right, all on the right speaker
23.4%
DER with vocal sounds in the reference: all of it missed-speech penalty for unattributed laughter
1.1%
DER with vocal sounds excluded: same output, same audio
The system transcribed 87 of 88 words correctly, every one on the right speaker; it just didn't attribute anyone's laughter, because it transcribes speech. Against a vocal-sounds-included reference, DER charges those 11 seconds as missed speech: 23.4%. Exclude vocal sounds from the reference (a checkbox in the scoring setup, and the more common choice) and the same output scores 1.1%. A 22-point swing controlled by an option that benchmark reports don't always state.
Two things follow. First, AMI DER numbers are only comparable when the reference build (vocal sounds in or out), collar, and overlap handling are all identical—worth verifying before putting two published numbers side by side. Second: if your product needs laughter at all, it usually needs “laughter detected”, not “speaker B laughed from 16.8s to 23.8s”. Optimizing for a vocal-sounds-included reference spends your accuracy budget on per-person laughter attribution, effort your users will never see. cpWER is unaffected either way: vocal sounds are a separate annotation category and never appear in the transcript text, so both scoring conventions give the same 1.1%.
A 22-point swing from a scoring checkbox is a property of the setup, not of the system. On AMI: check how the reference treats vocal sounds before you trust—or compare—DER numbers.
Why DER behaves like this
None of this is a bug in any particular scorer. DER predates speaker-attributed transcription: it compares time regions (who was speaking when) and totals missed speech, false-alarm speech, and speaker-confused speech as a fraction of reference speech time. Words never enter the computation. That design has three consequences, and you have now heard all of them:
- It needs segments that transcription APIs don't return. The evaluator must reconstruct segments from word-level output, and every reconstruction choice (pause merging, padding, overlap handling) changes the score (Example 1: 0.3% → 51.5% with identical words; 47.8% → 14.5% on real production output of a full meeting). Scoring conventions like the collar width and whether overlap regions count add further swings on top.
- It scores things users never see. Silent pauses inside a paragraph, segment padding, annotated non-speech: all move DER by tens of points with no change to the transcript.
- It weights by seconds, not by importance. A backchannel that decides “who agreed” costs milliseconds of DER; erasing the quietest participant costs almost nothing (Example 2). The failure modes customers report most are nearly invisible to it.
The metrics that match what your users see
If DER measures the wrong thing for a who-said-what product, what measures the right thing? cpWER and speaker count error are two representative alternative metrics, and they are what we optimize our own diarization against.
cpWER: the transcript metric
Concatenated minimum-permutation WER concatenates each speaker's words into one stream, finds the reference↔hypothesis speaker mapping that minimizes total word error, and reports the word error rate under that best mapping. In plain terms: for each person, what fraction of their words did the system get wrong: misrecognized, dropped, or credited to someone else? It moves when the user experience moves (Example 2) and stays put when the user experience is fine (Example 1). And its reference needs no timestamps: a human-checked, speaker-labeled transcript is enough, which makes building an eval set on your own audio roughly 10× cheaper than the frame-accurate annotation DER demands.
Speaker count error: the roster metric
The absolute difference between predicted and true number of speakers, averaged over files. It is the simplest metric that catches the two failure modes behind most customer escalations: merged speakers (Example 2's vanished P06) and phantom speakers (one voice split into several). Both corrupt everything built per-person downstream (CRM matching, talk-time analytics, coaching), and both can hide inside a good DER.
The bottom line
Every number in this post came from the same recipe: hold the output fixed, turn a knob DER exposes—pause
merging, max_gap, what counts as reference speech—and watch the score travel across
the entire quality scale while the transcript doesn't change at all. And when the transcript did
break (Example 2), DER barely moved. A metric that responds to formatting more than to content cannot
rank systems whose product is the content.
So when you evaluate speaker attribution for a who-said-what product, measure what your users read: cpWER, on speaker-labeled transcripts of your own audio, together with speaker count error to catch merged and phantom speakers. That is how we evaluate our own systems, and it is the test your users are already running, every time they open a transcript.
To see where our diarization stands on these metrics today, read the Universal-3.5 Pro announcement, or try it on your own audio in the Playground.