Speaker Diarization Accuracy Across Accents and Dialects

Word error rates in ASR have dropped for three decades straight, to the point where clean, single-speaker, native-accent transcription is close to solved for most everyday uses. Diarization is a different animal. Multi-speaker, spontaneous conversation stays genuinely hard, and even the leading systems in 2025 post error rates high enough that a transcript needs a human to check the speaker labels before anyone trusts them.

The published numbers depend heavily on what's being tested, and this is where a lot of vendor claims fall apart under a second look. Feed a system clean audio with two speakers taking clean turns, and DER (diarization error rate) can look small enough to seem solved. Feed the same system spontaneous, overlapping, domain-specific audio, and the number climbs fast. The HOARD framework, for instance, posted a high DER on the AMI meeting corpus and a steep DER on the DISPLACE 2024 multilingual dataset, and those aren't rigged tests built to embarrass anyone. AMI and DISPLACE are standard benchmarks that every serious lab already uses.

Here's what trips up anyone comparing vendors: DER numbers from different papers aren't apples to apples. Collar tolerances (how much timing slack gets forgiven around a speaker change), dataset splits, and even the exact definition of the metric shift from one benchmark to the next, so a system that tops the leaderboard on one dataset can land mid-pack on another. "State of the art" here is a conditional statement, true only given this audio, this speaker count, this language, this domain. Change one variable and the ranking shuffles. Treat a vendor's headline DER the way you'd treat a car's fuel economy sticker: technically true, under conditions nobody actually drives in.

How accents and dialects introduce acoustic variation that diarization systems weren't built for

Modern diarization runs on speaker embeddings: short clips of audio get turned into fixed-length vectors meant to capture the acoustic fingerprint of a voice, and the system clusters those vectors to decide who's talking. This works fine when the voices being clustered resemble the voices the embedding model learned from.

That's the catch, and it's the whole ballgame. Embedding models learn what "a voice" sounds like from training data, and training data is never a neutral sample of humanity; it's a sample of whoever got recorded, transcribed, and included, which has historically skewed toward a narrow band of accents and dialects. A speaker whose vowels, consonants, or speech rhythm fall outside that band runs into a model that never built a good representation of that acoustic space to begin with. The effect looks like unfair treatment whether or not anyone intended it.

Accent and dialect stack several kinds of variation on top of ordinary between-speaker differences. Phonological shifts change vowels and consonants; prosodic differences change rhythm and stress; lexical and syntactic differences don't touch diarization directly but corrupt the ASR output that usually feeds the same pipeline. Code-switching adds another layer: the same phrase can mean different things depending on which language surrounds it, so a system doing moment-to-moment language detection works with far less context than it needs.

Overlap makes everything worse, and not by coincidence. Spontaneous, accent-diverse conversation produces more overlapping speech than the polite, turn-taking audio in a lot of training sets, and overlap is already where every diarization system does its worst work. Stack accent variation on top of that and the errors don't add, they multiply.

One finding is worth sitting with: research has found that even after controlling for linguistic factors, ASR performance correlates with the political alignment between a speaker's birth country and the United States. That points upstream, toward whose speech got collected in the first place, sitting ahead of any model architecture anyone could fix with a better algorithm.

Measured performance on African-accented English: what the AfriSpeech-Dialog benchmark found

AfriSpeech-Dialog, introduced at NAACL 2025, is a benchmark built specifically to test diarization and ASR on African-accented English conversation, using 50 simulated dialogues split between medical and general contexts. Across eight diarization systems, the headline number was a performance drop of more than 10% against native-accent baselines. That's the topline, and it undersells the real story sitting underneath it.

Clinical speech averaged 33.38% DER, while general speech averaged 15.18%. That's more than double, and it's the gap between "usable with light editing" and "roughly a third of the conversation is misattributed or missing." Clinical speech is harder on two fronts at once: it's acoustically dense with domain-specific terminology, and the register shift between clinician and patient adds a layer the model has to track on top of everything else. Stack that on an already underserved accent population, and the error rate doesn't add up, it compounds.

Fine-tuning offered a real fix, though only a partial one. Adapting the Pyannote segmentation model on in-domain African-accented English data brought overall DER from 21.30% down to 10.65%, close to cutting the error in half. Worth taking seriously: that's evidence the gap is fixable, not fixed. Fine-tuning narrowed the medical-general split but didn't close it. The baseline model reflected a mismatch with this population's acoustic distribution, and pointing it at the right data helped a lot without solving the underlying problem.

Scale matters for context here. The broader AfriSpeech corpus, referenced in the same body of research, covers a wide range of speakers and indigenous accents spanning multiple African countries. That's the size of the blind spot standard training sets carry around: a continent's worth of phonological variation that most commercial systems have simply never heard once.

The ASR bias finding for African American English and what it implies for diarization pipelines

Diarization and ASR are usually bolted together in production, so an ASR failure doesn't stay an ASR failure. It becomes a diarization failure by association. Garble the transcription layer, and the speaker-turn boundaries built on top of it get shakier too.

The clearest data point here comes from a 2020 study published in PNAS (Koenecke et al.), which tested five major ASR systems and found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers. The gap held across all five systems, ruling out any one vendor's quirk as the explanation. The study went further and measured dialect density, a count of African American Vernacular English (AAVE) phonological and grammatical features per word, and found it tracked directly with error rate. Cities with lower average dialect density, like Rochester at 0.047, had lower error rates; cities with higher density, like Princeville at 0.19, had higher ones. That's a straight line from how much AAVE structure sits in the speech to how often the system gets it wrong.

Part of the mechanism sits in the language model underneath ASR. These models predict likely word sequences, and non-standard dialect phrasings consistently come out as less probable, so the model treats correct AAVE speech as a kind of statistical error before it's even processed the audio. Perplexity, the technical measure of how "surprised" a language model is by a sequence, runs consistently higher for AAVE constructions than for their Standard English equivalents.

The human response to that gap says as much as the number does. In one body of user research, a large share of participants reported changing how they talk specifically to be understood by voice technology. That quietly moves the whole burden of an acoustic training gap from the system onto the speaker. For diarization, the consequence is direct: if the ASR layer keeps stumbling over one speaker's words, that speaker's turn boundaries stay uncertain, and confusion errors pile up disproportionately on exactly the people the system already understands worst.

Dialect and multilingual complexity beyond English: German dialects, code-switching, and low-resource languages

English is just the best-documented case of a much wider pattern. Research comparing diarization systems across German dialects found measurable differences between systems even within one high-resource language. German has plenty of training data by global standards, yet dialect variation alone still moved the needle.

Multilingual settings push the problem further. The DISPLACE 2024 Challenge built dedicated tracks for multilingual speaker diarization, language diarization in multi-speaker audio, and ASR across multiple accents, and the fact that organizers felt the need for separate tracks says plainly that existing systems weren't handling these conditions by default. On that same dataset, the HOARD framework's DER climbed steeply, a steep jump from what the same kind of system posts on clean, controlled English benchmarks. Code-switching adds a genuinely different kind of difficulty on top of the acoustic one: the meaning of a phrase can hinge entirely on the language spoken around it, which is a semantic problem no amount of better audio processing solves on its own.

Low-resource languages compound all of it at once. Kurdish is a useful example: limited annotated training data, multiple distinct dialects, and frequent code-switching, stacking into a problem that high-resource-language systems have no default answer for. Regional variation doesn't need to cross a language boundary to matter, either. A study on Newcastle English found ASR errors correlated directly with regional phonological, lexical, and morphosyntactic features, with social factors playing a smaller role. Regional dialect bias stays less studied than racial or gender bias in this field, which is its own quiet gap sitting inside the research itself.

Step back and the pattern holds across wildly different linguistic contexts: German dialects, African-accented English, AAVE, Kurdish, regional British English. That consistency points to something structural, not an edge case confined to one exotic language. It shows up wherever training data runs thin relative to the acoustic diversity of real speakers.

Where failure carries the highest stakes: healthcare and other high-consequence domains

Clinical speech is where the hardest technical problem and the highest real-world stakes land in the exact same place, and that's not a coincidence worth glossing over. The AfriSpeech-Dialog finding of 33.38% average DER for clinical speech means roughly a third of spoken time in that setting gets misattributed or missed outright. In a medical record, that translates into diagnostic information, medication history, or symptom description attached to the wrong speaker, or dropped entirely.

Telehealth visits and in-person consultations increasingly lean on ASR and diarization to generate documentation, and errors there don't stay abstract; they land in the patient's actual chart. The population most likely to be affected, African-accented English speakers and AAVE speakers, overlaps substantially with populations that already face documented gaps in health equity. That's one underlying problem compounding itself, not two separate ones stacking by chance.

The same stakes show up outside medicine, just with different paperwork. Legal depositions, regulatory hearings, and HR investigations all rely on speaker-attributed transcripts where misattributing a statement carries factual weight that can shape an outcome, not merely a formatting slip. Call centers scale the exposure differently: fewer catastrophic single incidents, but millions of interactions running through the same accent-sensitive scoring systems, so a systematic bias in diarization quietly skews quality metrics against certain speaker populations across an entire customer base.

Here's the practical upshot for anyone deploying this technology: an aggregate DER number on a vendor's site tells you almost nothing about performance on your actual population and your actual domain. Domain- and accent-specific testing is the only way to know what you're actually buying, and skipping that step is how organizations end up finding the gap the hard way, after it's already in someone's chart or transcript.

What actually predicts diarization reliability in linguistically diverse audio

So what tells you, in advance, whether a system holds up? Training data composition sits at the top of the list, and it's not close. Systems trained on broader, more multilingual corpora degrade less across accents, and the AfriSpeech-Dialog research backs this up directly: models trained on wider multilingual data showed smaller gaps between clinical and general speech. That's the single biggest lever, and it's also the least glamorous one, resting on data curation rather than any clever new algorithm.

Domain match works independently of accent, and that's worth pausing on, because it means fixing accent bias alone won't fix everything. Clinical versus general speech produces a DER gap even within the same accent group, which is why domain-specific fine-tuning isn't optional for anyone deploying diarization in a hospital, courtroom, or specialized industry setting.

Dialect density, that same per-word count of non-standard phonological and grammatical features from the AAVE research, turns out to be a genuinely useful predictive tool. Higher density, higher expected error. That gives evaluators a way to spot trouble before deployment instead of discovering it afterward in a botched transcript.

Overlap ratio matters too. Check, before deploying anything, how much your target audio actually overlaps, because spontaneous, accent-diverse conversation runs higher overlap on average, and overlap is the one condition where every system, regardless of training data, performs at its worst. Pipeline architecture deserves scrutiny as well: systems that jointly model ASR and diarization let an accent-driven transcription error bleed straight into a speaker-attribution error, so testing the two components separately misses exactly the failure mode that matters most.

On the tooling side, SDBench, introduced at Interspeech 2025, brings together 13 different datasets with built-in infrastructure for consistent, fine-grained evaluation across on-device and server-side systems alike. That kind of shared benchmarking is the difference between an honest comparison and a cherry-picked one. Ask any vendor for DER broken out by accent group, domain, and overlap condition, rather than a single tidy aggregate number that hides exactly the variation this whole piece is about.

Mitigation approaches that have demonstrated measurable improvement

Domain-specific fine-tuning has the clearest evidence behind it, full stop. Fine-tuning Pyannote's segmentation model on in-domain African-accented English cut DER from 21.30% to 10.65%, a real, measured result. The catch is that fine-tuning needs curated in-domain data, which happens to be the scarcest resource for exactly the populations that need it most. Collecting that data takes time, funding, and access to the speaker communities in question, so this is a fix that works but that not everyone can afford to run.

Training speaker embeddings on large, multilingual corpora improves generalization to accents the model hasn't explicitly seen, though the benefit lands unevenly. Low-resource languages still lag, because there's only so much a broader corpus can do for a language that barely shows up in it.

LLM-based post-correction is newer and worth watching: ensembles of multiple language models, each trained on a different ASR system's output, then merged together, solve a specific practical headache, namely that a correction model trained on one vendor's mistakes often breaks the moment it's pointed at a different vendor's output. That matters a lot for any organization running a multi-vendor pipeline instead of betting everything on one system.

Participatory, community-grounded data collection, building dialect-specific datasets with input from the communities being recorded, has been tried for AAVE specifically, alongside syntactic tools built to handle its grammatical structure. These efforts tend to stay stuck inside siloed academic disciplines, which caps how much systemic change they produce even when the individual project itself works well.

What's left unresolved deserves to be said plainly instead of smoothed over. Fine-tuning narrows the medical-general gap for accented speech without closing it, and code-switching in low-resource settings doesn't have a solid answer yet. The deepest issue, whose speech got recorded and whose didn't, needs sustained investment in actually collecting that data, a slower and less exciting fix than a new algorithm, but one aimed at the actual root cause. Every mitigation on this list depends on having in-domain data, the ability to fine-tune, and evaluation infrastructure sharp enough to find the gaps before deployment rather than after. Skip any one of those three, and the gap just sits exactly where it was.