Handling Overlapping Speech in Multi-Speaker Meeting Transcripts

Overlapping speech shows up in around 13 to 15 percent of recorded meeting audio, with one corpus measuring 13.2 percent and another measuring 14.7 percent across separate studies. That means about one in every seven seconds of a typical meeting has two or more people talking at once. That single fact explains most of the gap between how a transcription tool performs in a vendor demo and how it performs in your actual weekly stand-up.

How accuracy collapses as acoustic conditions move from clean audio to real meeting rooms

Clean-audio benchmarks are where the transcription industry likes to live. Leading models now post word error rates (WER) around 3 percent on clean audio, with no cross-talk or background noise. Put that same model in a real meeting room, people talking over each other, a laptop fan running, and WER climbs past 12 percent. Push it further into far-field recordings, where the mic sits in the middle of a conference table instead of clipped to someone's collar, and error rates exceed 35 percent.

Diarization, the task of figuring out who said what, hasn't caught up either. State-of-the-art systems still carry error rates in the 11 to 13 percent range on hard meeting corpora, and overlap is consistently the biggest single cause of that number. A study of Zoom's Otter-powered live transcription in psychiatric interview settings found a median WER of 19.2 percent. An independent benchmark run by TestDevLab put Microsoft Teams at 11.54 percent WER against 7.40 percent for Zoom under the same test conditions.

The industry treats 88 percent accuracy as the floor for a transcript to be readable, and 92 percent as the floor for something you'd trust to search later and cite as a record. Given the gap between clean-benchmark numbers and real-meeting numbers, a lot of production transcripts never clear that second bar, even when the marketing page says otherwise.

Raw WER is the wrong yardstick for a meeting transcript, and vendors lean on it anyway because it flatters them. What matters is cpWER, concatenated minimum-permutation word error rate, which scores transcription accuracy and speaker assignment together. A transcript can get every word right and still be useless if it puts those words in the wrong mouth. The rest of this piece traces where, specifically, that failure happens.

Where the pipeline breaks: the three compounding failure points in overlap handling

A meeting transcript doesn't come out of one model. It comes out of a chain: audio capture, then speech separation, then voice activity detection, then speaker diarization, then automatic speech recognition (ASR), then post-processing cleanup. Overlap breaks each stage differently, and the breaks stack on top of each other rather than canceling out.

The first failure point is overlap detection itself. Before diarization or ASR can run, something has to notice that two people are talking at the same time. Miss that, and the overlapping stretch gets quietly treated as single-speaker audio. Nothing downstream flags it as wrong, because as far as the pipeline is concerned, nothing went wrong.

The second failure point sits inside diarization, and it's the one most commercial systems still haven't fixed. Most of them cluster: they assign each frame of audio to exactly one speaker. When two people talk over each other, the clustering step has to pick a winner, usually whichever voice is louder or clearer in that frame. That's a speaker confusion error, and because every later use of the transcript, from search to action-item extraction, inherits that speaker label, the mistake doesn't stay contained. It spreads.

The third failure point is ASR decoding. Even when diarization gets the speaker boundaries roughly right, the ASR model still receives mixed audio during the overlap window. Without a separation step to pull the two voices apart, it either drops the quieter voice entirely or blends the two into a sentence that sounds plausible and is simply wrong. That kind of error is the hardest to catch on a quick read, because it doesn't read like an error. It reads fine.

Because the pipeline is cascaded, a diarization mistake locks in before ASR even runs. There's no mechanism for the ASR stage to go back and fix a speaker label, even when it transcribes the words themselves correctly. That's the structural reason patching one stage rarely fixes the transcript: the damage from stage two is already baked in by the time stage three starts. State-of-the-art systems still land around 11 to 13 percent DER (diarization error rate, the summed time of speaker confusion, false alarms, and missed detection, divided by total recording duration) on hard meeting corpora, with overlap doing most of the damage. That number is what's pushed engineering teams away from bolting overlap detection onto existing pipelines and toward architectures built to handle all three failure points at once.

The limits of traditional clustering-based diarization when speakers talk at the same time

The standard diarization pipeline runs speaker embedding extraction (something like x-vectors or ECAPA-TDNN), followed by a clustering step (something like VBx or spectral clustering), and ends with one speaker label assigned per audio frame.

That one-speaker-per-frame rule isn't a limitation you patch around later. It's built into how clustering works: every frame gets exactly one label, full stop. Handling overlap means either bolting on a separate detection module before clustering runs, or replacing the architecture entirely, and most vendors have picked the cheaper option first. That's the wrong bet if overlap is more than a rounding error in your meetings, and for most real workplaces, it is.

The common bolt-on is an overlap detection module (OSD), inserted before clustering to flag regions of simultaneous speech and route them differently, sometimes silencing them, sometimes sending them to a secondary process. Trouble is, OSD makes its own mistakes, especially in noisy or reverberant rooms. You haven't removed a failure point. You've added one.

The AliMeeting and M2MeT challenge results from ICASSP 2022 show what the ceiling looks like even with heavy engineering behind it. Bytedance's VolcSpeech system stacked front-end dereverberation, direction-of-arrival estimation, multi-channel combination, overlap detection, and DOVER-Lap system fusion on top of a clustering backbone. It reached 5.79 percent DER on the eval set and 7.23 percent on the test set, one of the strongest results of that generation, and overlap remained a persistent source of error. Five separate engineering interventions, stacked on top of each other, to get there.

Each added module fixes one thing and opens its own error surface. Clustering can be pushed a long way with enough augmentation bolted on, but its ceiling for overlap sits meaningfully below what architectures built without a forced one-speaker-per-frame rule can reach. That's the actual argument for abandoning clustering as the backbone, not some preference for newer architecture for its own sake.

What end-to-end neural diarization does differently and where it still falls short

End-to-end neural diarization (EEND) throws out the clustering step entirely. A single neural network maps raw audio directly to per-frame speaker activity probabilities, and critically, it can output nonzero probabilities for two speakers in the same frame. There's no forced winner, because there's no clustering step demanding one.

That's the core advantage for overlap. The network trains on data that already contains overlapping speech, so it learns to represent two active speakers in one frame instead of being architecturally forced to collapse them into one.

A 2026 system out of UIUC and Johns Hopkins, TagSpeech (Huo, Shao, Zhang), pushes this further with a unified framework for joint multi-speaker ASR and diarization. It uses Temporal Anchor Grounding to decouple the semantic content stream from the speaker identity stream, fine-tunes both through Serialized Output Training, and adds an interleaved time anchor mechanism for fine-grained timestamp prediction. Benchmarked on AMI and AliMeeting, it showed consistent DER improvements over strong end-to-end baselines, including Qwen-Omni and Gemini, specifically on the hardest overlap segments.

Serialized Output Training is worth understanding on its own terms, because it's a genuinely different way of handling the problem, not just a tweak. Instead of trying to untangle two overlapping voices simultaneously, the model learns to handle speaker turn-taking in a structured sequence, producing output for multiple speakers from the same mixed audio signal. The result is a structured output that pulls text for multiple speakers out of the same mixed audio signal.

None of this makes EEND a universal replacement, and treating it as one is the mistake to avoid. These systems still struggle when the number of speakers isn't known ahead of time, when recordings run very long, and, somewhat counterintuitively, in low-overlap scenarios where old-fashioned clustering is actually the more stable choice. Neither architecture wins across every condition, and that gap is exactly what's pushed the field toward systems that switch between them rather than commit to one.

How the 2025 state-of-the-art handles overlap by switching between methods based on the recording itself

The winning system at the MISP 2025 Challenge, built by Huang and colleagues at Fosafer and the Institute of Forensic Science under the Ministry of Public Security, didn't pick a side between clustering and end-to-end, and that refusal to pick a side is the whole point. It combined both: a WavLM-based end-to-end segmenter alongside a traditional multi-module VBx clustering pipeline, with a routing mechanism deciding which one handles which part of the recording.

The routing logic is simple, almost embarrassingly so given how long the field spent arguing clustering versus end-to-end as an either-or choice. The system measures the proportion of overlapping speech in each segment of a meeting. Below 1 percent overlap, it routes to the traditional VBx clustering method, where clustering is genuinely more stable. At or above that threshold, it hands the segment to the WavLM end-to-end segmenter, where overlap handling is stronger.

The results back this up: 9.48 percent CER on Track 2 (ASR) and 11.56 percent cpCER on Track 3 (diarization plus ASR combined), first place in both. The adaptive design means the system's overall accuracy never gets dragged down by whichever method happens to be weaker under a given set of conditions. It uses the tool suited to the actual acoustic profile of that specific meeting, segment by segment, rather than committing to one architecture for the whole recording.

The same system introduced ASR-Aware Observation Addition, built to compensate for how Guided Source Separation (GSS) degrades under low signal-to-noise conditions. It fuses three signals, the raw noisy speech, Mossformer2-separated speech, and GSS-separated speech, through a weighted fusion mechanism guided by a sentence-level bridging module. The underlying shift matters more than the specific technique: speech separation and ASR get co-optimized together here rather than treated as two independent steps bolted end to end.

Video-based diarization, using lip movement to help identify who's speaking, faces practical challenges from blurred images and variable lighting that limit its reliability under real-world conditions. Blurred lip images and bad lighting undercut the approach often enough that the the MISP 2025 winning system's documented improvements were driven by audio-side architectural choices.

What commercial tools have improved on overlap in 2025–2026 and what the benchmarks actually show

Overlap-aware models built specifically to handle crosstalk now sustain around 87.2 percent accuracy on overlapping segments, up from roughly 70 percent in 2024. That's real progress. It's also still below the 88 to 92 percent range the industry treats as the floor for readable and searchable transcripts, so "improved" and "solved" remain two different claims, and vendors tend to blur that line on purpose.

Fireflies.ai posted 94.2 percent overall word-level accuracy in a March 2026 Cotera Independent Benchmark. More relevant here: during crosstalk, the system held onto correct speaker attribution in cases where older models simply merged the overlapping voices into one hallucinated speaker.

AssemblyAI's Universal-3-Pro, detailed in a February 2026 API technical release, reported a 10.1 percent improvement in DER and a 13.2 percent improvement in cpWER over its predecessor. Its neural diarization can isolate micro-interruptions as short as 250 milliseconds.

None of these benchmarks show what happens outside the test set, and that's the gap buyers should be most skeptical of. Most public benchmarks use controlled corpora with known speaker counts and reasonably clean acoustics. Real enterprise meetings don't cooperate: room acoustics vary, someone joins late from a car with wind noise, a colleague starts a screen share with audio bleeding through their laptop speakers. Those conditions push error rates back toward the harder end of the range no matter what the headline benchmark number claims.

This is exactly why cpWER matters more than raw WER for anyone actually buying one of these tools. A system that transcribes every word correctly but assigns them to the wrong person hands you an action item addressed to the wrong colleague, or a search result that surfaces the right quote under the wrong name. Anyone evaluating vendors should ask for performance specifically on overlapping-speech corpora such as AliMeeting, instead of accepting a headline WER number almost certainly measured on clean, non-overlapping audio.

Practical techniques that improve overlap handling without replacing the underlying tool

A fair amount of overlap-handling improvement doesn't touch the model at all. It starts with the microphone, and that's the cheapest lever available by a wide margin, cheaper than any software upgrade a vendor will try to sell you.

Near-field mics, headsets or directional desk mics placed close to each speaker, cut down dramatically on the reverberation and signal attenuation that make overlap errors worse in the first place. Far-field array recordings, by contrast, are the setups pushing WER past 35 percent in hard conditions. Multi-channel array recording also opens the door to direction-of-arrival estimation and beamforming: the multi-channel combination step in Bytedance's VolcSpeech system at M2MeT 2022 produced a measurable DER improvement just from combining results across eight channels. For hybrid meetings specifically, getting remote participants onto headsets instead of laptop mics is probably the single highest-leverage change available, and it costs nothing to put in place.

Speaker enrollment helps too. Linking diarization to known voice signatures, tied to SSO profiles for recurring meeting attendees, gives the system something more stable to work from than pure clustering, especially in meetings where the same group of people shows up week after week.

Custom vocabulary matters more in overlap regions than people expect. Uploading domain-specific terms, product names, acronyms, technical jargon, cuts down ASR errors precisely where the model is already least confident. A model that's never seen a word is more likely to mis-decode it, and overlap regions are where that uncertainty compounds fastest.

None of this removes the need for a review step. Even well-tuned systems still produce overlap errors, so a lightweight post-meeting workflow, flagging segments where diarization confidence is low or speaker labels flip rapidly, catches the mistakes that matter before an action item goes out under the wrong name.

The most reliable fix for overlap isn't a smarter recovery algorithm. It's less overlap in the first place. Explicit turn-taking norms, raise-hand protocols on video calls, a facilitator naming who speaks next, cut down the amount of overlapping speech the system ever has to untangle, and no downstream model improvement beats simply generating less of the problem to begin with.

How AI visibility in transcript-derived content connects to the same accuracy requirements

Meeting transcripts increasingly become source material for published content: blog posts, thought-leadership pieces, Q&A articles drafted straight from a recorded conversation. A misattributed quote or a hallucinated crosstalk segment corrupts that source material before a writer even opens the document.

This matters more now that AI citation systems, the mechanisms behind generative engine optimization and answer engine optimization, reward content built on verifiable, correctly attributed statements. A transcript that merges two speakers' expertise into one voice, or attributes a specific claim to the wrong person, undermines exactly the authority signal that content needs to earn an AI citation.

For agencies handling multiple client accounts, this isn't an abstract risk. A speaker confusion error in a recorded client interview, one that folds a client's distinctive position into a generic "Speaker 2" label or merges it with the moderator's voice, can mean that position never makes it into the published piece at all. The client's visibility in AI-generated answers suffers for a reason nobody on the account team would ever think to check.

The 92 percent accuracy threshold for searchable archives isn't just an IT benchmark sitting in a procurement document somewhere. It's the floor below which transcript-derived content carries real factual risk once it's published. Any agency using recorded client conversations as raw material for content should be auditing a transcription vendor's cpWER performance specifically, not the headline WER number on the marketing page, and running the post-meeting review workflow described above before a single sentence of that transcript goes into a draft.