Custom Vocabulary and Domain Adaptation in Transcription APIs

Custom vocabulary and domain adaptation decide whether a transcription API survives contact with an actual pharmacy counter, courtroom, or call center floor. The top models on standard benchmarks cluster very closely on word error rate, and that near-tie can look like the field has plateaued, but the gap shows up the moment real vocabulary enters the picture. Most teams pick their adaptation technique backwards: they reach for a six-week fine-tuning project when a five-minute keyword list would have closed the gap. That backwards instinct, more than any technical limit in the models themselves, is the real subject here.

What out-of-vocabulary terms actually are and why they break ASR systems structurally

An ASR language model assigns probabilities to word sequences it has seen before. Feed it enough transcripts of people saying "thank you for calling," and it gets fast at predicting that "thank you for calling" is a likely thing to come next. Flip the input to something the model never trained on, and the pattern falls apart, because there's no probability left to assign. The model doesn't know what to do with a word it has zero prior exposure to, so it grabs the closest-sounding word it does know and prints that instead.

This is where out-of-vocabulary terms, OOV for short, do their damage. Two kinds show up in practice. The first is personal and situational: a caller's last name, an account number, an internal product SKU, unique to one deployment and gone the next day. The second is domain-systemic: drug names, legal citations, clinical shorthand, jargon that's rare in general speech but shows up constantly within one vertical. A model trained mostly on podcasts and audiobooks has functionally never heard "lisinopril" enough times to learn it, so it makes its best phonetic guess and prints "listen pro," close enough to satisfy a spellchecker and nowhere close to satisfying anyone standing at a pharmacy counter.

Domain-systemic terms don't fail one at a time, and that's the part that compounds. Clinical sentences cluster them: a single line of a physician's note might carry three or four rare terms back to back, so the error probability isn't additive, it's multiplicative. Drop a drug name, an anatomical term, and an abbreviation into the same breath and the odds of a clean transcript fall fast. Healthcare makes the point sharply: word error rates in loosely controlled, conversational clinical settings run very high in some studies, versus low single digits when the same system handles slow, controlled dictation. Same acoustic model, same vendor, wildly different outcome, because the second scenario is stuffed with exactly the vocabulary the first one never learned.

General-purpose ASR gets built to be as accurate as possible across the broadest slice of everyday speech, and that goal fights directly against learning rare terms well, since rare terms by definition don't move an aggregate accuracy score trained across millions of hours of common speech. Bigger training runs on more general audio won't fix this on their own; the model is doing exactly what it was built to do, and that job happens to be the wrong one for a cardiology unit.

The four adaptation techniques, from lightest touch to deepest investment

Four techniques close this gap, sitting on a spectrum from adding a parameter to an API call to training a new model outright. Each step up costs more time and data and buys back more accuracy. Here's the part worth saying plainly: most vocabulary problems are sparse problems, and sparse problems don't need six weeks of audio labeling to fix. Skipping straight to the top of the ladder when the bottom rung would work is the most common way teams waste a budget on this.

Keyword boosting, sometimes called phrase lists, sits at the light end. It's a runtime, stateless nudge: pass a list of terms with a request, and the decoder gets pushed toward those words by adjusting scores during inference. No training happens, and nothing persists between calls. Gains land somewhere in the 5 to 15 point range on domain vocabulary, modest-sounding until it's the difference between a call center agent seeing the right SKU number or the wrong one. The catch is the boost weight needs tuning: push too hard and accuracy on ordinary words starts to slip, push too soft and the rare terms never surface anyway, and loading in too many keywords makes the system guess in stranger directions still. This fits anywhere the vocabulary changes fast and per-customer, like a sales tool full of shifting prospect names.

Custom language model fine-tuning is the next rung, and it's the one most teams underrate. Feed the model domain text, no audio required, and it reshapes its sense of which word sequences are likely without touching how it hears sound at all. Four to eight weeks of assembling and training on domain text can cut word error rate 10 to 20% in specialized settings, with most of that gain showing up in the first 2,000 to 5,000 tokens of custom vocabulary; past that point, returns taper off. This works well wherever large volumes of domain text already sit around waiting to be collected: legal filings, insurance claims, financial disclosures.

Neural contextual biasing goes further still. The biasing mechanism trains jointly with the audio encoder, so context phrases get encoded as embeddings that interact directly with the acoustic signal instead of getting bolted on afterward. The bias list can shift mid-conversation without reconnecting or retraining anything, and that's the advantage that matters most here. One dynamic-vocabulary method improved bias-phrase accuracy by 3.1 to 4.9 points over older deep-biasing approaches, and a separate trie-based method, fine-tuning Whisper with synthetic training data, brought WER down from 30.86% to 12.19%. This is the layer built for real-time voice agents, where the vocabulary is too large or too fluid for a static list to keep up.

Full acoustic and language model fine-tuning sits at the top, touching how the system hears as well as what it expects to hear. It takes six to twelve weeks with labeled domain audio and addresses accent, background noise, and speaking style alongside vocabulary. Adapter-based tuning in some studies pushed keyword recall from 60% up to 96% without wrecking general performance elsewhere. This is the tier for healthcare, regulated finance, anywhere a wrong word carries legal or clinical weight and the deployment has a long enough runway to justify the cost. Teams reach for this tier far too often when their real problem was a hundred unrecognized drug names, and spending twelve weeks retraining an acoustic model to fix a vocabulary gap is a poor trade against the cost of the fix that was actually needed.

A fifth layer runs alongside all four: a second-pass LLM correction step that reads the raw transcript and fixes what it can catch after the fact. It's cheap to bolt on, and it helps on noisy audio, though its usefulness is limited on its own; it works best paired with adapting the primary system rather than standing in for that work.

How major transcription APIs implement these techniques in practice

Deepgram runs a two-tier system that tracks its own model generations. The older Nova-2 tier uses a Keywords feature: a numeric boost applied externally to logit scores, capped at 100 terms and 500 total tokens, affecting only out-of-vocabulary words. Nova-3 replaces this with a newer prompting-based approach designed to handle domain vocabulary more natively within the model. In streaming deployments, vocabulary hints can be updated mid-session, so a live call can shift vocabulary as the conversation moves without dropping the connection. Deepgram also offers custom model training that accepts domain audio; the entry tier starts at 1 to 4 hours of audio, with returns flattening out around 860 hours. Domain-specific model variants tuned for specialized vocabulary such as pharmaceutical names and clinical shorthand are available as dedicated offerings rather than customization layers bolted onto the general model.

Azure's Custom Speech layers two kinds of adaptation on top of its base model. A lightweight runtime hint mechanism can be passed with the request without any training. The deeper custom model layer accepts a supplementary text corpus to widen domain vocabulary and audio with matched transcriptions to handle specific acoustic environments. Additional customization options address nonstandard terms and output formatting needs.

Amazon Transcribe offers custom language models for domain vocabulary alongside speaker diarization, automatic language detection, and PII redaction. A HIPAA-eligible medical variant runs at $0.075 per minute. Pricing follows a volume curve, from $0.024 per minute at the base tier down to $0.0078 per minute at the highest volume tier, a discount up to 67.5% at scale.

AssemblyAI's Universal-2 model is built around usable output alongside raw WER: reported improvements of 21% in alphanumeric accuracy, things like phone numbers and customer IDs, and 15% in text formatting, both aimed squarely at contact center and CRM use. Its word and phrase list supports up to 1,000 entries on async jobs, a notably larger cap than Deepgram's legacy Keywords tier. AssemblyAI has continued expanding its model lineup with newer offerings, and pricing dropped 43% during 2024 to $0.37 per hour.

OpenAI's Whisper takes a different approach, with domain adaptation relying on self-hosting and fine-tuning rather than a native runtime boosting mechanism. Domain adaptation means self-hosting and fine-tuning, or leaning on prompt engineering at inference time. The large-v3-turbo release cut decoder layers down to 4, delivering a 5.4x speed gain at only modest accuracy cost. Whisper's architecture is oriented toward batch processing, which limits its fit for live use cases. It offers full data privacy for anyone self-hosting, or a low-cost managed API for anyone willing to trade customization for simplicity. It fits organizations with the engineering bandwidth to fine-tune their own checkpoints, or ones where keeping audio data in-house outweighs the convenience of a vendor's boosting API.

Choosing the right technique given domain, data availability, and latency constraints

Three questions decide which technique fits: how stable is the vocabulary, how much data exists, and how much latency the use case can tolerate? Get the order of these questions wrong and it's easy to talk yourself into full acoustic fine-tuning when a phrase list would have done the job by Friday. Reaching for the acoustic model first because it sounds like the more serious solution is the single most common misstep teams make here, and it rarely pays off. There are cases (heavy accents, noisy floors, a call center next to an airport) where it's the only fix, but those are the exception, not the default starting point.

Vocabulary stability is the first fork. Static domain terms, drug names, legal boilerplate, the stuff that doesn't change week to week, justify the heavier investment of a custom language model or full fine-tuning, because the payoff compounds over time. Dynamic vocabulary, a rotating contact list or a session-specific set of product names, points toward runtime boosting or contextual biasing instead, since there's no fixed target worth training against when the target moves daily.

Data availability decides things next, and in practice it's often the real bottleneck. Labeled audio, meaning audio paired with an accurate transcript, is expensive and slow to produce, while text corpora tend to be far more accessible: call logs, documentation, and existing transcripts are usually sitting around already. That asymmetry is exactly why custom language model tuning is so often the first real step up from boosting; the data cost is far lower than what full acoustic fine-tuning demands.

Latency rules options out entirely rather than just making them less attractive. Streaming, real-time use cases, voice agents, live captioning, can't absorb a full inference round-trip for post-processing, so runtime boosting and contextual biasing fit. Second-pass LLM correction adds a round-trip and belongs only in batch or near-real-time work with room to spare.

One question cuts through most of this faster than any framework: is the OOV problem sparse or dense? A handful of product names scattered through otherwise ordinary speech is sparse, and keyword boosting alone closes it, full stop, no further spend justified. A clinical note or legal filing packed with rare terms in nearly every sentence is dense, and boosting won't move the needle enough there; that's a CLM or fine-tuning problem. A common staging pattern in production reflects this logic directly: start with boosting to get a quick accuracy floor, add CLM fine-tuning once enough domain text is collected, and reach for full acoustic fine-tuning only once remaining errors trace back to acoustic conditions, accent, noise, speaking style, rather than vocabulary gaps.

The benchmarking gap that makes evaluating custom vocabulary accuracy unreliable

Here's the part nobody's WER chart tells you: standard benchmarks like LibriSpeech and Common Voice measure how well a model handles common vocabulary, which means they can't say much of anything about domain-specific or personalized terms.

No standardized benchmark for custom vocabulary accuracy existed for a long time, so vendor claims about domain performance got nowhere near the scrutiny that general WER claims received. That gap has drawn increasing attention as teams running domain-specific deployments find standard benchmarks offer little guidance on how well a given API will handle their actual vocabulary.t production scale. Once these approaches scale up from a small proof-of-concept to something closer to a real deployment, prompting and boosting land in a similar range of accuracy. The choice between them ends up mattering less than the rigor of the implementation and the scale at which it gets tested, an uncomfortable finding for any vendor whose whole pitch rests on the superiority of one particular boosting method.

A vendor's WER number on a standard benchmark says close to nothing about how that system handles a radiology department's dictation or a customer service floor's SKU numbers. The only signal worth trusting is a test run on actual domain audio with the actual vocabulary in question, and any team that skips this step is buying the product without ever seeing it work.

Building that internal benchmark isn't complicated, just underused. Find the specific OOV terms responsible for the errors that actually matter, usually a small set doing a disproportionate share of the damage, then build a held-out test set that mirrors production conditions: background noise, speaker variety, how dense the rare vocabulary really is. Track recall on those specific terms separately from the aggregate WER number, because a system can look excellent on average while reliably failing the ten words a business actually cares about. Adaptation only proves itself through measurement; skip the domain-specific evaluation loop, and there's no telling whether any of the four techniques above did anything at all.

What a production-grade domain adaptation workflow looks like end to end

Step one is an audit. Run the base model against a representative sample of real production audio and find the specific out-of-vocabulary terms causing the highest-impact errors, not just whichever sentences carry the worst overall WER. A misheard filler word and a misheard dosage carry very different consequences, even if they contribute equally to a WER score.

Step two matches the technique to the problem type the audit uncovered. Sparse, shifting vocabulary gets keyword boosting first: cheap, fast to test, and often enough on its own. Dense, stable domain vocabulary justifies the longer runway of a custom language model. If errors trace to acoustic conditions rather than vocabulary, accent, background noise, cross-talk, none of the vocabulary techniques above will touch it; that's a signal to look at acoustic fine-tuning specifically, rather than building another keyword list and hoping.

Both steps tie back to the same point running through every section here: the benchmark number on a vendor's homepage is a starting point for a conversation, one data point among several. Closing the gap between lab accuracy and what happens on an actual production floor takes an audit, a matched technique, and an evaluation loop built on the vocabulary that specific deployment actually needs to get right.