The short version
- WER counts substitutions, deletions and insertions against a chosen reference transcript.
- The score changes with normalization, spelling conventions, segmentation, and the test material.
- Speaker attribution, timestamps, missing passages, names, and export quality require separate checks.
- The most useful benchmark is a fixed, representative set from your own work with a recorded review protocol.
1. What word error rate actually measures
Word error rate, usually shortened to WER, compares the system output with a human reference and counts substitutions, deletions, and insertions. The total is divided by the number of words in the reference. NIST evaluation plans use standardized scoring tools because even a simple-looking metric depends on consistent preparation and matching rules.1
WER is valuable for controlled experiments: same audio, same reference, same normalization. It becomes misleading when one vendor removes filler words, another preserves them, one reference spells a borrowed name in English, and another transliterates it. The number is not lying; the comparison may be answering a different question.
2. The test set can move the result more than the model
Read speech recorded close to a microphone is not a proxy for a customer call, a lecture hall, or four people around a laptop. Large multilingual models report different behavior across domains and languages, which is why responsible papers publish the datasets and conditions beside the score.2
Hebrew and Arabic make careless comparisons especially fragile. Spelling and tokenization choices can change the unit being counted. Dialect, code-switching, and the amount of spontaneous speech in the corpus change the difficulty. HebDB was created in part to expand natural, spontaneous Hebrew data; Arabic-English research similarly treats dialectal, morphologically rich, non-standardized speech as a distinct problem.34
3. What a single WER score does not tell you
WER does not tell you whether the right words were assigned to the right speaker. It does not tell you whether word timestamps drift, whether a minute disappeared after a network retry, or whether subtitle lines stay within a readable frame. It also treats a harmless article and a wrong medication dose as one word error each.
For practical work, track at least six dimensions: missing audio, words, speaker attribution, names and numbers, timing, and human correction time. Add export fidelity if the deliverable is a document or subtitle file. This turns ‘accuracy’ from a marketing adjective into a reviewable test.
4. A benchmark a small team can run honestly
Choose a frozen set of recordings that represents production: clean and noisy, short and long, single- and multi-speaker, and the language combinations your users actually speak. Keep the original files, the human references, the product settings, and the date of every run.
Blind the reviewer to the system name when possible. Count failures before polishing the transcript. Record both machine metrics and the minutes a person spends reaching a publishable result. Repeat after meaningful model or pipeline changes; do not silently replace the difficult files with easier ones.
5. What a responsible accuracy claim looks like
A defensible claim names the language, domain, dataset, audio conditions, metric, normalization rules, sample size, model or product version, and measurement date. It also states what was not measured. Without that context, ‘99% accurate’ is a slogan rather than reproducible evidence.
KolWrite’s recommended comparison is deliberately simple: upload the same difficult material, verify the important words and speakers against the audio, finish the actual export, and measure the work left. That test is less dramatic than a universal number and far more useful to a buyer.
Common questions
What does 95% transcription accuracy mean?
Unless the publisher defines the dataset and metric, it is ambiguous. It may mean five word errors per hundred reference words under one normalization scheme, or it may be an informal estimate. Ask for the test conditions.
Can two tools use the same audio and report different WER?
Yes, if they use different reference transcripts or normalization rules for punctuation, numbers, fillers, compounds or spelling. A fair comparison freezes those choices before scoring.
What should I measure besides WER?
Missing passages, speaker attribution, names and numbers, word or segment timing, subtitle layout, export fidelity, failure rate and the human time needed to finish the job.
Sources and further reading
Primary research and standards used for this article. Links open the original publication.
- 1 NIST — Open Speech Analytic Technologies Pilot Evaluation Plan
- 2 Radford et al. — Robust Speech Recognition via Large-Scale Weak Supervision
- 3 Turetzky et al. — HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing
- 4 Hamed et al. — Dialectal Arabic-English Code-Switching Speech Recognition
- 5 Chi & Bell — Improving Code-switched ASR with Linguistic Information