Evaluation

Accuracy is not one number

Word error rate is a real measurement, and it is easy to quote without its conditions. A transcript can score well and still fail the job you bought it for. Here is what the number counts, and what we measure instead.

KolWrite Research Desk 5 min read
A worked example of counting transcription errors: a human reference line sits above the machine output, with one substituted word, one deleted word and one inserted word marked, and the three counts divided by the number of reference words.
Worked example: where a word error rate comes from, and why the same sentence scores differently in different rooms.

The short version

  • Word error rate counts substitutions, deletions and insertions against a reference transcript that a person wrote.
  • Change the normalization rules, the spelling convention or the test files, and the same system earns a different score.
  • Speaker attribution, timestamps, passages that went missing, names and export quality each need their own check.
  • The benchmark that helps you is a frozen set of your own recordings, scored the same way every time.
  • We do not publish a single accuracy figure for KolWrite, because one figure would not survive contact with your audio.

1. What word error rate counts

Word error rate, usually shortened to WER, lines the machine output up against a reference a person wrote and counts three kinds of mistake: a wrong word, a missing word, and an added word. The total is divided by the number of words in the reference. NIST evaluation programs ship standardized scoring tools with their plans, because even a simple metric depends on how the two texts are prepared and matched.1

The metric earns its place in a controlled test. Same audio, same reference, same rules on both sides. It stops meaning much when one system strips filler words and another keeps them, or when one reference writes a borrowed name in Latin letters and the other transliterates it. The number is not lying. It answers a different question.

2. The test set moves the score more than the model does

Read speech recorded into a good microphone is no stand-in for a support call, a lecture hall, or four people around a laptop. The 2022 weakly supervised training work published results dataset by dataset and language by language rather than as one headline figure. That is the honest way to report.2

A careless comparison falls apart faster in Hebrew and Arabic. Spelling decides what counts as one word. Dialect, switching between languages, and the share of spontaneous speech change how hard a file is. A Hebrew dataset published in 2024 was built partly to add unscripted speech to a thin pool, and Arabic research treats dialectal, mixed-language speech as a problem of its own.345

3. What one score hides

WER does not say whether the right words landed on the right speaker. It says nothing about timestamps drifting, about a minute that vanished after a failed retry, or about a caption line too long to read before it leaves the screen. It charges the same price for a dropped article and a wrong dose of medication.

We track six things instead of one: missing audio, words, speaker attribution, names and numbers, timing, and the minutes a person spends fixing the result. When the deliverable is a document or a subtitle file, we add export fidelity. That turns accuracy from an adjective into a check anyone can repeat.

4. A benchmark a small team can run honestly

Freeze a set of recordings that looks like your real work: clean and noisy, short and long, one speaker and five, and the language mixes your people actually speak. Keep the original files, the human reference, the settings and the date of every run. A set quietly refreshed with easier files has stopped measuring anything.

Hide the product name from the reviewer where you can. Count the failures before anyone tidies the text. Write the machine score and the human minutes side by side. Run it again after a model or pipeline change, and keep the hard files in the set even when they are embarrassing.

5. What an honest accuracy claim contains

A claim you can act on names the language, the domain, the dataset, the recording conditions, the metric, the normalization rules, the number of files, the product version and the date. It also says what was not measured. Without that, a percentage is a slogan.

We have not published an accuracy figure for KolWrite, and we will not invent one for a page like this. The comparison we suggest is duller and more useful. Put the same hard recordings through each tool, check the names and speakers against the audio, finish the export you would really send, and count the work left over.

Common questions

What does 99% accurate mean?

On its own, close to nothing. It might mean one error per hundred reference words under one set of normalization rules, on files the seller chose. Ask which recordings, which metric, which rules and which date. Without those, treat it as advertising.

Can two tools score differently on the same audio?

Yes, easily. Different reference transcripts, different handling of punctuation, numbers, filler words and spelling all move the result. A fair comparison fixes those choices before anyone scores anything.

What should we measure besides word error rate?

Passages that went missing, speaker attribution, names and numbers, word or segment timing, caption layout, export fidelity, how often a job fails, and the human time needed to reach a finished file.

Is a public benchmark enough to choose a tool?

It is a starting point. Public sets rarely contain your rooms, your accents or your vocabulary. We would rather see ten of your own recordings scored consistently than a leaderboard built on somebody else's audio.

Sources and further reading

Primary research and standards used for this article. Links open the original publication.

  1. 1 NIST, Open Speech Analytic Technologies Pilot Evaluation Plan
  2. 2 Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision
  3. 3 Turetzky et al., HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing
  4. 4 Hamed et al., Dialectal Arabic-English Code-Switching Speech Recognition
  5. 5 Chi & Bell, Improving Code-switched ASR with Linguistic Information
All research

Continue reading

A caption timeline: each caption is drawn across the time it stays on screen, next to the time a viewer needs to read it, and one over-long caption is flagged beside a shortened version that fits its slot.

Accessibility

Captions are part of the content

A transcript, a translated subtitle track and a caption file can hold the same words and do three different jobs. Here is what each one is for, what the standard asks, and what we check before publishing.

5 min read
Read article