Speech recognition

How a recording becomes text you can work with

A transcription system does not hear words. It estimates them. Here is the chain that runs from a sound wave to a timed transcript, and the places where it tends to break.

KolWrite Research Desk 5 min read
Diagram of the transcription pipeline: a sound wave is cut into short windows, an encoder turns the windows into a representation of speech, a decoder scores candidate word sequences, and separate stages add speaker labels and timestamps to produce a timed transcript.
Five stages, and only the last one contains words. Everything before it is signal and probability.

The short version

  • Speech recognition predicts the most likely text for a stretch of audio. There is no hidden script in the recording to recover.
  • Training on very large multilingual audio made models steadier. Support for a language is not a promise of quality in it.
  • Speaker labels, timestamps, punctuation and export are separate components around the recognizer, and each one fails on its own.
  • The microphone and the room usually decide more than the choice of model.
  • We keep the audio, the timed text and the human correction in one place, because that is where the errors show up.

1. What the software receives is a signal, not a sentence

A recording reaches the software as a long list of numbers, each one the air pressure at a single instant. Before any word is proposed, the audio is cut into short windows, and each window is described by the way its energy spreads across frequencies. The physics does not change. Two voices, room echo, a laptop fan and file compression all arrive inside the same signal.

This is why the microphone matters more than most buyers expect. A microphone near the mouth keeps consonants and the small gaps between words. A phone in the middle of a meeting table collects the echo and everybody at once. No language model puts back a consonant the room swallowed.

2. The encoder builds a map of speech

The encoder turns short stretches of audio into vectors. They carry what helps prediction: phonetic detail, rhythm, and a partial sense of the difference between voice, noise and silence. Large pretraining changed this stage. A 2022 paper reported training on 680,000 hours of weakly supervised multilingual audio. Later work widened coverage past a hundred languages, and then past a thousand, by leaning on recordings nobody had transcribed.123

Scale buys exposure, not equality. A model that has met more microphones, accents and subjects is less surprised by yours. A language list does not tell you how the system handles a sales call in Hebrew, an interview in Palestinian Arabic, or a sentence that starts in one language and ends in another.

3. Decoding is a contest between plausible sentences

The recognizer does not pick one word at a time. It scores whole sequences against the audio and against what it learned about how sentences are built. That is why it prefers a coherent phrase over a run of similar sounds, and why it can hand back a fluent sentence that nobody said when the audio is muddy.

Names, numbers and moves between languages are where this shows. Research on code-switched recognition keeps reporting the same shortage: plenty of audio in each language alone, very little in which a speaker changes language inside a sentence. Hebrew and Arabic add unstable spelling and a thin supply of recorded spontaneous speech. Recent research datasets widen that supply without closing the gap.456

4. A finished transcript is a pipeline, not one answer

The recognizer produces words and times. Speaker separation answers a different question: who spoke when. Alignment ties every word back to the clock. Punctuation and paragraphing make the text readable. After that come storage, retries, editing, search and export. Each stage is a component in its own right, and each one can fail alone.

The stages also contradict each other. The words can be right while the speaker label is wrong. A summary can look tidy while a surname inside the transcript is mangled. We would rather show the timed source and let a reviewer settle it than print a confidence score on top.

5. What we check before trusting a workflow

Test with your own worst material. A video call where three people talk over each other, a lecture recorded from the back row, a voice message from a busy corridor, a surname that repeats forty times. Then measure the time it takes to reach a corrected file you would really send, not the time until text first appears.

KolWrite keeps playback, words, speakers and timestamps in one workspace, and carries that structure into documents and caption files. This does not remove the uncertainty from speech recognition. It makes the uncertain parts easy to find and quick to fix. That part of the work still belongs to a person.

Common questions

Is AI transcription the same thing as speech recognition?

Speech recognition is the step that turns audio into words. A transcription product wraps it in language detection, speaker separation, timing, punctuation, editing, search and export. Most daily frustration comes from the wrapper, not the core.

Why does a wrong transcript often read so well?

The decoder chooses a likely sequence of words from context. When the audio is unclear, a likely phrase beats what was actually said, and it still reads fluently. Check names, numbers and anything you plan to quote against the recording.

Does support for more languages mean equal quality in all of them?

No. Training hours, dialect coverage and evaluation quality differ sharply between languages. A language list is a statement of capability. It is not a claim about accuracy in your language.

Do the timestamps come from the same model as the words?

Not always. Timing, speaker labels and punctuation are often produced by separate stages. That is how a transcript ends up with the right words at the wrong moment. We check each of them separately.

Sources and further reading

Primary research and standards used for this article. Links open the original publication.

  1. 1 Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision
  2. 2 Pratap et al., Scaling Speech Technology to 1,000+ Languages
  3. 3 Zhang et al., Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
  4. 4 Chi & Bell, Improving Code-switched ASR with Linguistic Information
  5. 5 Turetzky et al., HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing
  6. 6 Hamed et al., Dialectal Arabic-English Code-Switching Speech Recognition
All research

Continue reading

A worked example of counting transcription errors: a human reference line sits above the machine output, with one substituted word, one deleted word and one inserted word marked, and the three counts divided by the number of reference words.

Evaluation

Accuracy is not one number

Word error rate is a real measurement, and it is easy to quote without its conditions. A transcript can score well and still fail the job you bought it for. Here is what the number counts, and what we measure instead.

5 min read
Read article
A caption timeline: each caption is drawn across the time it stays on screen, next to the time a viewer needs to read it, and one over-long caption is flagged beside a shortened version that fits its slot.

Accessibility

Captions are part of the content

A transcript, a translated subtitle track and a caption file can hold the same words and do three different jobs. Here is what each one is for, what the standard asks, and what we check before publishing.

5 min read
Read article