How does AI speech-to-text work?

From sound wave to text in four steps

A modern recognizer such as Whisper follows the same pipeline every time you give it a file:

1. Resample. The audio is converted to one channel at 16,000 samples per second. This is enough for speech, which carries most of its information below 8 kHz.
2. Make a spectrogram. The signal is cut into short overlapping windows (Whisper uses 25 ms windows every 10 ms) and each window becomes 80 numbers on the Mel scale, which spaces frequencies the way human hearing does. The result is an image: time on one axis, pitch bands on the other.
3. Encode. A Transformer encoder reads the spectrogram and builds a representation of what was said, including which sounds are speech and which are noise.
4. Decode. A Transformer decoder writes the transcript one token at a time, each time picking a likely next token given the audio and the text so far. Special tokens tell it the language, the task (transcribe or translate to English) and where to put timestamps.

Whisper processes audio in 30-second windows. A one-hour file is handled as a sliding sequence of windows, and the predicted timestamps tell the program where to start the next window. Timestamps are predicted in steps of 20 ms, which is why subtitle timings from Whisper are good but not frame-exact.

How the model learned: weak supervision at scale

Older systems combined separate parts: an acoustic model for sounds, a pronunciation dictionary and a language model for word order. Each part needed carefully labeled data in each language.

Whisper took a different route. Its authors collected 680,000 hours of audio paired with transcripts found on the internet. About 117,000 hours covered 96 languages other than English, and 125,000 hours were non-English speech paired with English text, which taught the model to translate. The labels were imperfect (that is the weak part of weak supervision), but the scale made the model robust to accents, background noise and recording quality without fine-tuning for each test set.

The same idea explains its weak points. Languages with little data in that collection are recognized less accurately, and the model learned habits of internet captions, such as adding phrases that often appear at the end of videos.

Measuring accuracy: word error rate

ASR accuracy is reported as word error rate (WER): the number of substituted, deleted and inserted words, divided by the number of words in the correct transcript. Lower is better, and it can exceed 100 percent if the model inserts a lot of text.

Reference:  the meeting moved to three thirty on friday   (8 words)
Model:      the meeting moved to tree  thirty    friday

Substitutions: three -> tree        1
Deletions:     on                   1
Insertions:                         0
WER = (1 + 1 + 0) / 8 = 25%

Before comparing, researchers normalize both texts: lower case, remove punctuation, expand contractions, and write numbers the same way. Without that step, "3:30" versus "three thirty" would count as errors even though both are correct. When a vendor quotes a WER, ask what audio it was measured on; clean read speech and a noisy phone call can differ by many times.

For Chinese, Japanese and Thai, which do not separate words with spaces, character error rate (CER) is used instead.

Types of speech recognizers

The Web Speech API's SpeechRecognition interface lets a web page ask the platform for live recognition from the microphone. By default it uses a service provided by the browser or operating system; some browsers can also run it on the device. A page that runs Whisper itself, for example cv.cm's speech-to-text tool, which runs Whisper in the browser and exports TXT, SRT or VTT, does not depend on that service, but needs to download the model once.

Where speech-to-text fails

• Names, brands and jargon. The decoder prefers common words, so a rare surname becomes a common one. Correct these by hand, or give a short prompt with the right spellings if your tool supports one.
• Hallucination in silence or music. The Whisper paper reports that long-form transcription can get stuck in repeat loops, skip the first or last words of a segment, or produce text unrelated to the audio. Silent intros, applause and background music are common triggers.
• Accents and code-switching. Speakers who mix two languages in one sentence, or strong regional accents with little training data, raise error rates.
• Overlapping speakers. Most recognizers write one stream of text and do not say who spoke. Speaker labels (diarization) are a separate step.
• Traditional vs Simplified Chinese. A model may output either script for Mandarin; convert the transcript afterwards if you need one script.
• Wrong language detected. Auto-detection looks at the start of the audio. If the first seconds are music or another language, choose the language manually.

Practical workflow for clean subtitles

1. Start with the best audio you have. Use the original file, not a re-recording from speakers. Reduce music under speech if you can.
2. Trim long silences and music at the start, which cause wrong language detection and invented lines.
3. Set the language manually when you know it.
4. Choose a larger model for final subtitles and a smaller one for quick notes. Larger models are slower but make fewer errors on names and accents.
5. Export SRT or VTT, then read the whole file once while watching the video. Fix names, numbers and any line that repeats.
6. Split long cues. A readable subtitle is about two lines and stays on screen for at least one second.

The two common subtitle formats differ only in small details:

SRT:
1
00:00:01,000 --> 00:00:04,000
Welcome to the lesson.

VTT:
WEBVTT

00:00:01.000 --> 00:00:04.000
Welcome to the lesson.

SRT numbers each cue and uses a comma before the milliseconds. WebVTT, the W3C format used by the HTML video element, starts with the WEBVTT header and uses a period.

Privacy and consent

Recordings of meetings, calls and interviews contain personal data, and in many countries recording a person needs their consent. Before using a cloud transcription service, check where the audio is stored and whether it is used for training. Running a model locally, in an app or in the browser, keeps the audio on your device, which is often the simplest way to meet a privacy requirement.