Whisper

Also known as: OpenAI Whisper

On this page 7

Whisper is an open-weight automatic speech recognition model released by OpenAI in September 2022, which converts spoken audio into text, identifies the spoken language and translates speech from other languages into English. In artificial intelligence, it is widely used as a general-purpose transcription engine because one model handles many languages, accents and recording conditions without task-specific fine-tuning.

Often called OpenAI Whisper, the model is an encoder-decoder transformer trained on a very large collection of weakly labeled audio paired with transcripts. Its weights and code are published under the permissive MIT license, which made it a common building block in subtitling tools, voice assistants, meeting recorders and research pipelines.

How Whisper turns audio into text

Whisper processes speech by converting audio into a log-Mel spectrogram and passing it through a transformer encoder, while a decoder generates the transcript token by token. Input audio is resampled to 16 kHz and split into 30-second windows; shorter clips are padded and longer recordings are processed window by window.

The decoder is conditioned on special task tokens that tell it what to do: detect the language, transcribe, translate into English, or predict timestamps. This multitask token format lets a single network replace a pipeline of separate components, such as voice activity detection, language identification and alignment, at least in simpler deployments.

ComponentRoleKey detail
Audio front endConverts waveform to features80 Mel bins, 128 in large-v3
EncoderBuilds acoustic representationsTransformer with convolutional stem
DecoderGenerates text tokensAutoregressive, byte-level BPE vocabulary
Task tokensSelect language and taskTranscribe, translate, timestamps
Main components of the Whisper architecture

Training relied on weak supervision at large scale. The original release used about 680,000 hours of audio collected from the web with existing transcripts, roughly a sixth of it multilingual. Instead of carefully curated datasets, sheer volume and diversity gave the model its robustness to accents and background noise.

Whisper model sizes and versions

Whisper is a family of checkpoints that trade accuracy against speed and memory. The smaller models also exist in English-only variants with the suffix .en, which perform slightly better on English at the same size.

ModelParametersTypical use
tinyAbout 39 millionEmbedded devices, quick drafts
baseAbout 74 millionLightweight local transcription
smallAbout 244 millionBalanced desktop use
mediumAbout 769 millionHigher accuracy on a consumer GPU
large-v2 and large-v3About 1.55 billionBest accuracy, server workloads
large-v3-turboAbout 809 millionNear-large accuracy at higher speed
Whisper checkpoints by size

large-v2 followed in December 2022 with longer training, and large-v3 arrived in November 2023 with more training data and 128 Mel bins. The turbo variant, released in 2024, keeps the large encoder but cuts the decoder from 32 layers to 4, which makes decoding several times faster while losing little accuracy for most languages.

Whisper compared with other speech models

Whisper differs from earlier open speech models mainly in its training approach and out-of-the-box usability. The table compares it with wav2vec 2.0, a self-supervised model listed among neighboring AI models, and with commercial cloud transcription services.

PropertyWhisperwav2vec 2.0Cloud transcription API
Training signalWeakly supervised transcriptsSelf-supervised audio, then fine-tuningProprietary
Usable without fine-tuningYesUsually noYes
Output stylePunctuated, cased textOften raw charactersPunctuated text
DeploymentLocal or self-hostedLocal or self-hostedVendor servers
Real-time streamingNot nativePossible with careUsually native
Whisper and alternative approaches to speech recognition

The practical advantage is that Whisper produces readable, punctuated transcripts immediately. Its main structural disadvantage is that it was designed for offline, window-based decoding, so live captioning requires extra engineering such as chunking with overlap and partial hypotheses.

Running Whisper in practice

Teams usually run Whisper through optimized community implementations rather than the reference code. whisper.cpp targets CPUs and Apple hardware, faster-whisper uses the CTranslate2 inference engine with quantization, and Distil-Whisper offers distilled English models that run much faster on modest hardware.

  • Choose the smallest model that meets the accuracy target, measured as word error rate on your own audio, not on benchmark sets.
  • Set the language explicitly when it is known, because automatic detection on short or noisy clips can choose the wrong language.
  • Add voice activity detection to skip silence, which saves compute and reduces invented text.
  • Use word-level alignment tools when subtitles need precise timing, since native timestamps are segment-level and approximate.

Accuracy varies widely by language. High-resource languages such as English and Spanish typically reach low word error rates on clean audio, while low-resource languages can produce error rates high enough to require human correction.

Limitations and known failure modes

The best-known weakness of Whisper is hallucination: during silence, music or heavy noise, the decoder can generate fluent text that nobody said. Because the output looks plausible, these errors are harder to spot than garbled words.

SymptomLikely causeRemedy
Invented sentences in quiet passagesDecoder fills silenceVoice activity detection, silence trimming
Repeated phrase loopsConditioning on previous textDisable previous-text conditioning, adjust temperature
Wrong language transcriptMisdetected languageSet the language parameter
Missing speaker labelsNo built-in diarizationAdd a separate diarization model
Drifting timestampsSegment-level timingForced alignment after transcription
Typical Whisper problems and remedies

These limitations matter most in high-stakes settings such as medical, legal or accessibility transcripts, where an invented sentence can change meaning. In such contexts, human review of the output remains necessary, and the original audio should be kept for verification.

In plain words

Whisper is a computer program that listens to recordings and writes down what people say. It can also tell which language is being spoken and translate it into English text.

It learned by studying an enormous amount of recorded speech together with matching written text, a bit like a person who has listened to thousands of radio shows in many languages. Because it is free to download, developers can run it on their own computers instead of sending recordings to someone else. It is very good but not perfect: in quiet or noisy parts it sometimes writes words that were never spoken, so important transcripts still need a human check.

A real-world example

A regional public broadcaster needs subtitles for a back catalog of about 2,000 hours of interviews and documentaries. Sending the archive to an external service is ruled out because of rights and data protection concerns, so the technical team runs Whisper large-v3-turbo with faster-whisper on two in-house GPU servers.

Before transcription, a voice activity detection step removes long silences and music beds, and the language is set explicitly for each program. The system processes the archive in a few weeks of unattended runs. Editors then review the subtitles at roughly a fraction of the time needed to type them from scratch, correcting mainly proper names and a small number of invented lines in noisy field recordings. The broadcaster ends up with searchable, subtitled archives produced entirely on its own infrastructure.