Whisper is an open-weight automatic speech recognition model released by OpenAI in September 2022, which converts spoken audio into text, identifies the spoken language and translates speech from other languages into English. In artificial intelligence, it is widely used as a general-purpose transcription engine because one model handles many languages, accents and recording conditions without task-specific fine-tuning.
Often called OpenAI Whisper, the model is an encoder-decoder transformer trained on a very large collection of weakly labeled audio paired with transcripts. Its weights and code are published under the permissive MIT license, which made it a common building block in subtitling tools, voice assistants, meeting recorders and research pipelines.
How Whisper turns audio into text
Whisper processes speech by converting audio into a log-Mel spectrogram and passing it through a transformer encoder, while a decoder generates the transcript token by token. Input audio is resampled to 16 kHz and split into 30-second windows; shorter clips are padded and longer recordings are processed window by window.
The decoder is conditioned on special task tokens that tell it what to do: detect the language, transcribe, translate into English, or predict timestamps. This multitask token format lets a single network replace a pipeline of separate components, such as voice activity detection, language identification and alignment, at least in simpler deployments.
| Component | Role | Key detail |
|---|---|---|
| Audio front end | Converts waveform to features | 80 Mel bins, 128 in large-v3 |
| Encoder | Builds acoustic representations | Transformer with convolutional stem |
| Decoder | Generates text tokens | Autoregressive, byte-level BPE vocabulary |
| Task tokens | Select language and task | Transcribe, translate, timestamps |
Training relied on weak supervision at large scale. The original release used about 680,000 hours of audio collected from the web with existing transcripts, roughly a sixth of it multilingual. Instead of carefully curated datasets, sheer volume and diversity gave the model its robustness to accents and background noise.
Whisper model sizes and versions
Whisper is a family of checkpoints that trade accuracy against speed and memory. The smaller models also exist in English-only variants with the suffix .en, which perform slightly better on English at the same size.
| Model | Parameters | Typical use |
|---|---|---|
| tiny | About 39 million | Embedded devices, quick drafts |
| base | About 74 million | Lightweight local transcription |
| small | About 244 million | Balanced desktop use |
| medium | About 769 million | Higher accuracy on a consumer GPU |
| large-v2 and large-v3 | About 1.55 billion | Best accuracy, server workloads |
| large-v3-turbo | About 809 million | Near-large accuracy at higher speed |
large-v2 followed in December 2022 with longer training, and large-v3 arrived in November 2023 with more training data and 128 Mel bins. The turbo variant, released in 2024, keeps the large encoder but cuts the decoder from 32 layers to 4, which makes decoding several times faster while losing little accuracy for most languages.
Whisper compared with other speech models
Whisper differs from earlier open speech models mainly in its training approach and out-of-the-box usability. The table compares it with wav2vec 2.0, a self-supervised model listed among neighboring AI models, and with commercial cloud transcription services.
| Property | Whisper | wav2vec 2.0 | Cloud transcription API |
|---|---|---|---|
| Training signal | Weakly supervised transcripts | Self-supervised audio, then fine-tuning | Proprietary |
| Usable without fine-tuning | Yes | Usually no | Yes |
| Output style | Punctuated, cased text | Often raw characters | Punctuated text |
| Deployment | Local or self-hosted | Local or self-hosted | Vendor servers |
| Real-time streaming | Not native | Possible with care | Usually native |
The practical advantage is that Whisper produces readable, punctuated transcripts immediately. Its main structural disadvantage is that it was designed for offline, window-based decoding, so live captioning requires extra engineering such as chunking with overlap and partial hypotheses.
Running Whisper in practice
Teams usually run Whisper through optimized community implementations rather than the reference code. whisper.cpp targets CPUs and Apple hardware, faster-whisper uses the CTranslate2 inference engine with quantization, and Distil-Whisper offers distilled English models that run much faster on modest hardware.
- Choose the smallest model that meets the accuracy target, measured as word error rate on your own audio, not on benchmark sets.
- Set the language explicitly when it is known, because automatic detection on short or noisy clips can choose the wrong language.
- Add voice activity detection to skip silence, which saves compute and reduces invented text.
- Use word-level alignment tools when subtitles need precise timing, since native timestamps are segment-level and approximate.
Accuracy varies widely by language. High-resource languages such as English and Spanish typically reach low word error rates on clean audio, while low-resource languages can produce error rates high enough to require human correction.
Limitations and known failure modes
The best-known weakness of Whisper is hallucination: during silence, music or heavy noise, the decoder can generate fluent text that nobody said. Because the output looks plausible, these errors are harder to spot than garbled words.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Invented sentences in quiet passages | Decoder fills silence | Voice activity detection, silence trimming |
| Repeated phrase loops | Conditioning on previous text | Disable previous-text conditioning, adjust temperature |
| Wrong language transcript | Misdetected language | Set the language parameter |
| Missing speaker labels | No built-in diarization | Add a separate diarization model |
| Drifting timestamps | Segment-level timing | Forced alignment after transcription |
These limitations matter most in high-stakes settings such as medical, legal or accessibility transcripts, where an invented sentence can change meaning. In such contexts, human review of the output remains necessary, and the original audio should be kept for verification.
In plain words
Whisper is a computer program that listens to recordings and writes down what people say. It can also tell which language is being spoken and translate it into English text.
It learned by studying an enormous amount of recorded speech together with matching written text, a bit like a person who has listened to thousands of radio shows in many languages. Because it is free to download, developers can run it on their own computers instead of sending recordings to someone else. It is very good but not perfect: in quiet or noisy parts it sometimes writes words that were never spoken, so important transcripts still need a human check.
A real-world example
A regional public broadcaster needs subtitles for a back catalog of about 2,000 hours of interviews and documentaries. Sending the archive to an external service is ruled out because of rights and data protection concerns, so the technical team runs Whisper large-v3-turbo with faster-whisper on two in-house GPU servers.
Before transcription, a voice activity detection step removes long silences and music beds, and the language is set explicitly for each program. The system processes the archive in a few weeks of unattended runs. Editors then review the subtitles at roughly a fraction of the time needed to type them from scratch, correcting mainly proper names and a small number of invented lines in noisy field recordings. The broadcaster ends up with searchable, subtitled archives produced entirely on its own infrastructure.