Speech-to-text is the process of automatically converting spoken audio into written text, also called automatic speech recognition. In artificial intelligence, and specifically in speech and language AI, it is the input side of voice systems: a model receives a waveform and returns a sequence of words, usually with timestamps, punctuation and confidence scores attached.
Modern systems replace the older chain of separate acoustic, pronunciation and language models with end-to-end neural networks trained on hundreds of thousands of hours of transcribed audio. The output feeds captioning, search, analytics, voice assistants and downstream NLP, so transcription errors propagate into every later step.
How speech-to-text turns sound into words
A speech-to-text system works by turning raw audio into acoustic features, mapping those features to linguistic units and then choosing the most probable word sequence. Audio is typically resampled to 16 kHz mono, which keeps the frequencies that matter for speech while reducing computation.
The waveform is cut into overlapping frames of about 25 milliseconds with a 10 millisecond step and converted into log-mel spectrograms, a compact picture of energy across frequency bands. An encoder network, usually a Transformer or Conformer, reads these frames and builds a representation that captures phonetic content and context.
A decoder then produces text. Depending on the architecture, it emits characters, subword tokens or whole words, and a beam search keeps several candidate hypotheses alive before selecting the best. External or internal language modeling biases the result toward plausible sentences, which is why a good system writes “recognize speech” rather than “wreck a nice beach”.
The table below summarizes the main stages of the pipeline and the decisions each stage involves.
| Stage | What happens | Typical choices |
|---|---|---|
| Audio preprocessing | Resampling, channel mixing, loudness normalization | 16 kHz mono, voice activity detection |
| Feature extraction | Frames converted to spectral features | 80 log-mel bins, 25 ms windows |
| Encoding | Neural network models context across frames | Conformer, Transformer, convolutional encoders |
| Decoding | Hypotheses scored and the best one selected | Greedy, beam search, language model fusion |
| Post-processing | Formatting of numbers, punctuation and casing | Inverse text normalization, diarization |
Model architectures and decoding approaches
Speech-to-text architectures differ mainly in how they align audio frames with output tokens, which determines latency, accuracy and suitability for streaming. Audio is much longer than the text it contains, so every design needs some alignment strategy.
| Architecture | Alignment approach | Strength | Limitation |
|---|---|---|---|
| Hybrid HMM-DNN | Hidden Markov model with neural acoustic scores | Mature, controllable lexicon | Complex multi-model pipeline |
| CTC | Independent per-frame predictions with blank symbol | Fast, simple, non-autoregressive | Weak internal language modeling |
| RNN-Transducer | Joint network combines audio and previous tokens | Natural fit for streaming | Harder to train and tune |
| Attention encoder-decoder | Decoder attends to the full encoded audio | High accuracy, strong context | Needs full segments, can hallucinate |
Streaming recognition processes audio in chunks and emits partial results within a few hundred milliseconds, which voice interfaces and live captions require. Batch recognition waits for the whole file and can use future context, so it is usually more accurate and is the preferred mode for media archives, meeting records and legal transcripts.
Large multilingual models trained on weakly supervised web audio can transcribe and identify dozens of languages with one network. They generalize well to new domains, but attention-based decoders may produce fluent text that was never spoken, especially during silence or music, so voice activity detection before decoding remains a sound practice.
Measuring speech-to-text accuracy
Speech-to-text accuracy is measured primarily by word error rate (WER), the number of substitutions, deletions and insertions divided by the number of words in a reference transcript. A WER of 5 percent means roughly one error in every twenty words. Because insertions count, WER can exceed 100 percent on very poor output.
WER depends heavily on normalization rules: whether “10” equals “ten”, whether punctuation and casing count, and how filler words are handled. Two vendors reporting different numbers on the same audio may simply normalize differently, so comparisons are only fair with identical test sets and scoring rules.
| Metric | What it measures | When it matters |
|---|---|---|
| Word error rate | Edit distance at word level | General accuracy benchmarking |
| Character error rate | Edit distance at character level | Languages without spaces, such as Chinese or Japanese |
| Real-time factor | Processing time divided by audio duration | Capacity planning for batch jobs |
| First-token latency | Delay before the first partial result | Voice agents and live captioning |
| Diarization error rate | Errors in assigning speech to speakers | Meetings, interviews, call centers |
Accuracy varies widely with conditions. Clean read speech from a close microphone may reach low single-digit WER, while overlapping speakers, far-field microphones, heavy accents, narrowband telephone audio at 8 kHz and domain-specific vocabulary can raise error rates several times over.
Speech-to-text compared with related speech and language tasks
Speech-to-text answers the question of which words were spoken, while neighboring tasks in speech and language AI generate audio, change language or interpret meaning. Keeping the boundaries clear helps when designing systems that chain several components.
| Task | Input | Output | Relationship to speech-to-text |
|---|---|---|---|
| Speech-to-text | Audio | Text in the spoken language | The task itself |
| Text-to-speech | Text | Synthetic audio | Inverse direction, generates speech |
| Speech translation | Audio | Text or audio in another language | Often cascades transcription and translation |
| NLP | Text | Labels, structure, answers | Consumes the transcript downstream |
| Voice cloning | Reference audio and text | Speech in a specific voice | Uses speech data but does not transcribe |
Speaker diarization and keyword spotting are often bundled with transcription but are distinct problems. Diarization answers who spoke when, and keyword spotting only detects a small set of phrases, such as a wake word, at far lower computational cost.
Deployment choices and common pitfalls
Choosing a speech-to-text setup means balancing accuracy, latency, cost and data protection. Cloud services offer strong general models and simple scaling, while on-premises or on-device models keep audio local, which matters for health, legal and other regulated recordings.
- Custom vocabulary and phrase boosting improve recognition of product names, drug names and jargon without retraining.
- Fine-tuning on domain audio, even a few dozen hours, can reduce errors substantially for a narrow use case.
- Audio quality beats model size: a better microphone placement often helps more than a larger model.
- Confidence scores should route uncertain segments to human review instead of being ignored.
- Personal data in recordings requires consent, retention limits and redaction of sensitive entities.
A frequent mistake is evaluating on clean demo clips and deploying on noisy real calls. Another is treating the transcript as ground truth for analytics; systematic errors on accents or dialects can bias every downstream decision, so error rates should be checked across speaker groups.
Put simply
Speech-to-text is software that listens and types. You give it a recording or a live voice, and it writes down what was said, often with the time of each sentence and the name of each speaker.
It works a bit like a very fast stenographer who has heard millions of hours of speech. It guesses each word from the sound and from what usually comes next in a sentence. It is very good with clear speech and makes more mistakes with noise, crosstalk and unusual names, so important transcripts still deserve a human check.
In practice
A regional hospital wants doctors to spend less time typing notes. Each consultation is recorded on a tablet, sent to an on-premises speech-to-text model, and turned into a draft note within about a minute after the visit ends.
In the first weeks, the system often misheard drug names, so the team added a custom vocabulary of about 2,000 medication terms and fine-tuned the model on anonymized dictations. Segments with low confidence are highlighted in yellow for the doctor to confirm.
After the adjustments, physicians review and sign a draft instead of writing from scratch. The recordings never leave the hospital network, which satisfies the data protection officer, and the error checks are repeated every quarter across different speaker groups.