Voice activity detection (VAD) is the process of deciding, for each short slice of an audio signal, whether it contains human speech or non-speech such as silence, music or background noise. In Speech & language AI, VAD is the gatekeeper of the audio pipeline: it tells speech-to-text engines, telephony codecs and voice assistants when to listen, when to transmit and when a speaker has stopped.
A detector rarely sees the meaning of words. It works on acoustic evidence in frames of 10 to 30 milliseconds and outputs a probability or a binary label, which is then smoothed into speech segments with start and end timestamps. The quality of those boundaries directly shapes recognition accuracy, latency and compute cost downstream.
How voice activity detection classifies audio frames
A VAD system splits the incoming signal into overlapping frames, extracts features from each frame, scores the likelihood of speech and then applies temporal smoothing to turn noisy frame decisions into stable segments. Each stage can be classical signal processing or a trained neural network.
The following table summarizes the shared processing stages that almost every detector uses, regardless of whether it is a decades-old codec component or a modern neural model.
| Stage | What happens | Typical setting |
|---|---|---|
| Framing | Audio is cut into short windows | 10-30 ms frames, 50% overlap |
| Feature extraction | Energy, spectrum or learned embeddings are computed | Log-mel bands, zero-crossing rate |
| Frame scoring | A model estimates speech probability | Value between 0 and 1 |
| Thresholding | Probability becomes a speech or non-speech label | Threshold near 0.5, tuned per domain |
| Hangover and smoothing | Short gaps are bridged and short bursts removed | 100-300 ms hangover |
The hangover period is the least visible yet most consequential setting. Speech contains natural micro-pauses between syllables and words, and without a hangover the detector would chop a sentence into fragments. A hangover that is too long, however, keeps the channel open after the speaker stops and adds latency to every turn.
Main families of VAD algorithms
VAD algorithms fall into three broad families: energy-based rules, statistical models and neural classifiers. They differ in robustness to noise, computational cost and the amount of tuning they require, and each family remains in production somewhere today.
| Family | Core idea | Strength | Weakness |
|---|---|---|---|
| Energy threshold | Speech is louder than the noise floor | Almost zero compute | Fails with loud noise or music |
| Statistical model | Gaussian or likelihood-ratio tests on spectral features | Runs on embedded chips | Struggles with nonstationary noise |
| Neural VAD | A small network learns speech patterns from labeled audio | High accuracy in noise | Needs training data and more compute |
| ASR-integrated detection | The recognizer itself predicts silence or end of speech | Shares context with transcription | Tied to one recognition model |
Classical detectors are standardized in telecommunications, where ITU-T G.729 Annex B and the VAD options of the AMR codec allow phones to stop transmitting during silence. The widely used WebRTC VAD follows the statistical approach with Gaussian mixture models. Modern neural detectors are compact recurrent or convolutional models, often under two megabytes, that run in real time on a single CPU core.
VAD compared with related speech tasks
VAD answers only whether someone is speaking, which separates it from neighboring tasks that answer who is speaking, what is said or whether a turn is complete. Confusing these tasks leads to wrong architecture choices, so the distinctions below matter in system design.
| Task | Question answered | Typical output |
|---|---|---|
| Voice activity detection | Is there speech in this frame? | Speech segments with timestamps |
| Speaker diarization | Who spoke when? | Segments labeled by speaker |
| Turn detection | Has the speaker finished the turn? | End-of-turn event |
| Speech-to-text | Which words were spoken? | Transcript text |
| Language identification | Which language is spoken? | Language code with confidence |
In practice these components are chained. VAD usually runs first, and diarization and transcription consume its segments. Turn detection often builds on VAD silence durations but adds linguistic or prosodic cues, because a pause of 600 milliseconds can mean either hesitation or a finished question. Barge-in handling also depends on VAD, since a voice agent must notice user speech while its own text-to-speech output is playing.
Tuning VAD for a specific application
There is no universally correct VAD configuration; the right setting depends on whether missed speech or false triggers cost more in the target product. Every tuning decision moves the system along a trade-off between sensitivity and false alarms, and between responsiveness and fragmentation.
| Use case | Recommended setting | Reason |
|---|---|---|
| Real-time voice agent | Fast neural VAD, short hangover | Low turn latency matters most |
| Batch transcription | Lower threshold, generous padding | Missing words is worse than extra audio |
| VoIP bandwidth saving | Codec VAD with comfort noise | Silence must not sound like a dropped call |
| Wake-word devices | Ultra-low-power VAD gating a larger model | Battery life depends on idle power |
| Call-center analytics | Robust VAD before diarization | Hold music and line noise are common |
Two practical techniques improve almost every deployment. Speech padding adds 100 to 300 milliseconds of audio before and after each detected segment so that soft word onsets such as plosives or fricatives are not clipped. Domain calibration sets the threshold on a few hours of real audio from the target environment instead of relying on default values tuned for clean studio speech.
Measuring VAD accuracy and common failure modes
VAD quality is measured with frame-level error rates: the miss rate counts speech frames labeled as non-speech, and the false alarm rate counts non-speech frames labeled as speech. Many evaluations combine them in a detection error trade-off curve or report the area under the ROC curve, because a single threshold hides how the detector behaves at other operating points.
Errors propagate downstream in characteristic ways. A missed speech onset deletes the first word and raises the word error rate, while false alarms on noise make recognizers hallucinate words from music or keyboard clicks. Long false segments also waste GPU time in transcription services billed per audio second.
- Clipped word beginnings: threshold too high or no pre-padding; lower the threshold or add padding.
- Fragmented sentences: hangover too short; extend it to bridge natural pauses.
- Triggering on music or TV: energy-based detector in a noisy room; switch to a neural VAD.
- Self-triggering on agent audio: missing echo cancellation; apply acoustic echo cancellation before VAD.
- Silence on whispered speech: model trained on normal voice levels; fine-tune with quiet speech samples.
A frequent misconception is that a more accurate recognizer makes VAD unnecessary. Large end-to-end recognizers do handle silence, yet feeding them long non-speech stretches increases hallucination risk and cost, so a dedicated detector remains a standard pre-processing step.
The simple explanation
Voice activity detection is like a careful note-taker at a meeting. It writes only when someone is talking. It ignores the air conditioner, the rustling paper and the quiet moments. When a person starts to speak, it starts paying attention.
Computers need this helper for two reasons. Processing every second of sound is slow and expensive. Sounds that are not speech also confuse transcription. The key idea is simple: find the speech, skip the rest. The detector also marks where each spoken part begins and ends.
How it is used in practice
A regional health insurer records about 4,000 hours of customer calls each month and transcribes them for quality review. The recordings contain long stretches of hold music, keypad tones and silence while agents look up records. Sending the full audio to a speech-to-text service produced costly bills and transcripts filled with invented words during the music.
The engineering team placed a neural VAD in front of the recognizer, with a speech threshold calibrated on 20 hours of their own calls, 200 milliseconds of padding and a 250-millisecond hangover. The detector removed roughly a third of the audio before transcription. As a result, transcription cost fell by about 35 percent, the phantom words during hold music disappeared, and the downstream speaker diarization ran faster because it received clean, well-bounded speech segments.