Voice activity detection

Also known as: VAD

On this page 7

Voice activity detection (VAD) is the process of deciding, for each short slice of an audio signal, whether it contains human speech or non-speech such as silence, music or background noise. In Speech & language AI, VAD is the gatekeeper of the audio pipeline: it tells speech-to-text engines, telephony codecs and voice assistants when to listen, when to transmit and when a speaker has stopped.

A detector rarely sees the meaning of words. It works on acoustic evidence in frames of 10 to 30 milliseconds and outputs a probability or a binary label, which is then smoothed into speech segments with start and end timestamps. The quality of those boundaries directly shapes recognition accuracy, latency and compute cost downstream.

How voice activity detection classifies audio frames

A VAD system splits the incoming signal into overlapping frames, extracts features from each frame, scores the likelihood of speech and then applies temporal smoothing to turn noisy frame decisions into stable segments. Each stage can be classical signal processing or a trained neural network.

The following table summarizes the shared processing stages that almost every detector uses, regardless of whether it is a decades-old codec component or a modern neural model.

StageWhat happensTypical setting
FramingAudio is cut into short windows10-30 ms frames, 50% overlap
Feature extractionEnergy, spectrum or learned embeddings are computedLog-mel bands, zero-crossing rate
Frame scoringA model estimates speech probabilityValue between 0 and 1
ThresholdingProbability becomes a speech or non-speech labelThreshold near 0.5, tuned per domain
Hangover and smoothingShort gaps are bridged and short bursts removed100-300 ms hangover
Processing stages of a typical VAD pipeline

The hangover period is the least visible yet most consequential setting. Speech contains natural micro-pauses between syllables and words, and without a hangover the detector would chop a sentence into fragments. A hangover that is too long, however, keeps the channel open after the speaker stops and adds latency to every turn.

Main families of VAD algorithms

VAD algorithms fall into three broad families: energy-based rules, statistical models and neural classifiers. They differ in robustness to noise, computational cost and the amount of tuning they require, and each family remains in production somewhere today.

FamilyCore ideaStrengthWeakness
Energy thresholdSpeech is louder than the noise floorAlmost zero computeFails with loud noise or music
Statistical modelGaussian or likelihood-ratio tests on spectral featuresRuns on embedded chipsStruggles with nonstationary noise
Neural VADA small network learns speech patterns from labeled audioHigh accuracy in noiseNeeds training data and more compute
ASR-integrated detectionThe recognizer itself predicts silence or end of speechShares context with transcriptionTied to one recognition model
Comparison of VAD algorithm families

Classical detectors are standardized in telecommunications, where ITU-T G.729 Annex B and the VAD options of the AMR codec allow phones to stop transmitting during silence. The widely used WebRTC VAD follows the statistical approach with Gaussian mixture models. Modern neural detectors are compact recurrent or convolutional models, often under two megabytes, that run in real time on a single CPU core.

VAD answers only whether someone is speaking, which separates it from neighboring tasks that answer who is speaking, what is said or whether a turn is complete. Confusing these tasks leads to wrong architecture choices, so the distinctions below matter in system design.

TaskQuestion answeredTypical output
Voice activity detectionIs there speech in this frame?Speech segments with timestamps
Speaker diarizationWho spoke when?Segments labeled by speaker
Turn detectionHas the speaker finished the turn?End-of-turn event
Speech-to-textWhich words were spoken?Transcript text
Language identificationWhich language is spoken?Language code with confidence
How VAD differs from neighboring speech tasks

In practice these components are chained. VAD usually runs first, and diarization and transcription consume its segments. Turn detection often builds on VAD silence durations but adds linguistic or prosodic cues, because a pause of 600 milliseconds can mean either hesitation or a finished question. Barge-in handling also depends on VAD, since a voice agent must notice user speech while its own text-to-speech output is playing.

Tuning VAD for a specific application

There is no universally correct VAD configuration; the right setting depends on whether missed speech or false triggers cost more in the target product. Every tuning decision moves the system along a trade-off between sensitivity and false alarms, and between responsiveness and fragmentation.

Use caseRecommended settingReason
Real-time voice agentFast neural VAD, short hangoverLow turn latency matters most
Batch transcriptionLower threshold, generous paddingMissing words is worse than extra audio
VoIP bandwidth savingCodec VAD with comfort noiseSilence must not sound like a dropped call
Wake-word devicesUltra-low-power VAD gating a larger modelBattery life depends on idle power
Call-center analyticsRobust VAD before diarizationHold music and line noise are common
Recommended VAD strategy by use case

Two practical techniques improve almost every deployment. Speech padding adds 100 to 300 milliseconds of audio before and after each detected segment so that soft word onsets such as plosives or fricatives are not clipped. Domain calibration sets the threshold on a few hours of real audio from the target environment instead of relying on default values tuned for clean studio speech.

Measuring VAD accuracy and common failure modes

VAD quality is measured with frame-level error rates: the miss rate counts speech frames labeled as non-speech, and the false alarm rate counts non-speech frames labeled as speech. Many evaluations combine them in a detection error trade-off curve or report the area under the ROC curve, because a single threshold hides how the detector behaves at other operating points.

Errors propagate downstream in characteristic ways. A missed speech onset deletes the first word and raises the word error rate, while false alarms on noise make recognizers hallucinate words from music or keyboard clicks. Long false segments also waste GPU time in transcription services billed per audio second.

  • Clipped word beginnings: threshold too high or no pre-padding; lower the threshold or add padding.
  • Fragmented sentences: hangover too short; extend it to bridge natural pauses.
  • Triggering on music or TV: energy-based detector in a noisy room; switch to a neural VAD.
  • Self-triggering on agent audio: missing echo cancellation; apply acoustic echo cancellation before VAD.
  • Silence on whispered speech: model trained on normal voice levels; fine-tune with quiet speech samples.

A frequent misconception is that a more accurate recognizer makes VAD unnecessary. Large end-to-end recognizers do handle silence, yet feeding them long non-speech stretches increases hallucination risk and cost, so a dedicated detector remains a standard pre-processing step.

The simple explanation

Voice activity detection is like a careful note-taker at a meeting. It writes only when someone is talking. It ignores the air conditioner, the rustling paper and the quiet moments. When a person starts to speak, it starts paying attention.

Computers need this helper for two reasons. Processing every second of sound is slow and expensive. Sounds that are not speech also confuse transcription. The key idea is simple: find the speech, skip the rest. The detector also marks where each spoken part begins and ends.

How it is used in practice

A regional health insurer records about 4,000 hours of customer calls each month and transcribes them for quality review. The recordings contain long stretches of hold music, keypad tones and silence while agents look up records. Sending the full audio to a speech-to-text service produced costly bills and transcripts filled with invented words during the music.

The engineering team placed a neural VAD in front of the recognizer, with a speech threshold calibrated on 20 hours of their own calls, 200 milliseconds of padding and a 250-millisecond hangover. The detector removed roughly a third of the audio before transcription. As a result, transcription cost fell by about 35 percent, the phantom words during hold music disappeared, and the downstream speaker diarization ran faster because it received clean, well-bounded speech segments.