Audio-visual speech recognition

Also known as: AVSR

On this page 7

Audio-visual speech recognition (AVSR) is the task of converting spoken language into text by jointly analyzing the acoustic signal and video of the speaker’s mouth region. In multimodal AI, it is a classic case of two modalities carrying partly redundant and partly complementary information about the same event, so that lip movements compensate for degraded audio.

The idea rests on a well-documented property of human perception: listeners understand speech in noise considerably better when they can see the face, and conflicting cues can even change what is heard, as the McGurk effect shows. AVSR systems formalize this advantage and are most valuable where acoustic conditions are poor, such as crowded rooms, vehicles, factory floors or recordings with overlapping speakers.

How audio-visual speech recognition works

A modern AVSR system is a pipeline of two modality-specific encoders, a fusion stage and a text decoder. The audio branch typically consumes log-Mel filterbank features or raw waveform frames, while the visual branch receives a cropped, grayscale region of interest around the lips, usually 88 by 88 pixels at 25 frames per second.

The visual front end is commonly a 3D convolution followed by a ResNet, which captures short-range lip motion, and both streams then pass through Transformer or Conformer encoders. Because audio frames arrive at a higher rate than video frames, the system must align the two streams in time, either by upsampling the visual features or by subsampling the acoustic ones.

The following table lists the main components of a typical end-to-end system and the role each one plays.

ComponentTypical input or designRole
Face and lip detectionLandmark detector on each video frameLocates and crops the mouth region
Visual front end3D convolution plus ResNet-18Encodes lip shape and motion
Audio front endLog-Mel features or 1D convolution on waveformEncodes acoustic content
Temporal encodersConformer or Transformer layersModel longer context in each stream
Fusion moduleConcatenation, attention or gatingCombines evidence from both streams
DecoderCTC, attention decoder or hybridProduces characters, subwords or words
Main components of an AVSR pipeline

Fusion strategies in AVSR

The fusion strategy determines where and how audio and visual evidence are merged, and it largely decides how gracefully the system handles a weak modality. Early statistical systems based on hidden Markov models used decision fusion with stream weights, whereas neural systems mostly fuse intermediate representations.

The comparison below summarizes the three broad options, which correspond to the general concepts of early, intermediate and late fusion in multimodal learning.

StrategyWhere streams meetStrengthWeakness
Early fusionConcatenated input featuresCaptures fine low-level correlationsSensitive to frame-rate mismatch and missing video
Intermediate fusionEncoder outputs or hidden layersBest accuracy in most modern systemsNeeds paired audio-visual training data
Late fusionSeparate posteriors or hypothesesRobust when one stream failsMisses cross-modal interactions
Fusion options for audio-visual speech recognition

Robust designs add reliability-aware weighting, in which a learned gate estimates how trustworthy each stream is at every moment. When the microphone is swamped by noise, the gate shifts weight toward the visual stream; when the face turns away or is occluded, it shifts back toward audio.

Training data, benchmarks and performance

Progress in AVSR has been driven by large public audio-visual corpora, most of them built from broadcast television and recorded talks. The best-known English benchmarks are LRS2 and LRS3, the Lip Reading Sentences datasets, which contain hundreds of hours of sentence-level speech with aligned face tracks.

Self-supervised pretraining has been a turning point. Models such as AV-HuBERT learn joint audio-visual representations from unlabeled video by predicting masked cluster targets, then fine-tune on a comparatively small labeled set. This approach brought visual-only lip reading on LRS3 to roughly 25 to 30 percent word error rate, a level that had seemed out of reach a few years earlier.

Exact figures depend on the model, the noise type and the training data, but the qualitative pattern in the next table is consistent across published studies.

ConditionAudio-only recognizerAudio-visual recognizer
Clean speechVery low error, near saturationSimilar or marginally lower error
Moderate noise, about 5 dB SNRError rises noticeablyClearly lower error than audio-only
Severe noise, 0 dB SNR or belowError often several times higherLargest relative gain from vision
Competing speaker (babble)Confuses target and interfererLips identify the target speaker
Indicative behavior of audio-only and audio-visual systems

The central lesson is that the visual gain grows as the signal-to-noise ratio falls. In clean conditions, a strong audio model leaves little room for improvement, so the visual stream mainly earns its cost in difficult acoustic environments.

AVSR belongs to a family of tasks that link faces and voices, and the boundaries matter when choosing a model or a dataset. It is a specific application within audio-visual learning, distinct from tasks that generate rather than recognize speech.

TaskInputOutputMain difference
Audio-only ASRAudioTextNo visual cue, weaker in noise
Visual speech recognitionSilent lip videoTextNo audio at all, much higher error
Audio-visual speech enhancementNoisy audio plus videoCleaner audioRestores the signal instead of transcribing it
Active speaker detectionAudio plus videoWho is speakingLocates the speaker, does not transcribe
Talking head generationAudio or textSynthetic face videoGenerates lips instead of reading them
How AVSR differs from neighboring tasks

Some multimodal large language models can accept audio and video together, yet general-purpose models are rarely tuned for frame-accurate lip reading. Dedicated AVSR models therefore remain the stronger choice when word error rate in noise is the metric that counts.

Limitations and common failure modes

AVSR adds a camera and a second model branch, so it also adds new ways to fail. Practitioners should evaluate the visual stream under the same conditions the deployed system will face, not only on broadcast benchmarks.

  • Visemes are ambiguous: sounds such as p, b and m look nearly identical on the lips, so vision alone cannot separate them.
  • Pose and occlusion: profile views, hands, microphones or face masks remove much of the visual signal.
  • Lighting and resolution: dim scenes, motion blur and heavy video compression degrade lip features quickly.
  • Audio-visual desynchronization: offsets of more than about 100 milliseconds can hurt fusion noticeably.
  • Dataset bias: broadcast corpora overrepresent certain languages, ages and speaking styles.
  • Privacy obligations: recording faces creates biometric and consent obligations under laws such as the GDPR.

A practical safeguard is modality dropout during training, in which the model randomly loses one stream so that it learns to operate on audio alone, video alone or both. Without it, a system can become overly dependent on audio and gain little from the camera when it is needed most.

Put simply

Audio-visual speech recognition is a computer doing what people do naturally in a loud bar: watching the mouth to fill in words the ears missed. The system listens to the sound and looks at the lips at the same time, then combines both clues before writing down what was said.

When the room is quiet, the sound alone is usually enough. When it gets noisy, the lips become the tiebreaker. The key point is that two imperfect clues together beat either one alone, especially when the sound is poor.

In practice

A hospital installs a voice-controlled documentation assistant in an operating theater, where suction pumps, alarms and conversation keep background noise high. The audio-only recognizer produces frequent errors on drug names and dosages, and surgeons wear masks, so lip reading is impossible for them.

The engineering team therefore applies AVSR at the nurse station, where a small camera faces the documenting nurse, who does not wear a mask while dictating. The model is pretrained on public audio-visual video, fine-tuned on about 40 hours of consented in-house recordings and trained with modality dropout. In noisy periods, internal tests show a substantially lower word error rate than the audio-only baseline, and the remaining errors concentrate on rare drug names, which the team handles with a custom vocabulary. The project also adds a clear consent process and on-premises processing so that no face video leaves the hospital network.