Audio captioning

On this page 7

Audio captioning is the task of automatically generating a natural-language description of an audio clip, summarizing which sound events occur, which sources produce them and in what acoustic scene they happen. Multimodal AI treats it as an audio-to-text generation problem: a model listens to a recording of rain, footsteps and a passing car and writes a sentence such as “rain falls steadily while a car drives past on a wet road”.

Audio captioning is not speech recognition. A transcription system writes down the words that people say, whereas a captioning model describes what the audio sounds like, including non-speech sounds, their order and their relationships. The task sits between sound event detection, which outputs fixed labels, and full audio language models, which answer open questions about sound.

How audio captioning models turn sound into sentences

Most audio captioning systems follow an encoder-decoder architecture. The audio is first converted into a log-mel spectrogram, a time-frequency image that approximates human pitch perception, and an audio encoder turns that spectrogram into a sequence of embeddings. A text decoder then generates the caption word by word, attending to the audio embeddings through cross-attention at every step.

The encoder is usually pretrained on large tagging datasets such as AudioSet, which contains about two million ten-second clips labeled with 527 sound classes. Common choices are convolutional networks such as PANNs, transformer encoders such as the Audio Spectrogram Transformer, and self-supervised encoders such as BEATs. Pretraining matters because captioning datasets are small, and an encoder that already recognizes hundreds of sound classes gives the decoder a strong starting vocabulary.

The following table summarizes the main components of a typical pipeline and the role each one plays.

ComponentTypical implementationRole in the system
Front endLog-mel spectrogram, 64-128 mel bandsConverts the waveform into a time-frequency input
Audio encoderPANNs, AST, HTS-AT, BEATsExtracts sound event and scene features
BridgeLinear projection or Q-FormerMaps audio embeddings into the decoder space
Text decoderTransformer, BART, GPT-2 or an LLMGenerates the caption token by token
Decoding strategyBeam search, nucleus samplingBalances accuracy against caption diversity
Components of an audio captioning pipeline

Newer systems replace the small decoder with a large language model. A lightweight projector maps audio embeddings into the language model input space as a short prefix of audio tokens, so the model writes captions with richer grammar and broader vocabulary. This design links audio captioning directly to the wider family of multimodal large language models.

Datasets and training strategies

Audio captioning depends on human-written reference captions, which are expensive to collect, so the field relies on a few curated datasets. Each clip normally has several reference captions because different listeners describe the same sound in different words.

DatasetClip lengthApproximate sizeDistinctive feature
Clotho15-30 sAbout 6,000 clips, five captions eachFreesound audio, captions written without metadata
AudioCaps10 sAbout 50,000 clipsBuilt on AudioSet with tag hints for annotators
MACS10 sAbout 4,000 clipsSeveral annotators per urban scene clip
WavCapsVariableAbout 400,000 clipsWeakly labeled captions rewritten by an LLM
Widely used audio captioning datasets

Training usually starts with cross-entropy loss on reference captions, where the decoder predicts each next word given the true previous words. Because this teacher forcing differs from inference, many systems add reinforcement learning that directly optimizes a caption metric such as CIDEr. That step raises benchmark scores but can produce repetitive, generic phrasing, so it is applied with care.

Data augmentation such as SpecAugment, mixup of two clips and caption paraphrasing reduces overfitting. Contrastive audio-text pretraining, as in CLAP models, aligns audio and text in a shared embedding space and gives the encoder representations that transfer well to captioning.

Evaluating caption quality

Audio captioning is evaluated by comparing generated captions with human references, but no single metric captures caption quality well. Early metrics were borrowed from machine translation and image captioning and reward word overlap, which penalizes valid paraphrases. Newer metrics use sentence embeddings to judge meaning rather than exact wording.

MetricWhat it measuresMain weakness
BLEU and METEORN-gram overlap with referencesIgnores synonyms and sentence meaning
CIDErConsensus with references, weighted by rarityFavors frequent dataset phrasing
SPICEOverlap of objects, attributes and relationsDepends on parser quality
SPIDErAverage of CIDEr and SPICEInherits both biases
FENSEEmbedding similarity plus fluency error checkRelies on a learned text model
Common evaluation metrics and what they reward

Practitioners therefore report several metrics together and complement them with human listening tests. A model with a high SPIDEr score can still produce fluent but wrong captions, for example describing a dog barking when the clip contains a sea lion, which is the audio counterpart of multimodal hallucination.

Audio captioning overlaps with several neighboring tasks, and the type of output is the clearest way to separate them. The comparison below shows where captioning fits.

TaskInputOutput
Audio captioningAudio clipFree-form descriptive sentence
Audio taggingAudio clipSet of class labels such as dog or siren
Sound event detectionAudio clipLabels with start and end times
Speech recognitionSpeech audioVerbatim transcript of spoken words
Video captioningVideo with or without soundSentence describing visual and audio content
Text-to-audio retrievalText queryRanked list of matching audio clips
Audio captioning compared with neighboring tasks

Captioning is harder than tagging because the model must express temporal order and relationships, such as one sound following or masking another. It is also the reverse direction of text-to-audio generation, and the same paired datasets often serve both tasks as well as cross-modal retrieval between audio and text.

Limitations and common failure modes

Audio captioning remains less mature than image captioning, mainly because sounds are ambiguous and training data is scarce. Typical problems include the following:

  • Acoustic ambiguity: frying food, rain and applause can produce very similar spectra, so models confuse them without context.
  • Generic captions: models trained with metric optimization often fall back on safe phrases such as “a person is talking”.
  • Weak temporal reasoning: many systems list sounds correctly but describe their order or overlap incorrectly.
  • Dataset bias: Freesound and YouTube audio overrepresent certain sounds, which reduces accuracy on industrial or medical recordings.
  • Long recordings: most models handle clips under 30 seconds and need segmentation for longer audio.

Deployments should therefore treat captions as descriptive metadata with uncertainty, not as verified facts, and keep a human review step wherever a wrong description could cause harm.

In everyday language

Audio captioning means a computer listens to a sound recording and writes a short sentence about it. It does not write down what people say. It describes what you would hear, such as birds singing, a door slamming or traffic in the background.

Think of a friend who listens to a recording with closed eyes and tells you what is happening. The computer learned this skill by studying thousands of clips that people had already described in words. It is often right, but it can mistake one sound for another that sounds similar.

Seen in the real world

A public broadcaster manages an archive of about 200,000 unlabeled sound-effect recordings collected over several decades. Editors searching for “thunder over open fields” previously depended on file names such as fx_0412.wav, so most of the archive was effectively invisible.

The archive team runs an audio captioning model with a pretrained transformer encoder over every clip and stores one generated caption per recording next to the existing metadata. Archivists review a random sample of 1,000 captions and correct the systematic errors they find, such as rain being described as frying.

The outcome is a searchable text index for the whole archive. Editors now find suitable effects by describing them in plain words, and the team uses the corrected samples to fine-tune the model for the next pass.