Audio captioning is the task of automatically generating a natural-language description of an audio clip, summarizing which sound events occur, which sources produce them and in what acoustic scene they happen. Multimodal AI treats it as an audio-to-text generation problem: a model listens to a recording of rain, footsteps and a passing car and writes a sentence such as “rain falls steadily while a car drives past on a wet road”.
Audio captioning is not speech recognition. A transcription system writes down the words that people say, whereas a captioning model describes what the audio sounds like, including non-speech sounds, their order and their relationships. The task sits between sound event detection, which outputs fixed labels, and full audio language models, which answer open questions about sound.
How audio captioning models turn sound into sentences
Most audio captioning systems follow an encoder-decoder architecture. The audio is first converted into a log-mel spectrogram, a time-frequency image that approximates human pitch perception, and an audio encoder turns that spectrogram into a sequence of embeddings. A text decoder then generates the caption word by word, attending to the audio embeddings through cross-attention at every step.
The encoder is usually pretrained on large tagging datasets such as AudioSet, which contains about two million ten-second clips labeled with 527 sound classes. Common choices are convolutional networks such as PANNs, transformer encoders such as the Audio Spectrogram Transformer, and self-supervised encoders such as BEATs. Pretraining matters because captioning datasets are small, and an encoder that already recognizes hundreds of sound classes gives the decoder a strong starting vocabulary.
The following table summarizes the main components of a typical pipeline and the role each one plays.
| Component | Typical implementation | Role in the system |
|---|---|---|
| Front end | Log-mel spectrogram, 64-128 mel bands | Converts the waveform into a time-frequency input |
| Audio encoder | PANNs, AST, HTS-AT, BEATs | Extracts sound event and scene features |
| Bridge | Linear projection or Q-Former | Maps audio embeddings into the decoder space |
| Text decoder | Transformer, BART, GPT-2 or an LLM | Generates the caption token by token |
| Decoding strategy | Beam search, nucleus sampling | Balances accuracy against caption diversity |
Newer systems replace the small decoder with a large language model. A lightweight projector maps audio embeddings into the language model input space as a short prefix of audio tokens, so the model writes captions with richer grammar and broader vocabulary. This design links audio captioning directly to the wider family of multimodal large language models.
Datasets and training strategies
Audio captioning depends on human-written reference captions, which are expensive to collect, so the field relies on a few curated datasets. Each clip normally has several reference captions because different listeners describe the same sound in different words.
| Dataset | Clip length | Approximate size | Distinctive feature |
|---|---|---|---|
| Clotho | 15-30 s | About 6,000 clips, five captions each | Freesound audio, captions written without metadata |
| AudioCaps | 10 s | About 50,000 clips | Built on AudioSet with tag hints for annotators |
| MACS | 10 s | About 4,000 clips | Several annotators per urban scene clip |
| WavCaps | Variable | About 400,000 clips | Weakly labeled captions rewritten by an LLM |
Training usually starts with cross-entropy loss on reference captions, where the decoder predicts each next word given the true previous words. Because this teacher forcing differs from inference, many systems add reinforcement learning that directly optimizes a caption metric such as CIDEr. That step raises benchmark scores but can produce repetitive, generic phrasing, so it is applied with care.
Data augmentation such as SpecAugment, mixup of two clips and caption paraphrasing reduces overfitting. Contrastive audio-text pretraining, as in CLAP models, aligns audio and text in a shared embedding space and gives the encoder representations that transfer well to captioning.
Evaluating caption quality
Audio captioning is evaluated by comparing generated captions with human references, but no single metric captures caption quality well. Early metrics were borrowed from machine translation and image captioning and reward word overlap, which penalizes valid paraphrases. Newer metrics use sentence embeddings to judge meaning rather than exact wording.
| Metric | What it measures | Main weakness |
|---|---|---|
| BLEU and METEOR | N-gram overlap with references | Ignores synonyms and sentence meaning |
| CIDEr | Consensus with references, weighted by rarity | Favors frequent dataset phrasing |
| SPICE | Overlap of objects, attributes and relations | Depends on parser quality |
| SPIDEr | Average of CIDEr and SPICE | Inherits both biases |
| FENSE | Embedding similarity plus fluency error check | Relies on a learned text model |
Practitioners therefore report several metrics together and complement them with human listening tests. A model with a high SPIDEr score can still produce fluent but wrong captions, for example describing a dog barking when the clip contains a sea lion, which is the audio counterpart of multimodal hallucination.
How audio captioning differs from related tasks
Audio captioning overlaps with several neighboring tasks, and the type of output is the clearest way to separate them. The comparison below shows where captioning fits.
| Task | Input | Output |
|---|---|---|
| Audio captioning | Audio clip | Free-form descriptive sentence |
| Audio tagging | Audio clip | Set of class labels such as dog or siren |
| Sound event detection | Audio clip | Labels with start and end times |
| Speech recognition | Speech audio | Verbatim transcript of spoken words |
| Video captioning | Video with or without sound | Sentence describing visual and audio content |
| Text-to-audio retrieval | Text query | Ranked list of matching audio clips |
Captioning is harder than tagging because the model must express temporal order and relationships, such as one sound following or masking another. It is also the reverse direction of text-to-audio generation, and the same paired datasets often serve both tasks as well as cross-modal retrieval between audio and text.
Limitations and common failure modes
Audio captioning remains less mature than image captioning, mainly because sounds are ambiguous and training data is scarce. Typical problems include the following:
- Acoustic ambiguity: frying food, rain and applause can produce very similar spectra, so models confuse them without context.
- Generic captions: models trained with metric optimization often fall back on safe phrases such as “a person is talking”.
- Weak temporal reasoning: many systems list sounds correctly but describe their order or overlap incorrectly.
- Dataset bias: Freesound and YouTube audio overrepresent certain sounds, which reduces accuracy on industrial or medical recordings.
- Long recordings: most models handle clips under 30 seconds and need segmentation for longer audio.
Deployments should therefore treat captions as descriptive metadata with uncertainty, not as verified facts, and keep a human review step wherever a wrong description could cause harm.
In everyday language
Audio captioning means a computer listens to a sound recording and writes a short sentence about it. It does not write down what people say. It describes what you would hear, such as birds singing, a door slamming or traffic in the background.
Think of a friend who listens to a recording with closed eyes and tells you what is happening. The computer learned this skill by studying thousands of clips that people had already described in words. It is often right, but it can mistake one sound for another that sounds similar.
Seen in the real world
A public broadcaster manages an archive of about 200,000 unlabeled sound-effect recordings collected over several decades. Editors searching for “thunder over open fields” previously depended on file names such as fx_0412.wav, so most of the archive was effectively invisible.
The archive team runs an audio captioning model with a pretrained transformer encoder over every clip and stores one generated caption per recording next to the existing metadata. Archivists review a random sample of 1,000 captions and correct the systematic errors they find, such as rain being described as frying.
The outcome is a searchable text index for the whole archive. Editors now find suitable effects by describing them in plain words, and the team uses the corrected samples to fine-tune the model for the next pass.