Voice conversion

On this page 7

Voice conversion is the task of transforming an utterance spoken by a source speaker so that it sounds as if a target speaker said it, while the linguistic content stays unchanged. In Speech & language AI it sits between analysis and synthesis: the system takes audio in and produces audio out, without passing through written text, which is what separates it from text-to-speech.

Modern systems are built on neural networks that separate what is said from who says it and then recombine the content with a new speaker identity. The technique supports dubbing, accessibility tools and privacy protection, but it also enables audio deepfakes, so consent, disclosure and detection are part of any serious deployment.

How voice conversion separates content from identity

Voice conversion works by disentangling speech into components and replacing only the component that carries speaker identity. A typical pipeline has three stages: an encoder extracts speaker-independent content, a conversion model combines that content with a target speaker representation, and a neural vocoder turns the resulting acoustic features back into a waveform.

The content representation is the critical design choice. Early systems used phonetic posteriorgrams from a speech recognizer; current systems often use self-supervised speech features such as HuBERT or WavLM units, which capture phonetic information while discarding much of the timbre. Speaker identity is usually injected as a speaker embedding, a fixed-length vector computed from a reference recording.

The following table summarizes the main components and what each one controls in the converted output.

ComponentTypical inputWhat it controls
Content encoderSource waveform or spectrogramPhonemes, words and timing
Speaker encoderReference audio of the targetTimbre and vocal tract character
Pitch extractorSource waveformFundamental frequency contour
Decoder or generatorContent, speaker and pitch featuresMel spectrogram of the new voice
Neural vocoderMel spectrogramFinal waveform quality and naturalness
Core components of a neural voice conversion pipeline

Types of voice conversion systems

Voice conversion systems differ mainly in what training data they need and how many speakers they can handle. Parallel systems require the same sentences recorded by both speakers, which is expensive; non-parallel systems learn from unrelated recordings and dominate current practice.

The table below compares the main variants by data requirement and typical use.

VariantTraining dataTypical use
Parallel one-to-oneIdentical sentences from two speakersResearch baselines, early systems
Non-parallel many-to-manyUnrelated speech from many speakersStudio dubbing with known voices
Any-to-oneLarge corpus plus one target voiceCharacter voices, voice banking
Zero-shot any-to-anyLarge multi-speaker corpusConversion to unseen voices from seconds of audio
Real-time streamingAny of the above, causal modelLive calls, streaming, gaming
Voice conversion variants by data and speaker coverage

Historically, the field moved from Gaussian mixture models that mapped spectral features frame by frame, through variational autoencoders and GAN-based models such as CycleGAN-VC and StarGAN-VC, to diffusion models and token-based language models of audio. Each step improved naturalness and reduced the amount of target data needed, from hours to roughly 3 to 30 seconds for zero-shot systems.

Voice conversion is often confused with voice cloning and text-to-speech, but the input modality is the clearest distinction. Voice conversion always starts from speech, so it inherits the source performance: rhythm, pauses, emphasis and emotion come from the person who actually spoke.

The comparison below shows how the neighboring tasks relate.

TaskInputOutputMain difference
Voice conversionSpeechSpeech in another voiceKeeps source content and performance
Text-to-speechTextSpeechProsody must be generated, not copied
Voice cloningShort reference audioA reusable synthetic voiceDescribes the goal, often realized by TTS or conversion
Speech-to-speech translationSpeech in one languageSpeech in another languageChanges the words, not only the voice
Speaker verificationSpeechAccept or reject decisionRecognizes identity rather than changing it
How voice conversion differs from neighboring tasks

The boundary with prosody is subtle. Pitch is usually normalized to the target speaker’s range, because a male-to-female conversion that keeps the original pitch sounds unnatural, but the relative intonation contour is preserved so the sentence keeps its meaning and emotion.

Quality metrics and evaluation

Voice conversion is judged on three separate axes: how natural the output sounds, how closely it matches the target speaker, and how well the original words survive. A system can score well on one axis and fail another, so all three are reported together.

The table lists the most common measurements and what they reveal.

MeasureWhat it assessesInterpretation
Mean opinion scorePerceived naturalness1 to 5 scale, higher is better
Speaker similarity scoreCloseness to the target voiceListener rating or embedding cosine similarity
Word or character error rateContent preservationMeasured with a speech recognizer, lower is better
Mel cepstral distortionSpectral distance to referenceLower is better, needs parallel data
F0 correlationPreservation of intonationCloser to 1 means contour is kept
Common evaluation measures for converted speech

Objective metrics are useful during development, but listening tests remain the reference. Automatic similarity scores can be fooled by artifacts that human listeners immediately notice, and the Voice Conversion Challenge series established crowdsourced listening tests as the accepted benchmark practice.

Limitations, risks and safeguards

The main technical limitation is imperfect disentanglement. If the content features still carry traces of the source speaker, the output sounds like a blend of both voices; if they discard too much, pronunciation degrades and the word error rate rises.

  • Accent leakage: the source accent usually survives conversion, because accent lives partly in the phonetic content.
  • Noisy or reverberant input degrades both content extraction and the final waveform.
  • Cross-gender and cross-age conversion remain harder than conversion between similar voices.
  • Real-time latency typically needs to stay below about 50 to 100 milliseconds for natural conversation, which limits model size.
  • Singing voice conversion requires precise pitch handling and is usually treated as a separate specialization.

The ethical risks are significant. Converted speech can impersonate real people in fraud calls or fabricated recordings and can defeat weak voice-based authentication. Responsible deployment includes documented consent from the target speaker, audible or inaudible watermarking, and anti-spoofing countermeasures of the kind evaluated in the ASVspoof challenge series.

Stripped to the essentials

Voice conversion takes a recording of one person talking and makes it sound like someone else said it. The words, the pauses and the feeling in the sentence stay the same. Only the sound of the voice changes.

Think of it as a costume for a voice. The actor underneath still decides how each line is delivered, but listeners hear a different person. Modern tools can learn a new voice from a very short sample, which is why permission from the voice owner matters.

Where it is applied

A video game studio needs one character to speak in twelve languages with the same recognizable voice. The original voice actor records only in English, so local actors perform the dialogue in each language, and a licensed any-to-one conversion model trained on two hours of the original actor’s studio recordings maps every local performance onto that voice.

The studio keeps the local actors’ timing and emotional delivery, checks content preservation by running each converted line through a speech recognizer, and flags any line whose word error rate exceeds 5 percent for re-recording. A final listening panel rates speaker similarity before release. The result is a consistent character voice across all markets, produced under a contract that covers the voice license and with every converted file marked by an inaudible watermark.