Voice cloning is the use of speech synthesis models to reproduce the voice of a specific person from recordings of that person, so that new speech can be generated in their timbre, accent and speaking style. In speech and language artificial intelligence, voice cloning is a specialized form of text-to-speech: instead of a generic synthetic voice, the output is conditioned on a target speaker’s identity.
Modern systems can produce convincing results from a few minutes or even a few seconds of audio, which has made voice cloning valuable for accessibility, dubbing and content production, and at the same time a serious fraud and misinformation risk. Responsible use therefore depends as much on consent and safeguards as on audio quality.
How voice cloning works
Voice cloning separates what is said from who says it. A speech model learns to generate audio from text or speech tokens, and a speaker representation tells it whose voice to use. The representation can be a fixed vector, an adapted set of model weights or a short audio prompt.
- Capture: record clean reference audio of the target speaker.
- Encode the speaker: a speaker encoder turns the audio into an embedding that summarizes timbre and style, or the reference audio is kept as a prompt.
- Generate acoustic features or tokens: the synthesis model predicts a mel spectrogram or discrete audio codec tokens from the input text, conditioned on the speaker.
- Vocode: a neural vocoder or codec decoder turns these features into a waveform, typically at 22.05-48 kHz.
Many recent systems treat speech like language: a neural audio codec compresses audio into discrete tokens, and a transformer predicts the next tokens given text and a short voice prompt. This in-context approach enables zero-shot cloning without any training on the target voice.
Cloning approaches and data requirements
Voice cloning methods differ mainly in how much target audio they need and whether they change the model’s weights. More data and adaptation generally yield higher speaker similarity and stability, at the cost of time and compute.
| Approach | Typical target audio | Characteristics |
|---|---|---|
| Zero-shot cloning | About 3-30 seconds | Instant, no training; similarity and consistency vary |
| Few-shot adaptation | About 1-30 minutes | Fine-tunes part of the model; noticeably closer match |
| Professional voice model | One to several hours of studio recordings | Highest quality and stability; costly to produce |
| Voice conversion | Reference audio of target plus source speech | Changes who speaks while keeping the original delivery |
Voice conversion deserves a clear distinction. Text-based cloning generates speech from written input, while voice conversion transforms an existing recording or a live microphone signal into the target voice. Conversion preserves the timing, emphasis and emotion of the source performer, which is why it is favored in dubbing and singing applications.
Quality factors and evaluation
A cloned voice is judged on naturalness, speaker similarity and intelligibility. These qualities can diverge: a clone can sound natural but not like the target, or similar but with mispronunciations.
| Quality dimension | Common measure | What affects it |
|---|---|---|
| Naturalness | Mean opinion score from listeners on a 1-5 scale | Model quality, prosody modeling, vocoder |
| Speaker similarity | Similarity MOS or cosine similarity of speaker embeddings | Amount and quality of reference audio |
| Intelligibility | Word error rate from a speech-to-text system | Pronunciation, articulation, noise |
| Prosody | Listener ratings of rhythm, stress and intonation | Context modeling, punctuation handling |
Reference audio quality has an outsized effect. Background noise, room echo and compression artifacts are learned as part of the voice, so a clone built from a phone call sounds like a phone call. Clean, varied recordings that cover different sentence types and emotions produce the most flexible clones.
Voice cloning compared with related speech technologies
Voice cloning shares components with other speech technologies but differs in its goal of reproducing a particular identity.
| Technology | Input | Output |
|---|---|---|
| Text-to-speech | Text | Speech in a predefined synthetic voice |
| Voice cloning | Text plus target speaker audio | Speech in the target speaker’s voice |
| Speech-to-text | Speech | Written transcript |
| Speech translation with voice preservation | Speech in one language | Speech in another language in the same voice |
| Speaker verification | Speech sample | Decision whether it matches an enrolled speaker |
The relation to speaker verification is a security concern: cloned voices can defeat voice-based authentication. Banks and call centers that rely on voiceprints increasingly add liveness checks and additional factors.
Risks, consent and safeguards
The main risks of voice cloning come from impersonation without consent. Typical misuse includes phone scams in which a cloned relative or executive requests urgent money transfers, fabricated audio of public figures and harassment through synthetic recordings.
- Documented consent: obtain explicit, revocable permission from the voice owner for defined uses.
- Speaker verification at enrollment: confirm that the person creating a clone is the person being cloned.
- Watermarking: embed inaudible signals that mark audio as synthetic.
- Provenance metadata: attach content credentials describing how the audio was produced.
- Usage restrictions: block cloning of public figures and monitor for abusive content.
- Disclosure: tell listeners when a voice is synthetic.
Detection of synthetic speech remains an arms race: detectors trained on one generation of models often fail on the next. Organizations should therefore treat a voice alone as insufficient proof of identity and verify unusual requests through a separate channel.
Stripped to the essentials
Voice cloning is software that learns how a particular person sounds and then speaks any new sentence in that voice. Give it some recordings of someone, type a sentence, and the result sounds as if that person said it.
This can help someone who is losing their voice to illness, or let a narrator record an audiobook in several languages. It can also be used to fake a phone call from a family member. That is why consent and caution matter: a familiar voice on the phone is no longer proof of who is really speaking.
Where it is applied
A teacher diagnosed with a progressive neuromuscular disease expects to lose the ability to speak within a few years. On the advice of a speech therapist, she records about two hours of varied speech while her voice is still strong, including questions, exclamations and names of students.
A clinical team uses these recordings to create a personal synthetic voice for her communication device. They test it with family members, who rate how much it sounds like her, and adjust pronunciations of names that the model got wrong.
When she later relies on the device, it speaks in a voice her students and family recognize rather than a generic one. The voice model is stored under her control, and her written consent specifies that it may be used only for her own communication.