Speech-to-speech model

Also known as: end-to-end voice model

On this page 7

A speech-to-speech model is a neural network that accepts spoken audio as input and generates spoken audio as output, so a user can talk to it and hear an answer, a translation or a converted voice in return. In multimodal AI, the term usually refers to an end-to-end voice model that processes speech directly rather than chaining separate recognition, language and synthesis systems, which allows it to keep tone, emotion and timing that a text transcript would discard.

Speech-to-speech models power real-time voice assistants, simultaneous speech translation and voice conversion. Their defining engineering goals are low response latency, natural turn-taking and faithful handling of paralinguistic information such as emphasis, hesitation and laughter.

How a speech-to-speech model processes audio

A speech-to-speech model converts an incoming waveform into discrete or continuous audio representations, reasons over them, and decodes new representations back into a waveform. The typical pipeline has three stages: an audio encoder or audio tokenizer, a sequence model (often a transformer language model), and a neural vocoder or codec decoder that reconstructs sound.

Most current designs rely on neural audio codecs that compress speech into tokens at roughly 12 to 75 frames per second, using residual vector quantization with several codebooks per frame. Semantic tokens capture what is said, while acoustic tokens capture voice identity, pitch and recording conditions. The sequence model predicts the next tokens autoregressively, and the codec decoder turns them into audio in small chunks so that playback can start before the full answer is generated.

The table below lists the main components and what each one contributes.

ComponentFunctionTypical design
Audio encoderTurns waveform into features or tokensConformer or self-supervised speech encoder
Audio tokenizerDiscretizes speech into codec tokensResidual vector quantization, 4 to 32 codebooks
Sequence modelUnderstands input and plans the responseDecoder-only transformer, often initialized from a text LLM
Codec decoder or vocoderReconstructs audible waveformStreaming neural codec decoder or GAN vocoder
Turn-taking controllerDecides when to speak or stopVoice activity detection or learned full-duplex modeling
Core components of a speech-to-speech model

Cascaded pipelines versus end-to-end models

A speech-to-speech system can be cascaded or end-to-end, and the difference decides latency, expressiveness and controllability. A cascade links automatic speech recognition (ASR), a text language model and text-to-speech (TTS); an end-to-end model maps audio to audio inside one network, sometimes producing text tokens in parallel as an internal guide.

Cascades remain common in production because each module can be tested, replaced and audited separately. End-to-end models win when the application needs natural prosody, interruptions and sub-second replies, because no information is lost at the text bottleneck and no module waits for another to finish.

PropertyCascaded ASR, LLM and TTSEnd-to-end speech-to-speech
Typical latencyAbout 1 to 3 secondsAbout 0.2 to 0.8 seconds
Emotion and prosodyLost in transcriptionPreserved and reproducible
InterruptionsHandled by external logicCan be modeled natively in full duplex
DebuggingInspectable text at each stepHarder, internal states are opaque
Training data needReuses mature text and ASR corporaRequires large paired or conversational audio
Tool use and reasoningStrong, inherits text LLM abilityOften weaker unless text reasoning is retained
Comparison of speech-to-speech architectures

Hybrid designs are frequent: an audio language model listens directly to speech but generates text first and then speech, or it emits text and audio tokens in an interleaved stream. This keeps much of the reasoning quality of text models while reducing latency.

Main types of speech-to-speech tasks

The term covers several task families that share the audio-in, audio-out interface but differ in what must be preserved. Conversational models change the content and answer; translation models change the language but keep the meaning; voice conversion keeps the words and changes the speaker.

TaskWhat changesWhat must stayTypical use
Spoken dialogueContent of the replyCoherent context and turn-takingVoice assistants, call centers
Speech-to-speech translationLanguageMeaning, often speaker voiceLive interpretation, dubbing
Voice conversionSpeaker identityWords and timingAccessibility, media production
Speech enhancementNoise and reverberationSpeaker and contentHearing aids, conferencing
Accent or style conversionAccent, emotion, speaking rateWords and identityTraining, localization
Speech-to-speech task families

Related Terms in multimodal AI include the audio tokenizer, the audio language model and the any-to-any model, which extends the same principle to images and video as well as speech.

Latency, quality and evaluation

The quality of a speech-to-speech model is judged on what it says, how it sounds and how fast it responds. Human conversation tolerates gaps of only about 200 to 500 milliseconds between turns, so voice systems aim for a time to first audio in that range. Streaming decoders, small codec frame rates and speculative generation are the main tools for meeting that target.

Evaluation combines automatic and human measures, because no single score captures conversational quality.

MeasureWhat it capturesNotes
Time to first audioResponsivenessMeasured from end of user speech to first output sample
Word error rate of outputIntelligibilityOutput is transcribed by an ASR system and scored
Mean opinion score (MOS)Perceived naturalnessListeners rate on a 1 to 5 scale
Speaker similarityVoice preservationCosine similarity of speaker embeddings
BLEU or ASR-BLEUTranslation accuracyUsed for speech-to-speech translation
Common evaluation measures

Limitations and risks

Speech-to-speech models inherit the weaknesses of language models and add audio-specific risks. The most important ones concern factual accuracy, misuse of synthetic voices and uneven performance across speakers.

  • Hallucination in speech is harder to notice, because listeners cannot skim or re-read an answer.
  • Voice cloning misuse enables impersonation fraud, so responsible systems restrict custom voices and add audio watermarking.
  • Accent and dialect bias appears when training audio overrepresents certain speaker groups.
  • Weaker reasoning can occur when a model is trained mostly on audio and loses text-model capability.
  • Privacy exposure grows because raw voice recordings carry biometric and emotional information.
  • Background noise and overlapping speakers still degrade recognition and turn-taking.

A common misconception is that every voice assistant is a speech-to-speech model in the end-to-end sense. Many products still use cascaded pipelines with fast components, which can be adequate for task-oriented dialogue but sound less natural in open conversation.

Put simply

A speech-to-speech model is a computer system you can simply talk to and that talks back with a voice. Older voice systems first wrote down what you said, then thought about the written words, then read an answer aloud, a bit like passing notes through three different people.

A speech-to-speech model does the whole job in one go. Because it hears your voice directly, it can notice whether you sound rushed or unsure, and it can answer faster and with a more natural tone. The key point is that it works with sound itself, not only with words.

In practice

A regional hospital runs a phone line where patients of many language backgrounds book appointments and ask about preparation for tests. The staff deploy a speech-to-speech translation model between the caller and a receptionist: the patient speaks Spanish or Vietnamese, the receptionist hears English in a voice that keeps the caller’s tone, and the reply is spoken back in the caller’s language.

The system streams audio in chunks of about 80 milliseconds and reaches a time to first audio of roughly half a second, short enough that both sides speak almost normally. Calls that previously waited for a human interpreter are now handled directly, while medical consent conversations are still routed to certified interpreters. The result is shorter call queues and fewer booking errors, with human review kept where accuracy is legally critical.