A speech-to-speech model is a neural network that accepts spoken audio as input and generates spoken audio as output, so a user can talk to it and hear an answer, a translation or a converted voice in return. In multimodal AI, the term usually refers to an end-to-end voice model that processes speech directly rather than chaining separate recognition, language and synthesis systems, which allows it to keep tone, emotion and timing that a text transcript would discard.
Speech-to-speech models power real-time voice assistants, simultaneous speech translation and voice conversion. Their defining engineering goals are low response latency, natural turn-taking and faithful handling of paralinguistic information such as emphasis, hesitation and laughter.
How a speech-to-speech model processes audio
A speech-to-speech model converts an incoming waveform into discrete or continuous audio representations, reasons over them, and decodes new representations back into a waveform. The typical pipeline has three stages: an audio encoder or audio tokenizer, a sequence model (often a transformer language model), and a neural vocoder or codec decoder that reconstructs sound.
Most current designs rely on neural audio codecs that compress speech into tokens at roughly 12 to 75 frames per second, using residual vector quantization with several codebooks per frame. Semantic tokens capture what is said, while acoustic tokens capture voice identity, pitch and recording conditions. The sequence model predicts the next tokens autoregressively, and the codec decoder turns them into audio in small chunks so that playback can start before the full answer is generated.
The table below lists the main components and what each one contributes.
| Component | Function | Typical design |
|---|---|---|
| Audio encoder | Turns waveform into features or tokens | Conformer or self-supervised speech encoder |
| Audio tokenizer | Discretizes speech into codec tokens | Residual vector quantization, 4 to 32 codebooks |
| Sequence model | Understands input and plans the response | Decoder-only transformer, often initialized from a text LLM |
| Codec decoder or vocoder | Reconstructs audible waveform | Streaming neural codec decoder or GAN vocoder |
| Turn-taking controller | Decides when to speak or stop | Voice activity detection or learned full-duplex modeling |
Cascaded pipelines versus end-to-end models
A speech-to-speech system can be cascaded or end-to-end, and the difference decides latency, expressiveness and controllability. A cascade links automatic speech recognition (ASR), a text language model and text-to-speech (TTS); an end-to-end model maps audio to audio inside one network, sometimes producing text tokens in parallel as an internal guide.
Cascades remain common in production because each module can be tested, replaced and audited separately. End-to-end models win when the application needs natural prosody, interruptions and sub-second replies, because no information is lost at the text bottleneck and no module waits for another to finish.
| Property | Cascaded ASR, LLM and TTS | End-to-end speech-to-speech |
|---|---|---|
| Typical latency | About 1 to 3 seconds | About 0.2 to 0.8 seconds |
| Emotion and prosody | Lost in transcription | Preserved and reproducible |
| Interruptions | Handled by external logic | Can be modeled natively in full duplex |
| Debugging | Inspectable text at each step | Harder, internal states are opaque |
| Training data need | Reuses mature text and ASR corpora | Requires large paired or conversational audio |
| Tool use and reasoning | Strong, inherits text LLM ability | Often weaker unless text reasoning is retained |
Hybrid designs are frequent: an audio language model listens directly to speech but generates text first and then speech, or it emits text and audio tokens in an interleaved stream. This keeps much of the reasoning quality of text models while reducing latency.
Main types of speech-to-speech tasks
The term covers several task families that share the audio-in, audio-out interface but differ in what must be preserved. Conversational models change the content and answer; translation models change the language but keep the meaning; voice conversion keeps the words and changes the speaker.
| Task | What changes | What must stay | Typical use |
|---|---|---|---|
| Spoken dialogue | Content of the reply | Coherent context and turn-taking | Voice assistants, call centers |
| Speech-to-speech translation | Language | Meaning, often speaker voice | Live interpretation, dubbing |
| Voice conversion | Speaker identity | Words and timing | Accessibility, media production |
| Speech enhancement | Noise and reverberation | Speaker and content | Hearing aids, conferencing |
| Accent or style conversion | Accent, emotion, speaking rate | Words and identity | Training, localization |
Related Terms in multimodal AI include the audio tokenizer, the audio language model and the any-to-any model, which extends the same principle to images and video as well as speech.
Latency, quality and evaluation
The quality of a speech-to-speech model is judged on what it says, how it sounds and how fast it responds. Human conversation tolerates gaps of only about 200 to 500 milliseconds between turns, so voice systems aim for a time to first audio in that range. Streaming decoders, small codec frame rates and speculative generation are the main tools for meeting that target.
Evaluation combines automatic and human measures, because no single score captures conversational quality.
| Measure | What it captures | Notes |
|---|---|---|
| Time to first audio | Responsiveness | Measured from end of user speech to first output sample |
| Word error rate of output | Intelligibility | Output is transcribed by an ASR system and scored |
| Mean opinion score (MOS) | Perceived naturalness | Listeners rate on a 1 to 5 scale |
| Speaker similarity | Voice preservation | Cosine similarity of speaker embeddings |
| BLEU or ASR-BLEU | Translation accuracy | Used for speech-to-speech translation |
Limitations and risks
Speech-to-speech models inherit the weaknesses of language models and add audio-specific risks. The most important ones concern factual accuracy, misuse of synthetic voices and uneven performance across speakers.
- Hallucination in speech is harder to notice, because listeners cannot skim or re-read an answer.
- Voice cloning misuse enables impersonation fraud, so responsible systems restrict custom voices and add audio watermarking.
- Accent and dialect bias appears when training audio overrepresents certain speaker groups.
- Weaker reasoning can occur when a model is trained mostly on audio and loses text-model capability.
- Privacy exposure grows because raw voice recordings carry biometric and emotional information.
- Background noise and overlapping speakers still degrade recognition and turn-taking.
A common misconception is that every voice assistant is a speech-to-speech model in the end-to-end sense. Many products still use cascaded pipelines with fast components, which can be adequate for task-oriented dialogue but sound less natural in open conversation.
Put simply
A speech-to-speech model is a computer system you can simply talk to and that talks back with a voice. Older voice systems first wrote down what you said, then thought about the written words, then read an answer aloud, a bit like passing notes through three different people.
A speech-to-speech model does the whole job in one go. Because it hears your voice directly, it can notice whether you sound rushed or unsure, and it can answer faster and with a more natural tone. The key point is that it works with sound itself, not only with words.
In practice
A regional hospital runs a phone line where patients of many language backgrounds book appointments and ask about preparation for tests. The staff deploy a speech-to-speech translation model between the caller and a receptionist: the patient speaks Spanish or Vietnamese, the receptionist hears English in a voice that keeps the caller’s tone, and the reply is spoken back in the caller’s language.
The system streams audio in chunks of about 80 milliseconds and reaches a time to first audio of roughly half a second, short enough that both sides speak almost normally. Calls that previously waited for a human interpreter are now handled directly, while medical consent conversations are still routed to certified interpreters. The result is shorter call queues and fewer booking errors, with human review kept where accuracy is legally critical.