Turn detection is the process by which a spoken dialogue system decides that the user has finished speaking and that the system may now take its turn. In Speech & language AI, it is also called end-of-turn detection or endpointing, and it sits between audio capture, speech recognition and response generation in every voice assistant, phone agent and real-time conversational model.
The task sounds simple, yet it is one of the main factors that make a voice agent feel natural or awkward. A detector that fires too early interrupts users mid-thought; one that waits too long produces dead air that makes the system seem slow. Good turn detection balances these two errors using acoustic, prosodic and linguistic cues, often within a budget of a few hundred milliseconds.
How turn detection decides that a speaker is done
Turn detection works by continuously estimating the probability that the current turn has ended and triggering a response when that probability crosses a threshold. The simplest systems rely on silence duration alone: once voice activity detection reports no speech for a fixed interval, the turn is declared over. More capable systems combine several signals so that a short pause after a complete sentence ends the turn, while a longer hesitation in the middle of a phrase does not.
The typical processing chain runs on short audio frames of 10 to 30 milliseconds. Each frame is classified as speech or non-speech, a streaming recognizer produces partial transcripts, and a turn model scores whether the utterance so far looks complete. The decision is then passed to dialogue management, which starts generating and speaking the reply.
The following table lists the main signals that a turn detector can use and what each contributes.
| Signal | What it measures | Strength | Weakness |
|---|---|---|---|
| Silence duration | Time since the last voiced frame | Cheap and robust | Cannot tell a pause from an ending |
| Prosody | Falling pitch, lengthened final syllables, energy decay | Captures natural ending cues | Varies by language and speaker |
| Syntactic completeness | Whether the partial transcript forms a complete clause | Separates hesitation from completion | Depends on transcript quality |
| Semantic completeness | Whether the request contains all needed information | Handles lists and spelled items | Requires a language model and adds latency |
| Dialogue context | What the system just asked | Expects short answers to yes or no questions | Needs integration with dialogue state |
Approaches to endpointing
Endpointing approaches range from fixed silence timeouts to learned turn-taking models that predict the end of a turn before the silence even begins. The choice depends on latency targets, available compute and how varied the user population is.
Modern voice stacks increasingly use a small transformer-based turn classifier that reads the streaming transcript, sometimes together with audio features, and outputs an end-of-turn probability for each new word. Fully duplex speech models go further and model listening and speaking jointly, so turn-taking becomes an emergent behavior rather than a separate module.
| Approach | Typical decision delay | Best suited for | Main limitation |
|---|---|---|---|
| Fixed silence threshold | 500 to 1,000 ms after speech stops | Simple command interfaces | Cuts off slow speakers or waits too long |
| Adaptive silence threshold | 200 to 800 ms, varied by context | IVR systems with known prompts | Still blind to sentence meaning |
| Acoustic and prosodic model | 150 to 500 ms | Languages with clear final intonation | Weak on flat or noisy speech |
| Text-based turn classifier | 100 to 400 ms plus recognizer delay | LLM voice agents | Errors in partial transcripts propagate |
| Duplex speech model | Near zero, predictive | Real-time conversational assistants | High compute cost and harder to control |
The delay values are indicative ranges, not fixed figures: they depend on frame size, recognizer latency, network round trips and how conservative the threshold is set.
Turn detection compared with related tasks
Turn detection is often confused with voice activity detection, but the two answer different questions. Voice activity detection asks whether there is speech in this frame; turn detection asks whether the speaker intends to yield the floor. A user who pauses to think produces silence without ending the turn.
| Component | Question it answers | Relation to turn detection |
|---|---|---|
| Voice activity detection | Is someone speaking right now | Supplies the raw speech and silence signal |
| Barge-in | Has the user started talking over the system | The reverse case, detecting a new user turn during system speech |
| Speaker diarization | Who spoke when | Labels turns after the fact, usually offline |
| Speech-to-text | What words were said | Provides partial transcripts for completeness checks |
| Dialogue management | What the system should do next | Consumes the end-of-turn decision |
In practice these components are tuned together. Aggressive endpointing works acceptably only when barge-in handling is reliable, because the user can then correct a premature reply by simply continuing to speak.
Latency, accuracy and how to measure them
Turn detection quality is measured as a trade-off between response latency and false cutoffs. Lowering the threshold makes the agent faster but raises the rate of interruptions; raising it reduces interruptions but adds silence before every reply.
- End-of-turn latency: the time from the true end of speech to the moment the system commits to responding.
- False cutoff rate: the share of turns ended while the user still intended to continue.
- Missed or late endpoint rate: turns where the system waited well beyond a natural response gap.
- Overlap and gap distribution: compared against human conversation, where typical gaps are around 200 ms.
Human speakers in conversation usually start responding within roughly 200 to 300 milliseconds, which is only possible because listeners predict the end of a turn before it arrives. A voice agent whose total response time, including recognition, generation and speech synthesis, exceeds about one second is commonly perceived as sluggish, so turn detection must consume only a small part of that budget.
Common failure modes and fixes
Most turn detection problems surface as users being cut off or waiting in silence, and each has a recognizable cause. The table maps frequent symptoms to their usual origin and a practical remedy.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Agent replies while users read out numbers | Silence-only rule ends turn at digit pauses | Raise threshold when expecting structured input |
| Long silence before every answer | Conservative fixed timeout | Add a semantic completeness model |
| Interruptions on hesitations such as uh or um | Filler words treated as complete clauses | Train the classifier on disfluent speech |
| Turns end on background noise | Weak voice activity detection | Use noise-robust VAD and echo cancellation |
| Poor results in a new language | Prosodic model trained on one language | Collect language-specific turn data |
A frequent design mistake is to tune the threshold on clean studio audio and then deploy over telephone lines, where codec artifacts, packet loss and background noise change the silence statistics substantially.
Understanding it simply
Turn detection is how a voice assistant knows it is its turn to talk. People do this without thinking: they hear a voice drop at the end of a sentence, notice that the question is complete and reply almost at once.
A machine has to guess. If it guesses too early, it talks over you while you are still thinking. If it guesses too late, there is an awkward pause. Good turn detection listens not only for silence but also for how a sentence sounds and whether it makes sense as a finished thought, much like a polite person waiting for you to finish.
Applied in practice
A health insurance provider runs a phone agent that collects policy numbers and appointment requests. Its first version ended a turn after 700 ms of silence. Callers reading a ten-digit policy number in groups of three or four were regularly cut off between groups, and roughly one call in five needed the number repeated.
The team added a text-based turn classifier that recognizes incomplete digit sequences and open-ended phrases, and it lengthened the silence window only while the agent was waiting for a policy number. For ordinary questions, the effective threshold dropped to about 300 ms. After the change, repeated number requests fell sharply and the median response gap on general questions shortened, so callers experienced the agent as both more patient and faster.