Turn detection

Also known as: end-of-turn detection · endpointing

On this page 7

Turn detection is the process by which a spoken dialogue system decides that the user has finished speaking and that the system may now take its turn. In Speech & language AI, it is also called end-of-turn detection or endpointing, and it sits between audio capture, speech recognition and response generation in every voice assistant, phone agent and real-time conversational model.

The task sounds simple, yet it is one of the main factors that make a voice agent feel natural or awkward. A detector that fires too early interrupts users mid-thought; one that waits too long produces dead air that makes the system seem slow. Good turn detection balances these two errors using acoustic, prosodic and linguistic cues, often within a budget of a few hundred milliseconds.

How turn detection decides that a speaker is done

Turn detection works by continuously estimating the probability that the current turn has ended and triggering a response when that probability crosses a threshold. The simplest systems rely on silence duration alone: once voice activity detection reports no speech for a fixed interval, the turn is declared over. More capable systems combine several signals so that a short pause after a complete sentence ends the turn, while a longer hesitation in the middle of a phrase does not.

The typical processing chain runs on short audio frames of 10 to 30 milliseconds. Each frame is classified as speech or non-speech, a streaming recognizer produces partial transcripts, and a turn model scores whether the utterance so far looks complete. The decision is then passed to dialogue management, which starts generating and speaking the reply.

The following table lists the main signals that a turn detector can use and what each contributes.

SignalWhat it measuresStrengthWeakness
Silence durationTime since the last voiced frameCheap and robustCannot tell a pause from an ending
ProsodyFalling pitch, lengthened final syllables, energy decayCaptures natural ending cuesVaries by language and speaker
Syntactic completenessWhether the partial transcript forms a complete clauseSeparates hesitation from completionDepends on transcript quality
Semantic completenessWhether the request contains all needed informationHandles lists and spelled itemsRequires a language model and adds latency
Dialogue contextWhat the system just askedExpects short answers to yes or no questionsNeeds integration with dialogue state
Signals used for end-of-turn decisions

Approaches to endpointing

Endpointing approaches range from fixed silence timeouts to learned turn-taking models that predict the end of a turn before the silence even begins. The choice depends on latency targets, available compute and how varied the user population is.

Modern voice stacks increasingly use a small transformer-based turn classifier that reads the streaming transcript, sometimes together with audio features, and outputs an end-of-turn probability for each new word. Fully duplex speech models go further and model listening and speaking jointly, so turn-taking becomes an emergent behavior rather than a separate module.

ApproachTypical decision delayBest suited forMain limitation
Fixed silence threshold500 to 1,000 ms after speech stopsSimple command interfacesCuts off slow speakers or waits too long
Adaptive silence threshold200 to 800 ms, varied by contextIVR systems with known promptsStill blind to sentence meaning
Acoustic and prosodic model150 to 500 msLanguages with clear final intonationWeak on flat or noisy speech
Text-based turn classifier100 to 400 ms plus recognizer delayLLM voice agentsErrors in partial transcripts propagate
Duplex speech modelNear zero, predictiveReal-time conversational assistantsHigh compute cost and harder to control
Main families of turn detection methods

The delay values are indicative ranges, not fixed figures: they depend on frame size, recognizer latency, network round trips and how conservative the threshold is set.

Turn detection is often confused with voice activity detection, but the two answer different questions. Voice activity detection asks whether there is speech in this frame; turn detection asks whether the speaker intends to yield the floor. A user who pauses to think produces silence without ending the turn.

ComponentQuestion it answersRelation to turn detection
Voice activity detectionIs someone speaking right nowSupplies the raw speech and silence signal
Barge-inHas the user started talking over the systemThe reverse case, detecting a new user turn during system speech
Speaker diarizationWho spoke whenLabels turns after the fact, usually offline
Speech-to-textWhat words were saidProvides partial transcripts for completeness checks
Dialogue managementWhat the system should do nextConsumes the end-of-turn decision
How turn detection relates to neighboring components

In practice these components are tuned together. Aggressive endpointing works acceptably only when barge-in handling is reliable, because the user can then correct a premature reply by simply continuing to speak.

Latency, accuracy and how to measure them

Turn detection quality is measured as a trade-off between response latency and false cutoffs. Lowering the threshold makes the agent faster but raises the rate of interruptions; raising it reduces interruptions but adds silence before every reply.

  • End-of-turn latency: the time from the true end of speech to the moment the system commits to responding.
  • False cutoff rate: the share of turns ended while the user still intended to continue.
  • Missed or late endpoint rate: turns where the system waited well beyond a natural response gap.
  • Overlap and gap distribution: compared against human conversation, where typical gaps are around 200 ms.

Human speakers in conversation usually start responding within roughly 200 to 300 milliseconds, which is only possible because listeners predict the end of a turn before it arrives. A voice agent whose total response time, including recognition, generation and speech synthesis, exceeds about one second is commonly perceived as sluggish, so turn detection must consume only a small part of that budget.

Common failure modes and fixes

Most turn detection problems surface as users being cut off or waiting in silence, and each has a recognizable cause. The table maps frequent symptoms to their usual origin and a practical remedy.

SymptomLikely causeRemedy
Agent replies while users read out numbersSilence-only rule ends turn at digit pausesRaise threshold when expecting structured input
Long silence before every answerConservative fixed timeoutAdd a semantic completeness model
Interruptions on hesitations such as uh or umFiller words treated as complete clausesTrain the classifier on disfluent speech
Turns end on background noiseWeak voice activity detectionUse noise-robust VAD and echo cancellation
Poor results in a new languageProsodic model trained on one languageCollect language-specific turn data
Symptoms, causes and remedies in turn detection

A frequent design mistake is to tune the threshold on clean studio audio and then deploy over telephone lines, where codec artifacts, packet loss and background noise change the silence statistics substantially.

Understanding it simply

Turn detection is how a voice assistant knows it is its turn to talk. People do this without thinking: they hear a voice drop at the end of a sentence, notice that the question is complete and reply almost at once.

A machine has to guess. If it guesses too early, it talks over you while you are still thinking. If it guesses too late, there is an awkward pause. Good turn detection listens not only for silence but also for how a sentence sounds and whether it makes sense as a finished thought, much like a polite person waiting for you to finish.

Applied in practice

A health insurance provider runs a phone agent that collects policy numbers and appointment requests. Its first version ended a turn after 700 ms of silence. Callers reading a ten-digit policy number in groups of three or four were regularly cut off between groups, and roughly one call in five needed the number repeated.

The team added a text-based turn classifier that recognizes incomplete digit sequences and open-ended phrases, and it lengthened the silence window only while the agent was waiting for a policy number. For ordinary questions, the effective threshold dropped to about 300 ms. After the change, repeated number requests fell sharply and the median response gap on general questions shortened, so callers experienced the agent as both more patient and faster.