It's a Wednesday morning. A customer calls about a home insurance claim. She speaks carefully, pausing between thoughts to find the right words. Mid-sentence, the AI agent jumps in before she's finished. She tries to correct it. The agent tries to correct itself. Both stop. Both start. The conversation is broken before it began.

This is the turn-taking problem. In chat and email, turns are explicit: the customer types a message and hits send. In voice, there's no send button. There's a continuous stream of audio, and something has to decide, in real time, whether a pause means "I'm done" or "I'm thinking."

We've spent a lot of time on this at Lorikeet. Here's what we've found works.

Silence alone is not enough

The most basic approach is Voice Activity Detection, or VAD. It monitors each 10–30ms chunk of audio and classifies it as speech or silence. After continuous silence, say 500ms, VAD decides the customer has probably finished speaking.

This works for a specific demographic: younger, faster speakers who pause briefly between thoughts. It falls apart across a wider population. Some people pause for a full second mid-sentence while they gather their next thought. Non-native speakers take longer to formulate phrasing. A fixed silence threshold treats all of these pauses as turn boundaries, and the agent interrupts.

You can raise the threshold. Wait 800ms or a full second. But now every exchange carries that delay, including the ones where the customer finished in 200ms. Now the conversation feels sluggish for everyone to avoid interrupting some people.

Listen to what was said, not just that it stopped

The next layer is semantic endpointing, or end of turn detection. Instead of relying on silence alone, analyze the content of what the customer said to estimate whether they're likely done.

If someone says "I need to cancel my subscription because…" and pauses, they're almost certainly not finished. "Because" signals an incomplete thought. If someone says "Can you check the status of my order?" and pauses, they're likely done.

A lightweight model evaluates the transcript in real time and produces a confidence score. High confidence the turn is complete means we shorten the silence wait. Low confidence means we extend it.

It reduces interruptions because we give people more time when context suggests they're mid-thought. And it reduces latency because we act faster when context suggests they're done. A fixed silence threshold is a compromise that's too slow for some utterances and too fast for others. Semantic endpointing lets us be aggressive where it's safe and patient where it's not.

Start before you're sure

Even with semantic endpointing, there's a fundamental tradeoff. The longer you wait to be confident the turn is over, the fewer interruptions but the more latency. The shorter you wait, the faster you respond but the more you cut people off. Every voice AI agent sits somewhere on this curve. We decided to break out of it.

The insight is that waiting to be certain before doing anything is wasteful. If there's a 60% chance the customer is done, there's a 60% chance we're sitting idle when we could already be working. So we start processing speculatively and kick off an LLM to generate a response.

In practice: the customer says "I need to check whether my claim was approved." Semantic detection gives moderate confidence the turn is complete, but maybe they're about to add "…for the water damage from last week." Rather than waiting, we begin processing immediately. We search the knowledge base for claim status, look up their open claims, start generating a response.

If the customer was done, we've shaved hundreds of milliseconds off the response time. The expensive work is already complete. Our target is a response within roughly a second, though telephony overhead can add up to half a second on top of that. If they keep talking, we discard the speculative work and start fresh with the complete utterance.

The constraint that makes this safe: we only take non-destructive actions during speculative execution. We'll search a knowledge base and look up account information. We won't issue a refund, cancel a subscription, or make any irreversible change. Because our system is resilient to being wrong about turn boundaries, we can be much more aggressive with detection thresholds. We don't need 95% confidence to start working. We can start at 40% or 50%, knowing that the downside of a wrong guess is wasted compute. In practice, the lower-confidence early trigger wins about 55% of the time, and when it wins, it shaves 200ms off the response.

Not every sound is an interruption

There's a conversational pattern that trips up most voice agents: back-channeling. The agent is explaining something and the customer says "yeah," "okay," "mm-hmm." That's not an interruption. They're signaling they're following along and want the agent to keep talking.

A naive system treats any detected speech as a potential interruption. The agent stops mid-sentence, tries to process "yeah" as a new turn, generates a confused response, then restarts its original explanation. The conversation stops and starts and stops and starts.

We handle this with back-channel detection that uses semantic understanding of the customer's words in context. If the agent is mid-utterance and the customer produces a short affirmative, and the agent's speech isn't at a natural stopping point, we classify it as back-channel and continue. The classification considers both what was said and when it was said relative to the agent's speech.

Break the interruption loop

The final layer handles a failure mode that's subtle but immediately recognizable when it happens: mutual interruption loops.

The agent starts responding at the same time the customer starts a new thought. Both detect the other's speech and stop. A beat of silence. Both start again. Both stop. This can repeat three, four, five times.

We call this floor contention. When the system identifies a pattern of mutual interruption, it deliberately holds. The agent yields the floor and waits for the customer to speak, breaking the interruption loop. It's the detail that determines whether a conversation feels like talking to a human or wrestling with a broken phone tree.

We built this because the customers who call Lorikeet-powered phone lines skew older. They speak more deliberately, pause more often, and take longer to formulate their thoughts. A voice AI agent that only works for fast-talking 25-year-olds is not a voice AI agent that works.