In a chat, a two-second pause is invisible. On a phone call, it is the moment your customer says "hello? are you still there?" and starts to lose trust. Voice AI lives or dies on latency, and most teams measure the wrong number.
Latency in voice AI is the time between a caller finishing their sentence and the agent starting to speak back. Human conversation runs on a tight clock: people expect a reply within roughly 200 milliseconds of turn-end, and gaps longer than about a second read as hesitation, confusion, or a dropped line. For an AI support agent that has to transcribe speech, reason over a customer account, and synthesize a spoken reply, hitting that window is the core engineering problem of the channel.
The number that matters is utterance-end to first audio, not average round-trip time. A caller hears the pause before your first word, not your median.
The large language model call is almost always the dominant cost in the pipeline, so optimizing the model step matters more than shaving milliseconds off transcription or audio.
Perceived latency and measured latency are different problems. Streaming the first words, backchannels, and natural disfluencies buy you time the raw numbers do not.
Latency trades off against cost, answer quality, and reliability. A guide to evaluating voice AI has to treat all four together, not latency in isolation.
Lorikeet runs production voice at roughly sub-1-second time to first sound, with an end-to-end p50 near 1.4 seconds, by keeping voice on a single workflow engine and optimizing the model call first.
Last updated: June 2026
Most buyers evaluate voice AI the way they evaluate chat: resolution rate, integrations, price per interaction. Those matter, but they miss the property that makes voice feel human or robotic. A chat agent can take six seconds to compose a careful answer and the customer barely notices, because the typing indicator covers the gap and reading is asynchronous. A voice agent cannot. Speech is real-time and unforgiving. If the agent pauses too long before answering, the caller talks over it, repeats themselves, or assumes the system is broken. This guide explains why sub-second response is the make-or-break metric for voice support, what actually drives latency inside the pipeline, how to evaluate a vendor honestly, and the trade-offs you accept when you push for speed.
Why Sub-Second Response Is Critical for Natural Voice
The reason latency dominates voice and not chat comes down to how human conversation is wired. Linguists who study turn-taking find that the average gap between one speaker finishing and the next starting is around 200 milliseconds across languages. That is faster than human reaction time to a simple stimulus, which means people are predicting when a turn will end and preparing their reply before the other person stops talking. A delay outside that rhythm is immediately noticeable. Three specific dynamics make this unforgiving for an AI agent.
Turn-Taking Has a Tight Clock
When a caller finishes a question and the agent does not begin speaking within roughly a second, the silence becomes information. The caller reads it as the agent being confused, not understanding, or having disconnected. In a human call center, agents fill that gap with backchannels ("mm-hm", "let me check that"), which is exactly why a voice AI that produces dead air feels worse than a human who is equally slow. The clock is not about total response time. It is about how long the line is silent after the caller stops.
Interruptions Have to Be Handled in Real Time
Real calls are not clean turn-by-turn exchanges. Callers interrupt, change their mind mid-sentence, and start talking again while the agent is still speaking. A natural voice agent has to detect that barge-in, stop its own speech almost instantly, and re-listen. If the system is busy generating a long response and cannot be interrupted, the customer ends up talking over an agent that ignores them, which is the single most frustrating failure mode in voice support. Low latency is a prerequisite for good interruption handling, because the agent has to be responsive enough to yield the floor.
Caller Patience Is Shorter on the Phone
A customer who called support is usually already mildly frustrated, has a specific problem, and is paying attention to the call in a way they do not pay attention to a chat window they left open in a tab. Patience for dead air is correspondingly thin. Every extra second of silence increases the odds the caller repeats the question (which resets the agent), asks for a human, or hangs up. The cost of latency in voice is not a worse score on a dashboard. It is abandoned calls and escalations that did not need to happen.
What Actually Drives Voice AI Latency
A voice agent is a pipeline, and total latency is the sum of every stage between the caller finishing their sentence and the first sound coming back. Understanding where the time goes is what separates a real evaluation from a vendor demo. The stages, in order, are roughly: detect that the caller has stopped talking, transcribe the speech to text, send that text to the language model and generate a reply, convert the reply text to speech, and stream the audio back. Each stage adds time, but they are not equal.
The Language Model Call Usually Dominates
In a well-built pipeline, speech-to-text and text-to-speech can each run in tens to low hundreds of milliseconds, especially with streaming transcription that starts working before the caller finishes. The step that dominates is the model call: the time the language model takes to read the conversation, reason over the customer's account and the relevant policy, and produce a reply. This is why the first rule of voice latency engineering is to optimize the model step before anything else. Shaving 50 milliseconds off audio synthesis is wasted effort if the model is taking two seconds to think.
Prompt Size and Agentic Loop Depth
Two things make the model step slow. The first is prompt size: a long system prompt, a large knowledge dump, and a full conversation history all take longer to process. The second, and often larger, is iteration depth. An agent that makes one model call and replies is fast. An agent that loops (call the model, call a tool, call the model again, call another tool) multiplies its latency by the number of round trips. Voice agents that try to do everything in one big reasoning loop are slow by construction. The faster pattern is a pipeline of small, focused model calls that can run on smaller, quicker models and can sometimes run in parallel.
Tool Calls and Waterfall API Lookups
If answering the caller requires looking up their account, checking a balance, or verifying identity, those tool calls add latency, and they add it on the critical path because the agent cannot answer until the data comes back. The worst pattern is a waterfall: call API A, wait, use its result to call API B, wait, then answer. Better-built agents prefetch likely-needed data (for example, resolving the caller's identity from their phone number before the conversation even starts) and parallelize independent lookups so the caller is not waiting through a chain of sequential requests.
The Text-to-Speech Handoff
One of the quietest sources of latency is the handoff between the model finishing (or starting to stream) its reply and the text-to-speech engine producing audio. If the system waits for the full reply text before it starts synthesizing speech, the caller waits through the entire generation plus the entire synthesis. The fix is eager streaming: feed the model's output to the speech engine token by token as it is produced, so the agent starts speaking the first words while it is still generating the rest. The choice of speech provider matters here too. Internally we see roughly half a second per turn of difference between providers, which is why the faster engine is the default for latency-sensitive deployments.
Provider Contention and the Tail
Average latency hides the problem. What customers feel is the tail: the slow calls, the p95. A common cause of tail latency is provider contention, where the model provider is busy and the request is slow or gets rerouted to a fallback. Mitigations include racing requests across multiple inference providers and isolating critical paths so one slow dependency does not stall the whole turn. A vendor that only quotes average latency is hiding the calls that actually annoy your customers.
How to Evaluate Voice AI Latency
Vendor latency claims are easy to game and hard to compare. The questions below are the ones that separate a marketing number from a production number.
Ask for Utterance-End to First Audio, at p50 and p95
The single most important metric is the time from when the caller stops speaking to when the agent starts speaking, because that is the silence the customer hears. Ask for it at both the median (p50) and the tail (p95 or p90), not as a single average. A vendor quoting "under two seconds" without specifying which number and which percentile is quoting the most flattering reading. As a rough bar, a strong voice agent targets utterance-end to first response under about two seconds at the tail, and the best deployments push the median under a second.
Separate Time to First Sound from Time to Full Answer
There are two useful latency numbers, and good systems report both. Time to first sound is how long until the agent starts talking, which governs the feel of the conversation. Time to full answer is how long until the complete response is delivered, which matters for the substance. A well-streamed agent can have a fast time to first sound (it starts with a natural opener while it finishes reasoning) and a longer time to full answer, and that is fine. A system that only reports one number is hiding the other.
Test on Your Hardest Calls, Not the Demo Script
Latency on a scripted demo where the agent answers a simple FAQ tells you nothing about latency on a real call that requires an account lookup, an identity check, and a multi-step resolution. Bring your actual call types. Ask the vendor to run a call that requires three or four tool calls and measure the latency on that, because that is where waterfall lookups and deep agentic loops show up. A demo that only shows knowledge-base answers is measuring the easy case.
Probe Interruption Handling Directly
Latency and barge-in are linked. On a test call, interrupt the agent mid-sentence and see how fast it stops and re-listens. A laggy agent will keep talking over you or take a long beat to recover. This is hard to fake in a recorded demo, so do it live. The responsiveness of interruption handling is a good proxy for the overall latency budget of the system.
Ask How Voice and Other Channels Relate
Many vendors run voice on a separate stack from chat and email, bolted together with a transcript handoff. That architecture tends to add latency at the seams and creates a second agent to maintain. Ask whether voice runs on the same workflow engine as the other channels and shares the same logic and memory. A single engine across channels is both a latency advantage and a consistency advantage, because there is no translation layer between a chat reply and a spoken one.
The Trade-Offs of Chasing Low Latency
Latency is not free to optimize, and a guide that pretends otherwise is selling something. Every technique that reduces latency trades against at least one of three other properties: cost, answer quality, and reliability. Treating these as a four-way trade-off is the honest way to evaluate the channel.
Latency Versus Quality
The fastest model is rarely the most capable. Using a smaller, quicker model for a turn cuts latency but can reduce the agent's reasoning quality on a hard ticket. The right answer is not one model for everything. It is a pipeline that uses small fast models for simple steps (classification, small talk, clarification) and routes the genuinely hard reasoning to a stronger model only when it is needed. Forcing every turn through a single large model is slow; forcing every turn through a single small model is wrong on the cases that matter.
Latency Versus Cost
Some latency reductions cost money. Racing the same request across multiple inference providers to beat the tail means paying for redundant calls. Prompt caching and prefetching add infrastructure. Running a faster premium speech provider can cost more than a slower one. None of these are wrong, but they are spend decisions, and a vendor should be able to explain which latency optimizations they pass on as cost and which they absorb.
Latency Versus Reliability and Safety
The riskiest trade is cutting safety checks to save time. In a regulated business, an agent's spoken output may need to pass guardrails before it reaches the caller. Naively running those checks on a streaming voice reply can truncate speech or produce dead air, so the temptation is to turn them off for speed. That is a real trade-off, and the right resolution is to design the checks for the voice context (for example, leaning on tone-shaping brand guidelines and post-call review rather than blocking mid-stream) rather than simply disabling them. Speed that comes from skipping the controls your compliance team requires is not speed, it is deferred risk.
Perceived Latency Is a Legitimate Lever
Not every fix is about making the pipeline literally faster. A large part of how fast a voice agent feels comes from masking the latency that exists. Starting the reply with a natural opener while the rest is still generating, using brief backchannels, allowing small disfluencies, and even a touch of ambient background noise so the line never sounds dead all make a given amount of latency feel shorter. These are not cheating. Human agents do exactly the same thing. The caveat is that perceived-latency tricks improve the experience but do not fix a pipeline that is genuinely slow on the calls that need real work done.
How Lorikeet Approaches Voice Latency
Lorikeet runs voice as a native channel on the same workflow engine as chat, email, and SMS, rather than as a separate bolted-on stack. In production that translates to roughly sub-1-second time to first sound and an end-to-end median near 1.4 seconds, with the tail kept under about two seconds. The approach follows the same priorities described above: optimize the model call first because it dominates, stream the model's output into the speech engine token by token so the agent starts talking before it finishes reasoning, default to the faster speech provider, and prefetch caller context (resolving identity from the phone number) so the first turn is not waiting on a lookup. An honest limitation: pushing latency this low means making deliberate trade-offs, including how output checks are run in the voice context, which is why Lorikeet leans on adversarial pre-launch simulation and post-call review to keep the safety bar high without adding mid-stream delay. The point is not that latency is solved. It is that latency is a budget you spend carefully against cost, quality, and reliability, with the model call as the first thing to fix.
Key Takeaways
Voice is unforgiving in a way chat is not, because human turn-taking runs on a roughly 200-millisecond clock and silence longer than about a second reads as a broken or confused agent.
The metric that matters is utterance-end to first audio, reported at p50 and p95, not a single average round-trip number.
The language model call almost always dominates the pipeline, so optimizing the model step, capping agentic loop depth, and parallelizing tool lookups beats micro-optimizing transcription or audio.
Latency trades off against cost, answer quality, and reliability, and perceived-latency techniques like eager streaming and backchannels are a legitimate part of the toolkit.
Lorikeet runs voice at roughly sub-1-second time to first sound with an end-to-end p50 near 1.4 seconds by keeping voice on one engine and optimizing the model call first.
Conclusion
Latency is the property that decides whether a voice agent feels like a conversation or a hold queue. It is also the metric most likely to be misrepresented, because averages hide tails and demo scripts hide the hard calls. If you are evaluating voice AI, anchor on utterance-end to first audio at the tail, test on the calls that require real work, probe interruption handling live, and ask how the vendor spends the latency-cost-quality-reliability budget. A voice agent that answers in under a second and keeps that speed on a four-tool-call resolution is doing something genuinely hard. See how Lorikeet runs sub-second voice on regulated workflows.
