A customer calls to check whether their insurance claim was approved. The AI agent has the answer in 500 milliseconds. It searches the policy, pulls up the claim, and confirms the status. Total processing time: under a second. But between the customer finishing their sentence and hearing a response, two and a half seconds pass.

Voice AI is built on a stack of three technologies.

• Speech-to-text (STT), or transcription, models convert the customer's audio into text.

• A large language model (LLM) processes that text, interprets the intent, and decides what to do.

• A text-to-speech (TTS) model converts the response back into audio.

Each step adds a few hundred milliseconds. They compound. This is voice AI latency. Not that the AI agent is slow at doing things, but that the pipeline between hearing and speaking has a lot of steps. And on a phone call, every extra beat of silence erodes trust.

We've spent a lot of time on this at Lorikeet. We think about it as two distinct problems: making things actually faster, and making them feel faster. Both matter but they require different approaches.

The inversion

Human call center agents have the opposite latency profile to AI agents. A human can respond instantly. They hear the question, they start talking. But taking action is slow. They navigate screens, click through tabs, put you on hold while they look something up while hold music plays in the background.

AI agents invert this. Taking action is fast. We can search a knowledge base, call an API, look up account details in milliseconds. But generating a spoken response has overhead, such as transcription and model inference. Each step in the pipeline takes time, and you can't skip any of them.

This inversion is an advantage. The things that make human support slow (looking up information, checking policies, taking actions) are the things AI does instantly. But you only unlock that advantage if you can get the response pipeline fast enough that the customer doesn't notice the mechanical overhead.

Optimizing the actual latency

Stream everything

STT streams words as the customer speaks. The large language model streams tokens as it generates. TTS streams audio as tokens arrive. The total time to first audible word is driven by time-to-first-token at each stage, not by waiting for any stage to fully complete. There is a lot of engineering work, but it compresses response time by hundreds of milliseconds.

Prefetch APIs aggressively

Any read-only data, such as customer account details, open orders, recent interactions, gets fetched at the start of the phone call, before the customer even asks their first question. When they do ask, the context is already loaded.

Over-provision for voice

Chat can queue. Email can buffer. Voice can't. We run voice instances with significant headroom because a traffic spike that causes even modest latency degradation is noticeable on a live call. If the system is slow for three seconds on a phone call, the customer knows.

Fail fast, recover faster

Every component in the stack has a timeout and error recovery path. For LLMs, that means hedging and soft failover to alternate providers. For speech-to-text and text-to-speech, it means connection timeouts that trigger recovery before dead air reaches the customer. The goal is to bound the maximum delay any single component failure can introduce to a few seconds.

Right-size the model

Voice is more latency-sensitive than email or chat. A customer reading a chat response doesn't notice whether it took 800ms or 1,500ms to generate but a customer on the phone notices. We use faster models with lower reasoning overhead for voice, and reserve heavier models for channels where latency tolerance is higher.

Hedge slow requests

Model inference times are variable. A request that usually takes 300ms might occasionally take 1,200ms. We cap the tail latency by hedging: if a request hasn't returned within a threshold, we fire a duplicate request and race them. Whichever returns first wins, and the loser is aborted. We'd rather pay for a redundant request than make the customer wait for an outlier.

Track inference provider health in real time

We use multiple inference providers, and any one of them can degrade without warning. The naive approach is to send a request, watch it fail after a few seconds, then failover. That's seconds wasted on a voice call. Instead, we track each provider's recent error rate and probabilistically route away from unhealthy ones.

It's not a hard cutoff. If a provider recovers, traffic naturally flows back. This avoids wasting time on requests that are likely to fail, without manual intervention.

Start before the customer finishes

The biggest latency win isn't in the pipeline at all. It's starting work before you're certain the customer has finished speaking. When our speech-to-text model gives moderate confidence that the turn is over, we begin processing speculatively: searching the knowledge base, looking up account details, generating a response. If the customer keeps talking, we discard the work and start fresh. If they were done, we've already finished the expensive work. In practice, the lower-confidence early trigger wins about 55% of the time, and when it wins, it shaves 200ms off the response. The constraint that makes this safe: we only take non-destructive actions during speculative execution. We'll search a knowledge base, but we won't issue a refund or cancel a subscription.

Hiding the perceived latency

Some latency can't be eliminated. A third-party API that takes eight seconds to return a result takes eight seconds. You can't optimize that away. What you can do is make sure the customer doesn't experience it as eight seconds of silence.

Fill the silence

A subtle background presence reduces the perception that the voice AI agent has frozen. The kind of ambient audio cue that signals someone is still on the line resolves a specific anxiety: did the call drop? Even a small audio signal makes the difference.

Acknowledge fast, but carefully

When the customer asks something that requires processing, the agent responds immediately with a short acknowledgment while work happens in the background. The key is that the acknowledgment has to be contextually appropriate. If a customer says "you charged me twice and I want my money back," the worst thing the agent can say is "absolutely, happy to help!" The right response is closer to "that sounds concerning, let me pull up your account and check."

This is harder than it looks. A naive implementation that optimizes purely for speed will produce acknowledgments that agree with whatever the customer said. That's fine when the customer says "can you check my order status" and dangerous when they say "you scammed me!"

Provide progress updates

If an action takes more than a few seconds, silence is not acceptable. The agent should provide periodic, contextual updates: "I'm still looking into that" or "just pulling up your claim details now." The updates should reflect what's actually happening, not be generic filler. And there's a rhythm to it. A brief pause before the update sounds natural, but too long and you get dead air.

Why this matters

The phone calls Lorikeet handles are often complex. They're calling about insurance claims, financial transactions, healthcare questions. These are conversations where trust is established in the first few seconds, and lost just as quickly.

That customer checking on their insurance claim? The AI agent had the answer in 500 milliseconds. The question is whether the two seconds of overhead made them feel like nobody was there.