Why your AI voice agent sounds slow to reply, and how to fix it

This page may contain affiliate links from which Blur Studio earns commissions.

A voice agent that pauses a beat too long before replying breaks the illusion of a real conversation, and callers notice immediately, talking over the agent, hanging up, or assuming the call dropped. The instinct is to blame the voice itself, but that's often the wrong target. A phone call has three separate stages that each add delay, and text to speech, the step that turns the agent's decided reply into audio, is usually the fastest of the three if you've picked the right model. For the basics of how these agents work and what they cost, see AI voice agent for business.

The three places delay actually comes from

• The phone line itself, the connection between the caller and whatever system is answering, which has its own fixed latency depending on your telephony provider and setup.
• The reasoning model, the AI deciding what to actually say in response to what the caller just said, understanding the request, checking any data it needs, and composing a reply. This step is frequently the slowest, especially if the agent is looking something up or running a more complex reasoning model.
• Text to speech, converting that decided reply into spoken audio. This is the step ElevenLabs covers, and on its fastest model, it's genuinely quick.

Most people assume the whole delay sits in the third step, since that's the part they actually hear as "the agent talking." In practice, the first two steps combined are usually the bigger contributor.

Some real numbers help put this in perspective. ElevenLabs publishes latency figures for its own conversational agent platform in the range of roughly 680 milliseconds at the median (P50) and up to around 1,560 milliseconds at the 95th percentile (P95), meaning the slowest one call in twenty takes noticeably longer than a typical call. That full figure covers the entire round trip, not just the text to speech step, which on Flash v2.5 alone is roughly 75 milliseconds. Do that subtraction and the gap is obvious: even in a well-tuned setup, something other than voice generation is eating the majority of that time, which is exactly why fixating on the voice model alone rarely closes the gap between "fast" and "feels fast" to an actual caller.

How to actually diagnose this

1. Time each stage separately if your platform logs this, or test with a simplified setup if it doesn't: how long between the caller finishing speaking and the reasoning step producing a reply, versus how long between that reply being ready and audio actually playing.
2. Confirm which text to speech model you're actually using. If your agent is still configured to use the older Turbo v2.5 model, know that ElevenLabs deprecated it in July 2026 in favor of Flash v2.5, which it says matches Turbo's voice quality at lower latency. If you haven't explicitly switched, check whether your platform auto-migrated you or whether you're still pointed at a model that's no longer the fastest available option.
3. Switch to Flash v2.5 specifically if speed is the priority and you haven't already. At roughly 75 milliseconds, it's among the fastest text to speech options available and the one ElevenLabs recommends for exactly this use case. A wider look at speed-focused voice APIs is in ElevenLabs vs Cartesia vs Fish Audio.
4. If the delay persists after confirming the fast model is in use, the bottleneck is almost certainly in the reasoning step or the phone line itself, not the voice. That's where to focus next, simplifying what the reasoning model needs to check before it can reply, or looking at your telephony provider's own latency.
5. If the reasoning step itself is the bottleneck, check whether your platform supports ElevenLabs' Fast LLM service tier, added 26 October 2026 per ElevenLabs' own changelog, which offers higher throughput and lower latency for supported reasoning models at a premium per-token price. It's new enough that most existing agent setups won't have switched to it yet, and it's worth a look before you restructure the reasoning logic itself.

Streaming vs waiting for the full reply

One architectural detail that makes a bigger practical difference than most people expect: whether the system streams audio as it's generated, or waits for the entire reasoning step to finish composing a full reply before text to speech even starts. A streaming setup can start speaking the first part of a sentence while the reasoning model is still finishing the rest of it, which shaves a meaningful chunk off the perceived delay even if the total processing time is identical. A system that waits for the complete reply before generating any audio always feels slower to a caller, even when the raw numbers say otherwise, because the caller experiences the full wait with no audio feedback at all until everything's ready. If your platform supports a streaming mode and you're not using it, that's often a bigger win than switching voice models.

Also worth checking if your agent uses multiple tools or lookups per turn (checking a calendar, then checking a database, then composing a reply): whether those calls happen one after another or at the same time. ElevenLabs enabled parallel tool calls for agents on 21 September 2026, letting an agent fire off more than one tool call in a single turn instead of waiting for each to finish before starting the next. If your agent's logic still runs tool calls sequentially out of habit rather than necessity, and several of them don't actually depend on each other's results, switching to parallel calls can cut real seconds off a multi-step turn without touching the voice model at all.

My honest take for someone building a phone agent

I'd flag this clearly: swapping the voice model is the easy fix and worth doing first, but it's rarely the whole fix if a system genuinely feels sluggish. If you've already switched to the fastest available text to speech model and the agent still feels slow, don't keep tuning the voice, the problem has moved elsewhere, usually into how much the reasoning layer has to do before it can respond, or a phone line with its own latency you don't control directly. Spending more time on voice settings at that point doesn't move the needle.

A realistic example of how this plays out: a home services business sets up a voice agent to answer after-hours calls and book appointments. It sounds fine for simple requests ("what are your hours") but noticeably slow whenever a caller asks to book a specific appointment, since that turn requires checking calendar availability, confirming the caller's address is in the service area, and then composing a response, three separate lookups. If those three happen one after another, the caller hears a long silence before any reply. Running the availability check and the service-area check in parallel, since neither depends on the other's result, and only composing the final reply once both are back, can cut a multi-second wait down to whichever single check takes longest, not the sum of all three. That's a reasoning-layer and tool-call fix, not a voice model fix, and it's the kind of change that actually closes the gap between "technically fast" and "feels fast on the phone."

One thing worth knowing: a small amount of natural pause before a reply isn't necessarily bad, real human conversation has brief pauses too. The problem is specifically when the pause is long enough or inconsistent enough that callers start talking over the agent or assume the call dropped, which is a noticeably longer delay than natural conversational timing.

When this needs a proper fix, not a setting change

If you've confirmed the text to speech step isn't the bottleneck and the real issue is in how the reasoning or lookup logic is structured, that's an architecture problem, not something you fix by changing a dropdown. I build voice agent systems for clients where exactly this kind of latency tuning, deciding what the agent needs to check before replying, how much of that can happen in parallel versus in sequence, is part of the build itself, not an afterthought bolted onto a slow system.

The practical starting point I'd use with a client in this situation is always the same: get real numbers before changing anything. Log how long each stage of a handful of real calls actually took, rather than guessing from how the call felt, since "it feels slow" and "the reasoning step is taking 2.5 seconds" point to very different fixes, and you only find out which one you're dealing with by measuring it. Callers who still get stuck need a way out, which is covered in how to transfer an AI phone call to a human.