Voice AI Agent Latency: What Callers Hear

Voice AI agent latency is the delay a caller hears between finishing a sentence and the start of the agent's reply. It spans voice activity detection, speech-to-text, LLM reasoning, text-to-speech and the phone network, so it is almost never the number on a speech model's spec sheet. When ElevenLabs shipped Eleven v4 on 28 September 2026, it quoted about 150 milliseconds median time to first sound for v4 Turbo. Independent measurements of real phone calls put ElevenLabs' full agent stack at 1,424 milliseconds median. Both numbers are accurate. They measure different things.

Key facts

Eleven v4 Turbo: about 150ms median time to first sound in a WebSocket test, of which roughly 100ms is model inference (Unite.AI).
Real phone calls, median time to first audio byte (TTFAB): Telnyx 1,296ms, ElevenLabs 1,424ms, Bland AI 1,520ms, Vapi 1,558ms, Retell AI 1,740ms (Openbenchmarks).
Daily's Pipecat team puts the budget for a natural-feeling conversation at under 1,500ms voice-to-voice, with around 700ms of that going to LLM time-to-first-token (Daily).
The fastest platform in the real-call benchmark is a telephony carrier, not a speech-model company.

What Eleven v4 actually improved

Eleven v4 is a real upgrade. TechCrunch reported support for 90 languages (up from 70 in v3), stacked expression tags for handling interruptions and escalations, and voice cloning from 10 seconds of reference audio. Within hours, Artificial Analysis' Provider Voice Arena ranked it first, with an Elo around 1,315 to 1,319 in blind comparisons against Cartesia Sonic 3.6 and Google's Gemini TTS.

The latency figure is the part that matters for phone agents. ElevenLabs' v4 page quotes a "150ms median time to first sound" for v4 Turbo. ElevenLabs never claimed that is what a caller experiences on a phone line, and it isn't.

Where the time goes on a phone call

A speech model benchmark times one step: turning text into audio once the text exists. A live inbound call goes through six.

flowchart TD A["Caller stops talking"] --> B["1. Endpoint detection (VAD)"] B --> C["2. Speech-to-text"] C --> D["3. LLM reasoning and state update"] D --> E["4. Tool calls, if any"] E --> F["5. Text-to-speech"] F --> G["6. Audio over SIP / PSTN"] style A fill:#2a78d6,stroke:#2a78d6,color:#fff style B fill:#4a3aa7,stroke:#4a3aa7,color:#fff style C fill:#eb6834,stroke:#eb6834,color:#fff style D fill:#e34948,stroke:#e34948,color:#fff style E fill:#e34948,stroke:#e34948,color:#fff style F fill:#eb6834,stroke:#eb6834,color:#fff style G fill:#1baf7a,stroke:#1baf7a,color:#fff

The 150ms figure describes step 5. Callers experience all six.

Stage	What it does	Why it costs time
Endpoint detection	Decides the caller has finished	Must tolerate pauses and "um"s without cutting people off
Speech-to-text	Transcribes the turn	Streaming helps, but the final words arrive last
LLM reasoning	Decides what to say and updates state	Often the largest single slice, around 700ms to first token per Daily
Tool calls	Calendar checks, CRM writes	Added in full if run inline
Text-to-speech	Generates audio	Where Eleven v4 Turbo's ~100ms of inference sits
Telephony	SIP trunk, carrier, jitter buffer	Invisible to any model benchmark

A faster TTS model helps. It doesn't close a 700ms LLM gap or a slow transcription pass. And none of the model leaderboards see the telephony leg at all: a WebSocket test runs over the open internet, while a real call crosses SIP trunking, a carrier, media relays and a jitter buffer built for lossy, codec-constrained circuits.

What real calls measure

Openbenchmarks exists to measure this. A caller bot dials each platform's live phone agent, both sides are recorded on one clock, and speech endpoints are detected from the audio itself with Silero VAD rather than from vendor timestamps. The metric is TTFAB: time from the caller going silent to the first byte of reply audio.

Platform	Median TTFAB	Notes
Telnyx	1,296ms	A carrier; sells no proprietary TTS model
ElevenLabs	1,424ms
Bland AI	1,520ms	p95 of 2,248ms
Vapi	1,558ms
Retell AI	1,740ms	Slowest median in the set

Every platform lands between 1.3 and 1.75 seconds, far above any model-card number. The telling detail is who's fastest. Telnyx doesn't sell a speech model. The lowest latency in the benchmark belongs to the infrastructure layer.

Why direct SIP trunking matters more than the TTS model

If every platform already uses a fast enough speech model, the remaining time is structural: how many network hops the audio makes between the caller and wherever inference runs, and how much of the pipeline runs one step after another instead of concurrently.

That's what we built Langoedge's telephony layer around. Calls connect directly over Telnyx and Twilio SIP trunking rather than through an intermediary voice API that then re-bridges into LiveKit, which removes a hop that stacked architectures carry by default. Our design target for that path is under 1.2 seconds end to end. That is our own target, not an independent measurement: we haven't been through Openbenchmarks' test yet, and a number a vendor states about itself is weaker evidence than one a third party records. That distinction is the whole point of this post.

The other structural lever is backend work. If a calendar check or CRM write runs inside the voice turn, its duration is added straight to TTFAB, however fast the TTS is. We keep that work in an asynchronous graph beside the conversation, which we describe in voice-to-text sub-routing. Integrations added over MCP need the same treatment, covered in MCP integrations for AI voice agents.

Three questions to ask any voice AI platform about latency

Is this a model benchmark or a phone-call benchmark? A WebSocket number describes the model. A number recorded from a live PSTN call describes the product you would deploy.
Who measured it? A vendor dashboard and an independent test using the same recorded-audio method across competitors are not equivalent evidence, even in the same unit.
Does it include backend work? A demo with no database lookup and no branching will always beat the agent you actually ship.

Latency is also only half of call quality. An agent that answers fast but books the wrong slot still failed; see LLM-as-judge for voice AI for how to score the other half.

FAQ

What is voice AI agent latency?

Voice AI agent latency is the time between a caller finishing what they're saying and hearing the agent's reply begin. It is measured end to end across endpoint detection, speech-to-text, LLM reasoning, any tool calls, text-to-speech and the telephony network, not just the time a single model takes to generate audio.

Why does ElevenLabs quote 150ms if real calls take over a second?

The 150ms figure is Eleven v4 Turbo's median time to first sound in a WebSocket streaming test, with about 100ms of model inference. Openbenchmarks measured ElevenLabs' full voice agent stack at 1,424ms median on real phone calls, because a call also includes endpoint detection, transcription, LLM reasoning and delivery over SIP and the phone network.

What is TTFAB?

TTFAB means time to first audio byte: the gap between a caller going silent and the agent's reply audio starting, measured from recordings of real calls rather than vendor timestamps. It reflects what a caller hears, including every stage a model benchmark leaves out.

What is a good latency target for a voice AI agent?

Daily's Pipecat team suggests keeping voice-to-voice latency under about 1,500ms for conversation to feel natural. In Openbenchmarks' real-call test, the five platforms measured ranged from 1,296ms to 1,740ms median.

Does a faster text-to-speech model make a voice agent noticeably faster?

It helps, but text-to-speech is one of six stages. LLM reasoning (around 700ms to first token in Daily's benchmarking), transcription and the telephony leg usually account for more. Removing a network hop or moving tool calls out of the voice turn often saves more time than a faster TTS model.

Sources

About the author. Arnab Chakraborty is the founder of Langoedge, a Melbourne company building voice and text AI agents on LangGraph with direct SIP telephony. Disclosure: Langoedge competes with some of the platforms named here. Their figures come from the independent sources cited.