ElevenLabs Agents: building an AI phone agent for your business

This page may contain affiliate links from which Blur Studio earns commissions.

ElevenLabs Agents is a platform for building conversational AI voice agents, systems that can hold a real-time spoken conversation with a caller, answer questions, and take actions, rather than reading a static script. This guide covers what it actually is under the hood, how setup works at a high level, real per-minute cost, and where phone agent projects like this tend to actually break down in practice.

What an agent actually is

An ElevenLabs agent combines four separate components into one system:

• A fine-tuned speech-to-text model that converts the caller's spoken words into text the agent can process.
• A language model of your choice (or a custom LLM you connect yourself), which decides what the agent should actually say in response.
• A low-latency text to speech engine, drawing from over 10,000 voices across 70+ languages, that turns the response back into natural-sounding speech.
• A proprietary turn-taking model that manages conversation timing, deciding when the caller has actually finished speaking so the agent doesn't interrupt or leave an awkward silence.

For grounding the agent in your actual business information, the platform includes a retrieval-augmented generation (RAG) knowledge base: you upload your own documents (pricing sheets, FAQs, policy documents) and the agent references them when answering, rather than relying purely on the underlying language model's general knowledge. Keeping that knowledge base current matters as much as building it in the first place, a pricing document that's three updates out of date will have the agent confidently quoting an old number to a real caller, which is a worse outcome than if the document had never been uploaded at all.

Setup steps at a high level

ElevenLabs supports three setup paths, and which one fits depends on your technical comfort:

1. Web dashboard: sign in, create a new agent, configure its behavior (first message, system prompt), add your knowledge base documents, select a voice, and test it directly in the browser. This is the non-technical path and is enough for most straightforward phone agent builds.
2. CLI: install ElevenLabs' command-line tool, authenticate, create an agent from a template, and push it to the platform, suited to teams that want agent configuration managed as code alongside their other development work.
3. API: create an API key, install the SDK, and define the agent's configuration directly in code, the most flexible option if the agent needs to be generated or modified programmatically rather than configured by hand.

To actually connect an agent to a phone number, ElevenLabs supports a native integration with Twilio, a cloud telephony provider that supplies phone numbers and call routing (import a Twilio number into the dashboard, and the platform configures the call routing automatically), and SIP trunk connectivity for businesses with an existing phone system, plus batch calling for triggering multiple outbound calls programmatically. Connecting your Twilio number to an AI voice agent walks through it.

Cost per call, and where it fails

As of October 2026, included agent call minutes scale with plan: 15 minutes on Free, 75 on Starter, 275 on Creator, 1,238 on Pro, 3,738 on Scale, and 12,375 on Business. Extra minutes beyond your plan's allowance bill at $0.08 a minute for standard calls, and $0.16 a minute if you exceed your plan's concurrent call limit. Two costs sit outside that structure entirely: LLM usage is billed separately, based on whichever model you chose, and telephony (the actual phone call connection through a provider like Twilio) passes through at that provider's own rate with no ElevenLabs markup. My separate agent pricing breakdown works through full monthly cost examples at different call volumes.

The failure modes that actually matter in practice tend not to be about voice quality, which is usually the part that impresses people in a demo. The real risk points are:

• An ungrounded knowledge base. If the uploaded documents are thin, outdated or missing a question callers actually ask, the agent can answer confidently with a wrong or made-up answer instead of saying it doesn't know, which is worse than no answer at all for something like pricing or policy questions.
• Latency from a heavier language model. A more capable LLM can introduce a noticeable delay before the agent responds, which feels unnatural on a phone call in a way it doesn't in a text chat. Why your AI phone agent replies slowly has fixes.
• Edge cases outside the script. A caller who goes off-topic, speaks unclearly, or asks something the agent genuinely can't handle needs a clean, honest handoff to a human or a clear limit, not the agent pretending it understood.
• Underestimated LLM and telephony costs. Since these bill separately from your ElevenLabs plan, a call volume that looked affordable on the headline per-minute rate can cost noticeably more once the LLM and telephony charges are added in.

A realistic first agent

Rather than launching an agent that tries to handle every possible call type on day one, a realistic first build covers a narrow, high-volume, low-risk call type well: hours and location questions, appointment confirmations, basic intake before a human callback. Get that version genuinely solid, tested against real questions your actual callers ask, before expanding its scope to handle more complex or higher-stakes conversations. This staged approach also makes the knowledge base problem more manageable, since a narrower scope means fewer documents to keep accurate and fewer edge cases the agent needs to handle gracefully.

Testing before a real call ever comes in

• Test with the questions your actual callers ask, not a clean demo script. Pull a sample of real call transcripts or notes from your existing phone line if you have them, and run those exact questions against the agent before launch.
• Test interruptions and unclear speech deliberately. A caller who talks over the agent, mumbles, or has a strong accent the model struggles with is a realistic scenario, not an edge case, and the turn-taking model's behavior under these conditions matters more than how it performs on a clean, scripted test call.
• Test the handoff path, not just the happy path. Confirm what actually happens when a caller asks something genuinely outside the agent's knowledge base, and make sure that path is a clean, honest "let me connect you with someone" rather than the agent guessing. Transferring an AI call to a person covers the setup.
• Test at the call volume you actually expect, not just one call at a time. A handful of simultaneous test calls shows far more about how the agent behaves under real concurrency than a single test call ever will, and it's the only reliable way to catch a concurrency limit problem before a real busy period does instead.

How I'd build a phone agent for a client

When I build a phone agent for a client, the knowledge base is where I spend the most real effort, not the voice selection. I test it against the actual questions real callers ask, not just the happy-path demo script, and I build in an honest fallback for anything it genuinely can't answer. Get in touch if you want a phone agent built and tested properly before it ever takes a real call.