Why does my voice agent mishear callers on noisy job sites?

This page may contain affiliate links from which Blur Studio earns commissions.

When a voice agent gets something wrong on a call, the instinct is to blame the "AI" broadly, but the actual failure point is usually narrower than that: the speech-to-text layer that turns the caller's voice into text the agent reasons over. If that layer mishears the words, nothing downstream can fix it, no matter how good the agent's logic is. On a genuinely noisy job site, where there's traffic, machinery, wind or other people talking nearby, this layer is doing a much harder job than it would on a clean office call, and it shows.

Why noise breaks speech recognition specifically

Speech-to-text models learn to separate speech from everything else, and background noise competes directly with that task. Steady background noise (a fan, a running engine) is usually handled reasonably well by modern engines; irregular, loud or speech-like noise (another conversation, a radio, sudden machinery) is much harder, because the model has to actively decide what counts as the caller's voice versus not, in real time, with no chance to go back and listen again.

Not all job-site noise is the same problem

It's worth separating the noise types your sites actually produce, since they don't all cause the same kind of failure. Steady, continuous noise like road traffic at a distance, a running generator, or HVAC equipment is the easiest case: it's constant enough that the model can effectively learn to treat it as background and focus on the voice sitting on top of it. Intermittent, sudden noise, a jackhammer starting, a truck backing up with a beeper, a door slamming, is harder, because it arrives unpredictably and can mask a word or two entirely with no warning. The hardest case by far is crosstalk, another person's speech in the background, since the model has to distinguish one human voice from another rather than separating voice from non-voice noise, and that's a fundamentally harder acoustic problem than filtering out mechanical sound. If your job sites produce a lot of crosstalk specifically (a crew talking nearby, a radio with speech on it), expect worse accuracy than a site with mostly mechanical noise, even at the same rough loudness. When you're collecting test recordings to evaluate an engine, make sure you capture examples of each noise category your sites actually produce, not just one loud recording, since a single test file can give you a misleading sense of how an engine handles your full range of real conditions. For recorded video, see removing background noise from a job site video.

What actually differs between engines

Different speech-to-text engines are trained on different mixes of clean and noisy audio, and that training mix is a big part of why one engine handles a construction site better than another, even if both perform similarly on a quiet recording. ElevenLabs markets its Scribe v2 Realtime specifically on accuracy "even in challenging audio conditions," built for sub-150-millisecond latency so a live voice agent doesn't feel like it's lagging, at $0.39 an hour through the API. That's a real, stated design goal, not a universal guarantee for every noise type your specific sites produce. Independent benchmark testing of ElevenLabs' Scribe v2 family has put accuracy under 5% word error rate across a wide set of languages in general testing, which is a strong baseline, but general benchmark conditions and your actual job site aren't the same thing. Scribe itself is covered in ElevenLabs Speech to Text (Scribe).

The hardware on the caller's end matters at least as much as the engine, and it's the part people overlook while comparing speech-to-text providers. A phone held at arm's length on speaker in the wind will produce a worse signal than any engine can fully recover from, no matter how good its noise handling is in testing. A headset or a dedicated microphone positioned close to the mouth gives any engine a meaningfully cleaner signal to start from, and this single change often improves real-world accuracy more than switching providers does. If you're troubleshooting poor accuracy on a specific site, check how the caller is actually holding or wearing their phone before assuming the software needs replacing.

Noise cleanup tools are a separate lever worth knowing about too, distinct from the speech-to-text engine itself. A dedicated audio cleanup step, run on the recording before it reaches the transcription engine, can strip out some background noise ahead of time, which sometimes improves accuracy on a borderline recording without needing to switch speech-to-text providers at all. This is a different tool from the transcription engine, used as a preprocessing step, not a replacement for it.

What I'd actually tell someone building this

Don't trust a vendor's noise-handling claim, mine included in what I've said above, without testing it against your own real recordings. Pull a handful of actual calls or recordings from your noisiest sites, the kind with machinery running or wind picking up, and run them through whichever engine you're considering before committing. The difference between "handles noise well in a demo" and "handles noise well on your actual job sites" is exactly the gap that causes frustrated callers later, and it's cheap to check up front with real audio compared to finding out after you've built around the wrong engine.

Practical steps to actually fix this

1. Collect several real recordings from your noisiest actual sites, not a quiet test call.
2. Run those exact recordings through the engine you're considering (ElevenLabs' Scribe v2 Realtime among them) and compare the transcript against what was actually said.
3. If accuracy is genuinely close but not perfect, add custom vocabulary (keyterms) for words specific to your trade, which can meaningfully improve recognition on the exact terms most likely to matter.
4. Keep a human fallback path for calls where the agent's confidence in what it heard is low, rather than letting it guess and proceed on bad information.
5. Re-test periodically as your sites or equipment change; a mix of noise that worked fine before can shift, especially if you add a new piece of equipment, change crews, or start taking calls from a new kind of site your original test recordings didn't cover.

Designing what the agent does when it isn't sure

A good noisy-environment setup isn't just about raw accuracy, it's about what the agent does when it genuinely can't tell what was said. Most speech-to-text engines return a confidence score alongside the transcribed text, and a well-built agent should use that score to decide its next move rather than always guessing at the most likely interpretation. On low confidence, the better response is a short, specific clarifying question ("Sorry, did you say the pump or the valve needs replacing?") rather than either guessing silently and proceeding on bad information, or a generic "I didn't understand, please repeat" that makes the caller repeat the whole sentence. Asking a targeted question about the specific unclear word keeps the call moving while still catching the error, and callers tend to tolerate one good clarifying question far better than a vague repeat request or, worse, an agent that confidently acts on the wrong information. Slow replies are a separate problem, covered in why your AI phone agent replies slowly.

When to skip building this yourself

If you're testing engines against real recordings and still seeing it fail often, the issue may not be the engine at all, it could be where the microphone sits, how far the caller is from it, or genuinely extreme noise no software fix will solve well. In that case, a human fallback for the worst conditions is more realistic than chasing a perfect automated answer. Trying to squeeze another few percent of accuracy out of software when the real constraint is an unavoidably loud environment is a poor use of time compared to simply routing those specific calls to a person and automating everything else.

How I'd approach this

I've tested multiple speech-to-text engines against a client's own real, noisy recordings before picking one, specifically because vendor noise-handling claims are marketing, not a guarantee for any particular environment. If you're building or already running a voice agent that keeps misfiring on noisy calls, that kind of real-audio comparison is worth doing properly once, rather than guessing at which engine to trust. Beyond picking the right engine, the clarifying-question logic and human-fallback routing described above are usually where I spend the most build time, since they determine whether a mishear turns into a quick, cheap recovery or a frustrated caller who hangs up. Getting the engine right solves most of the problem; getting the fallback behavior right is what makes the remaining failures cheap instead of costly.