PolyAI's new voice model listens to raw call audio instead of a transcript

What PolyAI announced

PolyAI, a company that builds voice agents for businesses, has introduced Dialog-RSN-1. As poly.ai described it, this is a dialog model that takes in the caller's audio directly. It combines four jobs that are usually separate: deciding when the caller has finished talking, turning speech into text, calling software tools, and writing the reply.

PolyAI says the model is already handling live customer calls and is the fastest system it has measured. The pitch is that a business's phone agent can behave more like a good human one: noticing hesitation, picking up frustration early, and knowing when to jump in or hold back.

Why the usual designs fall short

Most voice agents today are cascades: a speech-recognition program (ASR) converts audio to text, and a large language model reads only that text. PolyAI points out the weakness. The language model gets one best guess at the words, with no tone and no sign of which words were uncertain. The recognizer is also a smaller model that cannot reason about the conversation, so the language model often has to clean up its mistakes.

The alternative is speech-to-speech models, such as GPT Realtime and Gemini Live, which listen and speak with one system. PolyAI says these have costs of their own. The voice is built into the model, so you cannot easily fix how it says your brand name. They are also expensive to run, especially versions that stay on for the whole call and tie up a GPU, the specialized chip that runs AI models.

PolyAI also doubts that some advertised speech-to-speech products are truly end-to-end. Its tests found GPT Realtime scoring about the same as cascades on tasks that need real audio awareness, which it takes as a sign that those products may be well-built cascades underneath. That is PolyAI's reading of its own tests, not an established fact.

How the model decides when to speak

PolyAI's design takes a middle path: audio-aware on the way in, conventional on the way out. Dialog-RSN-1 hears the raw audio, but a separate text-to-speech system produces the reply voice. The company argues that most of the benefit of audio awareness is on the listening side, and that keeping the voice separate preserves control over how it sounds.

The model runs as a normal request-based system. A simple detector of speech activity, set to fire easily and often, plus a few timers, prompts the model to look at the audio so far. A false alarm costs almost nothing. The model's very first output is a verdict: nothing said yet, the caller is still talking, or the turn is complete and it should answer.

Because that call is made by the language model, turn-taking becomes something you can steer in plain instructions. PolyAI's examples include waiting until a caller has said all 6 digits of a customer code, staying quiet if the caller is talking to someone else, and waiting for the beep before leaving a message. The model can also work out, from context, that a caller spelling a name or saying a brand like Audibel does not mean the similar-sounding common word.

PolyAI says transcription still happens, but last, in parallel with speech generation, so callers do not wait for it. The transcript is kept for logs and safety checks.

How it was built and measured

PolyAI built the model by further training open-weight models, ones whose internals are publicly available, using examples from real deployments and synthetic conversations, followed by reinforcement training that rewards better behavior. It says the method works with several base models, including Gemma, GPT-OSS, Qwen and Mistral, and that it targets sizes from 8B dense to 30B sparse so replies stay fast on A100 GPUs. A technical report comparing the base models is promised.

For testing, PolyAI uses its own benchmark, Dialog-Eval. Each example freezes a call at one decision point, with the caller's latest audio in place, and checks one next step, such as whether the model keeps listening after four of six digits or calls the booking tool correctly. The company says cascaded systems top out at around 77 on its audio-focused score because the transcript limits what they can know, and that Dialog-RSN-1 scores highest among models fast enough for live calls.

The one customer figure is an 11% relative increase in containment, meaning calls finished without handing off to a person, at a national restaurant group. It is a single customer result reported by the vendor, with no detail on how it was measured.

What to make of it

The design idea is clear and concrete: let one model hear the audio and own the timing decision, while keeping the voice and the cost under the business's control. But the comparisons with rival systems, and the claim about what GPT Realtime is doing inside, come from a company selling a competing product, on a test it wrote itself. PolyAI says it will open-source that benchmark, which would let outsiders check it.

For teams that run support lines, the practical question is whether the gains hold on their own calls, with their own accents, noise and jargon.