Full-duplex voice agent · native tool calling

DuopraFrom full-duplex conversation
to full-duplex agency.

It listens and speaks at the same time, and decides on its own when to act.

Chiyu Chen1,2* Pengzhou Cheng3* Yuezhao Ma1 Xiaocheng Li1 Can Liu1 Haonan Hu2 Yao Yao1 Yifei Yang1 Yijie Lu1 Taolei Wang1 Yansi Li1 Muhuan Wu4 Zhuosheng Zhang1† Gongshen Liu1† Chenglu Sun2‡

1Shanghai Jiao Tong University 2Tencent 3Shanghai University 4Independent Researcher

*Equal contribution †Corresponding authors ‡Project lead

Recorded session
It's actually 150 euros, not 100.Duopra hears the correction, decides on its own to call the tool, and sends the call while it is still speaking.
Demo video Coming soon
Recorded in the Duopra app (in Chinese; English at the end). While it talks, Duopra checks tomorrow's weather in Shanghai with a tool, hands the day's itinerary and a small budget sum to the backend model as one more tool, and keeps up when the user cuts in to turn its story from spring to autumn, then into English.
#1among open-source systems on Full-Duplex-Bench v349.3% Pass@1 with an open backend. The next open-source result, 48.0%, is self-reported with another harness and judge.
47.7%of tasks passed, calling tools itselfPass@1 over 300 sessions, with no backend involved.
1.8×the pass rate, from full-duplex RL26.0% after supervised fine-tuning, 47.7% after reinforcement learning, on the same 300 sessions.
91.3%of interruptions answeredFull-Duplex-Bench v1.0, three seeds, in our streaming pipeline.
Overview

Listen. Speak. Act.

Full-duplex models already know when to listen and when to speak. Duopra also knows when to act — it calls tools itself, mid-sentence, through actions that are never voiced.

Listens while it speaks

Every second, Duopra decides whether to listen or to speak. It takes the turn when you finish, holds back in a mid-sentence pause and gives way when you cut in.

Acts mid-sentence, silently

Tool calls go out on a private channel while Duopra is still talking. The call itself is never spoken — only its result is.

Keeps up with corrections

Change a value halfway through a request and only the final value reaches the tool. The abandoned 100 never does.

Several steps in one request

Two changes asked for in one hesitant breath become two calls, and each is reported only after its result comes back.

Calls a backend like any tool

Connect a backend model and it becomes one more tool, query(task). Duopra hands work over the way it calls any tool: mid-conversation, with no pause and no hand-off you can hear.

Bring your own tools

A tool is a JSON schema: a name, a description and its parameters, supplied at run time. Write your own — your calendar, your database, your home — and Duopra can call it.

Recorded sessions

Watch one arrive, second by second.

Two sessions from the evaluation, replayed in recorded time.

The recordings
Two selected sessions, not a success rate. Transcripts, tool calls and the order they happened in come from the saved run; the bars are measured from the audio.
The audio
User audio comes from Full-Duplex-Bench v3. The assistant audio is recorded output from the same run, voiced with the original projector rather than the final one.
Tools and text
The tools are benchmark mocks — no live financial service is involved. The assistant transcript comes from ASR and may contain recognition errors.
How it works

One policy decides what to say and what to do.

Every second, Duopra reads the new audio and any tool results, then emits three things at once.

yt = ( mt , wt , at )
Turn statelisten or speak
Wordsvoiced by the talker
Actionnever voiced

One causal policy, conditioned on the audio so far, the observations injected so far and its own past outputs. Conversation and tool use are decided together, not handed between components.

The backend is one more tool.

A background agent is exposed as query(task), whose argument is a self-contained description of the user's task. Calling a tool directly or delegating is the same kind of decision, made by the same policy — there is no external dispatcher.

Results land on unit boundaries.

A runtime executes each call asynchronously and injects the result at the first one-second boundary after it returns. The model keeps listening and speaking while it waits, and reports a value only once it has arrived.

Two timelines of the same request. In the frontend–backend design the frontend, which cannot see the tool schemas, hands the task to the backend three times; Duopra sees the schemas, asks for the flight and the date, and calls flight.status itself, twice.
Figure 1. Why the frontend has to act (schematic, 1-s units). (a) With tool schemas visible only to the backend, the frontend knows what to do but not how: it hands off at once, and the answer and the change each cost another hand-off. (b) Duopra sees the schemas — the backend tool query(task) included — asks for both values and calls flight.status itself, then again with the new date.

Trained in three stages

One stage for each problem: the form of an action, the decision to act, and a voice that drifts when the thinker changes.

  1. 1Stage 1

    Protocol SFT

    Teaches the form of an action.

    Supervised fine-tuning of the thinker on synthetic sessions that demonstrate the protocol: call tools by schema, wait for the result before reporting it, ask only for missing values.

    Duopra-SFT · Pass@126.0%Calls are well formed, but 55% of sessions still end without one.
  2. 2Stage 2

    Full-duplex RL

    Teaches the decision to act.

    Dr. GRPO on a thinker LoRA, eight rollouts per task, in a sandbox that plays the same scripted user to every rollout. Time-anchored deadlines and outcome tiers reward calls that are timely, complete and grounded.

    Duopra · Pass@126.0→47.7%Sessions without a call fall from 55% to 19%.
  3. 3Stage 3

    Projector-only repair

    Keeps the voice intelligible.

    Post-training shifts the thinker's hidden states, which condition the talker. Only the projector between them is retrained, on cached features; the text, control tokens and calls are untouched.

    Pilot · Chinese character error rate5.17→1.78%88 pilot samples; 2.05% before post-training.
The three training stages. Stage 1 trains the thinker on unit-level sessions. Stage 2 trains a thinker LoRA with Dr. GRPO in a sandbox with a scripted user, a backend agent, simulated tools and a scorer. Stage 3 trains only the projector on cached hidden states.
Figure 2. Duopra's three training stages on Realtime-Venus-Audio; each row marks what its stage trains. ht: thinker hidden state; P: projector; ct: talker conditioning.
Results

Tools that actually get called.

The highest pass rate among open-source systems on Full-Duplex-Bench v3.

Pass@1 on Full-Duplex-Bench v3

A session passes only when every executed call matches the gold call: right tool, every argument, no extra call.

GPT-Realtime*
60.0reported
Gemini Live 3.1*
54.0reported
MOSS-Live*
51.0reported
Duopra · DeepSeek backend
49.3± 3.5
Gemini Live 2.5*
49.0reported
Hu et al. · Qwen3-235B backend
48.0reported
Duopra · calling tools itself
47.7± 3.5
DuplexAgent-RSI
47.0reported
Cascaded*
45.0reported
Duopra · Qwen3-8B backend
44.7± 5.7
Grok*
43.0reported
Realtime-Venus-Audio · offline
42.0reported
Ultravox v0.7*
41.0reported
Gander · GPT-5.6
40.0reported
NemotronLabs VoiceChat
33.0reported

Full-Duplex-Bench v3: 100 real, disfluent English recordings and 12 simulated APIs. Duopra runs in our harness — three seeds, 300 sessions per configuration, judged by Qwen3.8-27B; ± is the standard deviation across seeds. Hatched bars are reported numbers, measured with each source's own harness and judge (mostly GPT-4o), so they compare only approximately. * Commercial system.

Training removes failures to act.

The outcome of each of 300 sessions, Duopra models calling tools themselves.

Duopra-SFTafter Stage 1
26%55%
Duopra-RL1after Stage 2a
27%44%
Duopraafter Stage 2b
48%34%19%

Multi-call requests become solvable: Duopra passes 28 of 102 sessions that need two or three calls, against 2 after SFT. The remaining headroom is in arguments — wrong arguments (19.3% of sessions) are now as common as sessions without a call (18.7%).

Show as a table
Run (300 sessions)PassWrong arg.MissingExtraNo call
Duopra-SFT7817381166
Duopra-RL18017720131
Duopra1435840356

Learning to act, without forgetting to converse.

FDB v1.0 takeover rates of Duopra and the model it is built on, Realtime-Venus-Audio, in the same pipeline (%).

Takes the turn when the user finishes higher is better

59.1
75.6

Takes the turn in a mid-sentence pause lower is better

18.1
5.4

Responds after an interruption higher is better

79.5
91.3

Not every behaviour survives: Duopra answers fewer interruptions than Duopra-RL1 (96.5%) and, like every post-trained stage, no longer backchannels.

More numbers — every metric for the systems above, and Duopra's conversational scoresShow
Full-Duplex-Bench v3 — tool selection, arguments, response quality and Pass@1 (%)
SystemToolArg.Resp.Pass@1
Commercial systems*, as reported
GPT-Realtime87.668.079.260.0
Gemini Live 3.181.758.871.854.0
MOSS-Liven/an/an/a51.0
Gemini Live 2.578.659.355.449.0
Cascaded80.356.260.045.0
Grok79.754.261.743.0
Ultravox v0.779.451.351.041.0
Open-source systems, as reported
Hu et al., Qwen3-235B backend71.755.267.048.0
DuplexAgent-RSI97.457.0n/a47.0
Realtime-Venus-Audio, offline82.052.2n/a42.0
Gander, GPT-5.675.950.349.040.0
NemotronLabs VoiceChat82.542.2n/a33.0
Duopra, our harness
Duopra, calling tools itself76.756.266.347.7 ± 3.5
Duopra, Qwen3-8B backend81.857.070.644.7 ± 5.7
Duopra, DeepSeek backend71.855.069.049.3 ± 3.5

Tool: tool-selection F1, which involves no judge (Hu et al. report tool-selection accuracy). Arg.: argument accuracy. Resp.: response quality. n/a: not reported. Bold: best within each group. Reported rows use each source's harness and judge, mostly GPT-4o; Duopra's use Qwen3.8-27B. DuplexAgent-RSI is measured on VoiceChat (49.0 on Qwen-Realtime) with exact-string argument matching; Realtime-Venus-Audio offline has the tool definitions in the prompt; MOSS-Live reports a tool-call success rate. Further open-source systems that we ran in our harness are listed in the paper's Table 1.

Conversational control and general ability — Duopra
MetricValue
Takes the turn after the user finishes ↑ FDB v1.075.6
Takes the turn during a pause ↓ FDB v1.05.4
Responds after an interruption ↑ FDB v1.091.3
Takes the turn at all ↑ FDB v3, calling tools itself93.0
VoiceBench, mean of nine categories ↑55.9
Task-completion latency ↓ domain tools and backend both visible, Qwen3-8B10.42 s

Pass@1 is read from the call log rather than the transcript. ± is the standard deviation across three seeds. Judged metrics use Qwen3.8-27B; tool-selection F1 involves no judge. Delegating every request instead raises the task-completion latency to 14.85 s.

Paper

Duopra: From Full-Duplex Conversation to Full-Duplex Agency

The code and the weights are available now; the preprint will follow.

Read the abstract

Full-duplex interaction models exhibit strong conversational intelligence, enabling natural, low-latency turn-taking, backchannels, and interruptions. However, successful task completion further requires task agency—timely tool invocation and argument revision following user corrections. In frontend–backend architectures, delegation and execution errors are often entangled, obscuring the primary bottleneck. We first investigate this question on Full-Duplex-Bench (FDB) v3. Realtime-Venus-Audio with its original Harness delegates only 11.0% of sessions and passes 4.7%; even a perfect backend could improve performance by at most 6.3 points. We then introduce Duopra, which directly performs non-verbalized tool calls and treats backend delegation as tool selection. Its three-stage training combines supervised fine-tuning, full-duplex reinforcement learning with time-anchored deadlines and tiered rewards, and projector-only talker repair. With a fixed backend, Duopra raises the pass rate from 4.7% to 44.7%, compared with a 4.7-point gain from a much larger backend. It achieves 47.7% standalone and 49.3% with an open backend, outperforming all compared open-source systems. Complementary evaluations show improved turn-taking control and overlap handling over the base model, though with reduced backchanneling, showing that task agency can build on conversational agency.

BibTeX

@misc{duopra2026,
  title  = {Duopra: From Full-Duplex Conversation
            to Full-Duplex Agency},
  author = {Chen, Chiyu and Cheng, Pengzhou and Ma, Yuezhao and Li, Xiaocheng
            and Liu, Can and Hu, Haonan and Yao, Yao and Yang, Yifei
            and Lu, Yijie and Wang, Taolei and Li, Yansi and Wu, Muhuan
            and Zhang, Zhuosheng and Liu, Gongshen and Sun, Chenglu},
  year   = {2026},
  note   = {Preprint forthcoming},
  url    = {https://cychen2003.github.io/duopra/}
}

The entry will be updated with a link once the preprint is public.