Full-duplex voice agent · native tool calling

DuopraFrom full-duplex conversation
to full-duplex agency.

It listens and speaks at the same time, and decides on its own when to act.

Chiyu Chen1,2* Pengzhou Cheng3* Yuezhao Ma1 Xiaocheng Li1 Can Liu1 Haonan Hu2 Yao Yao1 Yifei Yang1 Yijie Lu1 Taolei Wang1 Yansi Li1 Muhuan Wu4 Zhuosheng Zhang1† Gongshen Liu1† Chenglu Sun2‡

1Shanghai Jiao Tong University 2Tencent 3Shanghai University 4University of Bristol

*Equal contribution †Corresponding authors ‡Project lead

Recorded session
It's actually 150 euros, not 100.Duopra hears the correction, decides on its own to call the tool, and sends the call while it is still speaking.
Demo video Coming soon
Recorded in the Duopra app (in Chinese; English at the end). While it talks, Duopra checks tomorrow's weather in Shanghai with a tool, hands the day's itinerary and a small budget sum to the backend model as one more tool, and keeps up when the user cuts in to turn its story from spring to autumn, then into English.
49.3%of tasks passed with a backendPass@1 over 300 sessions, with the domain tools behind a DeepSeek backend that Duopra calls as one more tool.
47.7%of tasks passed, calling tools itselfPass@1 over 300 sessions, with no backend involved.
1.8×the pass rate, from full-duplex RL26.0% after supervised fine-tuning, 47.7% after reinforcement learning, on the same 300 sessions.
91.3%of interruptions answeredFull-Duplex-Bench v1.0, three seeds, in our streaming pipeline.
Overview

Listen. Speak. Act.

Full-duplex models already know when to listen and when to speak. Duopra also knows when to act — it calls tools itself, mid-sentence, through actions that are never voiced.

Listens while it speaks

Every second, Duopra decides whether to listen or to speak. It takes the turn when you finish, holds back in a mid-sentence pause and gives way when you cut in.

Acts mid-sentence, silently

Tool calls go out on a private channel while Duopra is still talking. The call itself is never spoken — only its result is.

Keeps up with corrections

Change a value halfway through a request and only the final value reaches the tool. The abandoned 100 never does.

Several steps in one request

Two changes asked for in one hesitant breath become two calls, and each is reported only after its result comes back.

Calls a backend like any tool

Connect a backend model and it becomes one more tool, query(task). Duopra hands work over the way it calls any tool: mid-conversation, with no pause and no hand-off you can hear.

Bring your own tools

A tool is a JSON schema: a name, a description and its parameters, supplied at run time. Write your own — your calendar, your database, your home — and Duopra can call it.

Recorded sessions

Watch one arrive, second by second.

Two sessions from the evaluation, replayed in recorded time.

The recordings
Two selected sessions, not a success rate. Transcripts, tool calls and the order they happened in come from the saved run; the bars are measured from the audio.
The audio
User audio comes from Full-Duplex-Bench v3. The assistant audio is recorded output from the same run, voiced with the original projector rather than the final one.
Tools and text
The tools are benchmark mocks — no live financial service is involved. The assistant transcript comes from ASR and may contain recognition errors.
How it works

One policy decides what to say and what to do.

Every second, Duopra reads the new audio and any tool results, then emits three things at once.

yt = ( mt , wt , at )
Turn statelisten or speak
Wordsvoiced by the talker
Actionnever voiced

One causal policy, conditioned on the audio so far, the observations injected so far and its own past outputs. Conversation and tool use are decided together, not handed between components.

The backend is one more tool.

A background agent is exposed as query(task), whose argument is a self-contained description of the user's task. Calling a tool directly or delegating is the same kind of decision, made by the same policy — there is no external dispatcher.

Results land on unit boundaries.

A runtime executes each call asynchronously and injects the result at the first one-second boundary after it returns. The model keeps listening and speaking while it waits, and reports a value only once it has arrived.

Figure 1 in two panels. (a) Frontend–backend separation, four hand-offs: the frontend has no tool schemas; the backend has them, injected from a catalog of search_products, add_to_cart and more. The user says “Search for a laptop…”, “Wait, a tablet.”, “Add 2 to the cart…” and “Actually, just 1.” The frontend says “Let me check.” and hands off “Find a laptop.”; the backend thinks and calls search_products(laptop). After “Wait, a tablet.” it hands off “Make it a tablet.”; the backend thinks again, undoes the search, searches for a tablet and replies, and only then does the frontend say “OK, a tablet.” It hands off “Add 2 to the cart.”; the backend thinks and calls add_to_cart(tablet_001, 2). After “Actually, just 1.” it hands off “Make it 1.”; the backend thinks, undoes the call, adds one and replies, and the frontend says “Done, it's $99.” (b) Duopra, no hand-offs: the frontend sees the tool schemas, query(task) included. It says “Sure, searching.” and calls search_products(laptop) itself; on “Wait, a tablet.” it undoes the call, searches for a tablet and says “OK, a tablet.”; it says “Adding two.” and calls add_to_cart(tablet_001, 2); on “Actually, just 1.” it undoes the call, adds one and says “Done, it's $99.”.
Figure 1. Frontend responsibility in an evolving FDB v3 request (ecommerce_19; schematic). The user corrects both the product (laptop to tablet) and the quantity (two to one); add_to_cart depends on the product ID returned by search_products. (a) When domain schemas are available only to the backend, the frontend may need successive hand-offs to convey the evolving request, making it harder to preserve corrections and tool dependencies. (b) With schemas visible, Duopra can issue the corrected domain-tool calls directly without a hand-off, while query(task) remains available. Crossed-out values and curved arrows denote superseded requests. Table 18(a) of the paper reports an observed delegated success on this request.

Trained in three stages

One stage for each problem: the form of an action, the decision to act, and a voice that drifts when the thinker changes.

  1. 1Stage 1

    Protocol SFT

    Teaches the form of an action.

    Supervised fine-tuning of the thinker on synthetic sessions that demonstrate the protocol: call tools by schema, wait for the result before reporting it, ask only for missing values.

    Duopra-SFT · Pass@126.0%Calls are well formed, but 55% of sessions still end without one.
  2. 2Stage 2

    Full-duplex RL

    Teaches the decision to act.

    Dr. GRPO on a thinker LoRA, eight rollouts per task, in a sandbox that plays the same scripted user to every rollout. Time-anchored deadlines and outcome tiers reward calls that are timely, complete and grounded.

    Duopra · Pass@126.0→47.7%Sessions without a call fall from 55% to 19%.
  3. 3Stage 3

    Projector-only repair

    Keeps the voice intelligible.

    Post-training shifts the thinker's hidden states, which condition the talker. Only the projector between them is retrained, on cached features; the text, control tokens and calls are untouched.

    Same thinker · Chinese character error rate6.01→2.58%Original vs. repaired projector, 221-utterance selection set; English WER 9.74→7.66%.
Duopra's three training stages on Realtime-Venus-Audio. Stage 1 trains the thinker on unit-level sessions and yields Duopra-SFT. Stage 2 trains a thinker LoRA with Dr. GRPO in a sandbox with a scripted user, a backend agent, simulated tools and a scorer; Stage 2a yields Duopra-RL1 and Stage 2b Duopra-RL. Stage 3 freezes the Duopra-RL thinker and trains only its projector, on cached hidden states, giving the final Duopra.
Figure 2. Duopra's three training stages on Realtime-Venus-Audio (ht: thinker hidden state; P: projector; ct: talker conditioning). Stage 1 trains the thinker on unit-level sessions and yields Duopra-SFT. Stage 2 trains a thinker LoRA with Dr. GRPO in the sandbox; Stage 2a yields Duopra-RL1 and Stage 2b yields Duopra-RL. Stage 3 freezes the Duopra-RL thinker and trains only its projector, producing the final Duopra model.
Results

Tools that actually get called.

Pass@1 on Full-Duplex-Bench v3: 49.3% with a backend, 47.7% calling tools itself.

Pass@1 on Full-Duplex-Bench v3

A session passes only when every executed call matches the gold call: right tool, every argument, no extra call.

GPT-Realtime*
60.0reported
Gemini Live 3.1*
54.0reported
MOSS-Live*
51.0reported
Duopra · DeepSeek backend
49.3± 3.5
Gemini Live 2.5*
49.0reported
Hu et al. · Qwen3-235B backend
48.0reported
Duopra · calling tools itself
47.7± 3.5
DuplexAgent-RSI
47.0reported
Cascaded*
45.0reported
Duopra · Qwen3-8B backend
44.7± 5.7
Grok*
43.0reported
Realtime-Venus-Audio · offline
42.0reported
Ultravox v0.7*
41.0reported
Gander · GPT-5.6
40.0reported
NemotronLabs VoiceChat
33.0reported

Full-Duplex-Bench v3: 100 real, disfluent English recordings and 12 simulated APIs. Duopra runs in our harness — three seeds, 300 sessions per configuration, judged by Qwen3.8-27B; ± is the standard deviation across seeds. Hatched bars are published results, measured with each source's own harness and judge (mostly GPT-4o): reference points, not a common-protocol ranking. * Commercial system.

Training cuts failures to act.

The outcome of each of 300 sessions, Duopra models calling tools themselves.

Duopra-SFTafter Stage 1
26%55%
Duopra-RL1after Stage 2a
27%44%
Duoprafinal model
48%34%19%

Multi-call requests become solvable: Duopra passes 28 of 102 sessions that need two or three calls, against 2 after SFT. The remaining headroom is in arguments — wrong arguments (19.3% of sessions) are now as common as sessions without a call (18.7%).

Show as a table
Run (300 sessions)PassWrong arg.MissingExtraNo call
Duopra-SFT7817381166
Duopra-RL18017720131
Duopra1435840356

Learning to act, while taking turns better.

FDB v1.0 takeover rates of Duopra and the model it is built on, Realtime-Venus-Audio, in the same pipeline (%).

Takes the turn when the user finishes higher is better

59.1
75.6

Takes the turn in a mid-sentence pause lower is better

18.1
5.4

Responds after an interruption higher is better

79.5
91.3
More numbers — every metric for the systems above, and Duopra's conversational scoresShow
Full-Duplex-Bench v3 — tool names, arguments, response quality and Pass@1 (%)
SystemToolArg.Resp.Pass@1
Commercial systems*, as reported
GPT-Realtime87.668.079.260.0
Gemini Live 3.181.758.871.854.0
MOSS-Liven/an/an/a51.0
Gemini Live 2.578.659.355.449.0
Cascaded80.356.260.045.0
Grok79.754.261.743.0
Ultravox v0.779.451.351.041.0
Open-source systems, as reported
Hu et al., Qwen3-235B backend71.755.267.048.0
DuplexAgent-RSI97.457.0n/a47.0
Realtime-Venus-Audio, offline82.052.2n/a42.0
Gander, GPT-5.675.950.349.040.0
NemotronLabs VoiceChat82.542.2n/a33.0
Duopra, our harness
Duopra, calling tools itself76.756.266.347.7 ± 3.5
Duopra, Qwen3-8B backend81.857.070.644.7 ± 5.7
Duopra, DeepSeek backend71.855.069.049.3 ± 3.5

Tool: tool-name F1, which involves no judge (Hu et al. report tool-selection accuracy). Arg.: argument accuracy. Resp.: response quality. n/a: not reported. Bold: best within each group. Published rows use each source's harness and judge, mostly GPT-4o, and are reference points, not a common-protocol ranking; Duopra's use Qwen3.8-27B. DuplexAgent-RSI is measured on VoiceChat (49.0 on Qwen-Realtime) with exact-string argument matching; Realtime-Venus-Audio offline has the tool definitions in the prompt; MOSS-Live reports a tool-call success rate. Calling tools itself is the paper's Direct setting; a named backend is its Backend setting, in which only query(task) is visible. Further open-source systems that we ran in our harness are in the paper's Tables 2 and 11.

Conversational control and general ability — Duopra
MetricValue
Takes the turn after the user finishes ↑ FDB v1.075.6
Takes the turn during a pause ↓ FDB v1.05.4
Responds after an interruption ↑ FDB v1.091.3
Takes the turn at all ↑ FDB v3, calling tools itself93.0
VoiceBench, mean of nine categories ↑55.9
Factual-answer latency ↓ domain tools and backend both visible, Qwen3-8B10.42 s

Pass@1 is read from the call log rather than the transcript. ± is the standard deviation across three seeds. Judged metrics use Qwen3.8-27B; tool-name F1 involves no judge. Factual-answer latency is measured until the factual answer, including runtime and tool execution.

Paper

Duopra: From Full-Duplex Conversation to Full-Duplex Agency

The code and the weights are available now; the preprint will follow.

Read the abstract

Full-duplex interaction models exhibit strong conversational agency, enabling natural, low-latency turn-taking, backchannels, and interruptions. However, successful task completion further requires task agency that timely tool invocation and argument revision following user corrections. Frontend–backend agents separate initiation from execution: the frontend decides whether and what to request, while the backend determines how. Execution can be offloaded, but unsent requests cannot be executed, and incomplete ones can misdirect the backend. We investigate this issue on FD-Bench-v3, most failures of Realtime-Venus-Audio + Harness in our streaming setting occur before a domain tool executes.

We introduce Duopra, a full-duplex interaction model that learns task execution alongside conversational control. Duopra treats the backend as a tool, accepting a self-contained natural-language request. Direct calls and backend delegation share an action interface and requirements for faithful initiation and correct execution. Its three-stage training combines protocol SFT, full-duplex RL rewarding timely and complete execution, and projector-only repair of talker conditioning after thinker updates. With a fixed backend, Pass@1 rises from 4.7% to 44.7%; Duopra achieves 47.7% under Direct (26.0% after SFT) and 49.3% under Backend with DeepSeek. It improves three FDB v1.0 takeover rates over the base model. These results show that task agency can coexist with key conversational agency.

BibTeX

@misc{duopra2026,
  title  = {Duopra: From Full-Duplex Conversation
            to Full-Duplex Agency},
  author = {Chen, Chiyu and Cheng, Pengzhou and Ma, Yuezhao and Li, Xiaocheng
            and Liu, Can and Hu, Haonan and Yao, Yao and Yang, Yifei
            and Lu, Yijie and Wang, Taolei and Li, Yansi and Wu, Muhuan
            and Zhang, Zhuosheng and Liu, Gongshen and Sun, Chenglu},
  year   = {2026},
  note   = {Preprint forthcoming},
  url    = {https://cychen2003.github.io/duopra/}
}

The entry will be updated with a link once the preprint is public.