Voice agents

Duotone is a live conversation you can interrupt: realtime speech to text hears the caller, a language model writes the reply, and streaming speech says it while it is still being written. Try it on the Duotone page. In your own app, use the Duotone socket, one WebSocket that runs the whole call, or build your own from the three pieces.

Use the Duotone socket

One WebSocket per call. You send the caller's microphone audio; it sends back what it heard, the reply as it is written, and the reply's audio. Turn-taking, interruptions and the language model all run on Yoob's side.

1. Connect and configure

The first message is the call's config, within 10 s. Every field is optional; {} starts a call with Duotone's own persona, voice w1243, and a greeting.

2. Stream the microphone

Send the caller's microphone as binary frames: raw 16-bit little-endian PCM, mono, 16 kHz, at most 1 s per frame (40 ms, 1,280 bytes, is a good size). Keep sending while Duotone speaks: that is how it hears the caller cut in. Capture with echo cancellation on, or it will hear its own voice and interrupt itself.

3. Play the reply, show the transcript

Audio arrives faster than it plays, so a reply is usually queued on your side ahead of what the caller hears. That is why an interrupted reply has to be cut off where you play it, not only where it is made.

Example

import WebSocket from "ws";

const call = new WebSocket("wss://tts-api.yoob.com/v1/duotone", {
  headers: { Authorization: `Bearer ${process.env.YOOB_API_KEY}` },
});

call.on("open", () => {
  call.send(JSON.stringify({ voice: "w1243", quality: "best" }));
  // s16le mono 16 kHz from the caller, ~40 ms at a time
  microphone.on("frame", (pcm) => call.send(pcm));
});

call.on("message", (data, isBinary) => {
  if (isBinary) return speaker.play(data);          // s16le mono 48 kHz
  const e = JSON.parse(data);
  if (e.type === "user" && e.final) console.log("caller:", e.text);
  if (e.type === "reply_done") {
    if (e.interrupted) speaker.flush();             // they cut in: drop what is queued
    console.log("duotone:", e.text);
  }
  if (e.type === "done") console.log("usage:", e.usage);
});

// hang up
// call.send(JSON.stringify({ type: "stop" }));

Typed turns and hanging up

• {"type": "text", "text": "..."} is a typed turn. It counts like speech, and cuts in if Duotone is talking.
• {"type": "stop"} hangs up. The server answers done with the call's usage and closes the socket.

Billing and limits

• A call is billed by the parts it uses: seconds of audio heard, characters spoken, and the language model's tokens, unless you bring your own model. done reports all four.
• Calls end after 10 minutes. Each takes one live session, as a transcription socket does; when none is free, the call is refused with busy.
• Turn-taking: a turn ends about 0.35–0.7 s after the caller stops. “Stop” or “wait” on their own end the reply without starting a new one.

Build your own

To run turn-taking or the language model yourself, put the pieces together on your server. It holds two sockets per call, and a request to your language model for each reply:

caller's audio ──▶ transcription socket ──▶ final transcript
                                                  │
                          your language model ◀───┘
                                  │ reply text, as it is written
                                  ▼
caller hears it ◀── 48 kHz audio ◀── speech socket (one per reply)

Keep both sockets on your server. Your app sends microphone audio up and plays audio down; the API key never leaves the server.

1. Listen

Stream the caller's microphone into realtime speech to text for the whole call, including while the agent speaks, so you can tell when they cut in. A final event is the end of their turn.

• Utterances end after about 0.7 s of quiet. To answer sooner, send {"type": "end"} once your own check says they are done, for example 350 ms after a vad event with speech: false.
• A session takes 60 s of audio. Open the next one a little before, and switch the microphone over to it.
• Drop finals that are empty or only a backchannel (“yeah”, “mm”, “okay”) instead of replying to them.

2. Think

Send the conversation to your language model with streaming on. Two things make the reply sound like Duotone:

• Ask the model for tags. One emotion tag at the start of each reply and a sound or two where a person would make one, from the lists the voice knows; two or three short spoken sentences; no markdown.
• Pass on how they sounded. Each final carries the caller's emotion. Add it to their message as a note, such as {they sound sad, 82%}, and tell the model to let it shape the tone without saying it aloud.

3. Speak

Open a speech socket for each reply and forward the model's text into it as it arrives. The API cuts the text into sentences and voices each as soon as it is complete, so the first words play while the model is still writing.

const tts = await openSocket("wss://tts-api.yoob.com/v1/audio/speech/stream");
tts.send(JSON.stringify({ voice: "w1243", quality: "best", response_format: "pcm" }));
tts.onmessage = (e) => {
  if (typeof e.data !== "string") sendToCaller(e.data);   // 48 kHz s16le, play in order
};

for await (const delta of llm.stream(messages)) tts.send(JSON.stringify({ text: delta }));
tts.send(JSON.stringify({ end: true }));

Ask the model and open the socket at the same time: each takes a few hundred milliseconds, and doing them one after the other adds that to every reply.

4. Interruptions

When the caller talks over the reply, stop it:

• Treat a partial of two or more real words, arriving while the reply plays, as the caller cutting in. Ignore backchannels, and ignore words that match what the agent is saying: that is its own voice coming back through the speakers.
• Fade playback out over about 15 ms, abort the language model request, and close the speech socket, which cancels the rest of the render.
• Keep the full reply in the conversation, and tell the model the caller cut in, for example {they cut in while you were talking}, so it doesn't repeat itself.
• “Stop”, “wait” or “hold on” on their own end the reply without starting a new one.

Latency

What the console's Duotone measures, from the end of the caller's words:

Use "quality": "best" on the speech socket: its first sentence is a single quick take, so it starts as fast as fast and sounds better after that.

Next steps