Emotion recognition

Undertone hears how something was said: the feeling of each utterance with a probability for every emotion, laughs, sighs and breaths, whispered or shouted words, a measured profile of the voice, and whether it is a person or an AI voice. Try it on the Undertone page.

From a file

Ask for response_format=verbose_json with model=yoob-asr-1. Send the file as a multipart upload, as the OpenAI SDKs do, or as the raw request body with its Content-Type.

curl https://tts-api.yoob.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $YOOB_API_KEY" \
  -F model=yoob-asr-1 \
  -F response_format=verbose_json \
  -F file=@clip.wav

A real response, shortened to the fields this page is about:

{
  "duration": 4.4,
  "text": "We still got some work to do and I look forward to getting that work done.",
  "caption": "[emo:joy] We still got some work to do and <shouting> I look forward to getting that work done. </shouting>",
  "segments": [
    {
      "start": 0.0,
      "end": 4.384,
      "caption": "[emo:joy] We still got some work to do and <shouting> I look forward to ... </shouting>",
      "tags": [["<shouting>", 1.84], ["</shouting>", 4.16]],
      "emotion": "joy",
      "emotion_probs": { "joy": 0.943, "sad": 0.031, "surprise": 0.007, "tender": 0.006, "fear": 0.004, "angry": 0.004, ... },
      "voice": { "f0_hz": 89.4, "wpm": 274.3, "snr_db": 15.9, "f0_st_std": 2.42 },
      "p_human": 0.997
    }
  ],
  "speaker": { "gender": {...}, "age": [...], "age_bins": [...], "accent": {...}, "timbre": [...], "seconds": 4.4 },
  "model": "yoob-asr-1"
}

Emotions

Each segment, one utterance, gets an emotion from eight families, or none when there is no clear one:

joy  tender  surprise  neutral  sad  fear  angry  disgust  none

• emotion_probs holds the probability of every family, summing to 1. Use it to set your own threshold, or to track a mix.
• A caption opens with the utterance's feeling as [emo:joy]; another [emo:…] mid-line marks a change.

Sounds and delivery

caption is the transcript marked up with how it sounded; tags lists every sound and span tag with its time in seconds.

Delivery spans include whisper, shouting and fast. The same markup, written by you, makes text to speech perform it, so a caption can be read back in another voice.

Live

On the transcription WebSocket, every partial and final carries emotion, emotion_probs, caption and tags for the utterance so far, so you can react to a caller's feeling while they are still talking.

ws.onmessage = (e) => {
  const m = JSON.parse(e.data);
  if (m.type === "partial" && m.emotion === "angry" && m.emotion_probs.angry > 0.7) escalate(m.utt);
  if (m.type === "final_meta") logVoice(m.voice, m.p_human);   // f0_hz, wpm, snr_db for that utterance
  if (m.type === "speaker") updateProfile(m);                  // the voice profile so far
};

Voice profile

speaker describes the voice from everything heard so far; each segment's voice has its own measurements. Age, gender and accent are estimates from the voice alone.

Limits and price

• Files: up to 60 seconds and 25 MB a request. Live: 60 seconds of audio a session; open another to continue.
• Billed as transcription, per second of audio: $0.055 an hour for files, $0.10 an hour live. The emotion, tags and voice profile cost nothing extra.

Next steps