How to Add an Avatar to the OpenAI Realtime API

If your voice agent already uses the OpenAI Realtime API, Liforma can add the animated character without replacing the realtime speech-to-speech stack. OpenAI continues to handle audio input, conversation state, reasoning, tools and native spoken output. Liforma consumes that output and turns it into the visual avatar experience.

Can you add an avatar to the OpenAI Realtime API?

Yes. Liforma provides two dedicated bridges in @liforma/client/openai:

• connectOpenAiRealtimeWebRtc() for the preferred browser media path; and
• connectOpenAiRealtime() when you want the Realtime WebSocket path and direct PCM control.

Both let OpenAI remain the speech-to-speech agent while Liforma handles the visual character, playback and speech-to-animation.

Why keep OpenAI Realtime intact?

The Realtime API works directly with audio and is designed for low-latency conversational voice agents with natural turn-taking, barge-in and tool use. If your application already relies on those behaviours, decomposing it back into a separate STT → text LLM → TTS pipeline just to add an avatar would be unnecessary.

Liforma can sit after the realtime model instead.

What stays in OpenAI?

• audio input and turn handling;
• Realtime conversation state;
• the model and instructions;
• tool calling;
• native generated speech; and
• your existing Realtime architecture.

What does Liforma add?

• a reusable animated character;
• realtime lip-sync / speech-to-animation;
• appearance, costumes and backdrops;
• browser-native rendering;
• Experience scenes and visual context; and
• the option to use the same character elsewhere in Liforma.

Preferred browser integration: OpenAI Realtime over WebRTC

For browser-based Realtime applications, Liforma's WebRTC helper maps the OpenAI remote audio track directly to a Liforma utterance and uses transcript events on the data channel to improve lip-sync.

import { connectOpenAiRealtimeWebRtc } from '@liforma/client/openai';

const bridge = await connectOpenAiRealtimeWebRtc(experience, {
  ephemeralKey,
  // instructions: 'You are a helpful voice assistant…',
  // captureMic: true
});

// Later:
await bridge.end();

Do not also attach the OpenAI remote track to a separate HTML audio element, or you can end up hearing the agent twice.

See the OpenAI → Liforma integration guide.

Mint the Realtime client secret on your server

The browser should receive an ephemeral Realtime client secret. Your long-lived OpenAI API key belongs on the server, not in frontend JavaScript.

Once the browser receives the ephemeral value, the Liforma bridge can establish the Realtime connection and route the resulting speech into the character.

When should I use the WebSocket helper instead?

Use connectOpenAiRealtime() when your application already terminates Realtime over a WebSocket or you specifically need access to PCM chunks.

import { connectOpenAiRealtime } from '@liforma/client/openai';

const bridge = await connectOpenAiRealtime(experience, {
  ephemeralKey,
  // captureMic: true,
  // mediaStream,
  // instructions,
  // model,
  // voice
});

The helper turns each OpenAI response into a Liforma utterance, writes the audio chunks, forwards transcript information for alignment and cancels avatar playback on unexpected disconnects or interruption.

Can I keep OpenAI tool calling and my existing application logic?

Yes. Liforma does not need to become the agent brain. OpenAI can continue to decide what to say and which tools to call. Liforma receives the resulting spoken response and provides the character layer.

Why use OpenAI Realtime + Liforma instead of Liforma Live?

Use this configuration when OpenAI Realtime itself is important to your product — for example because you want native speech-to-speech behaviour, a specific Realtime model, an existing tool stack, provider credits or a voice experience you have already tuned.

If you do not need those things and primarily want a simple complete interactive avatar, Liforma Live can be substantially simpler because STT, intelligence, TTS and animation are included as one managed path.

Can I use classic OpenAI TTS instead of Realtime?

Yes. Liforma can also play one-shot or streamed audio generated by OpenAI TTS. The provider guide includes a PCM example for this path. But if you are building a live voice agent, the Realtime API bridge is the more direct architecture.

Does Liforma change the OpenAI voice?

No. In the Realtime integration, the OpenAI-generated audio is the authoritative spoken output. Liforma animates that exact speech.

When is this a good fit?

• you already have an OpenAI Realtime voice agent;
• you want to preserve native speech-to-speech interaction;
• you use OpenAI tools and Realtime conversation state;
• you want a visual layer without rebuilding the agent;
• you want an avatar for a browser-based Realtime experience; or
• you want OpenAI speech inside a larger Liforma character experience.