An Expressive (V4) session is two-way: the page can publish the user's microphone and camera, interrupt the agent, and send its own data-channel messages. It can also change the speech-to-text language mid-conversation, so the agent hears the user in whatever language they switch to. This guide covers those methods, what they do on Talks (V2) and Clips (V3) agents, and how to tell which session you are in.
Read the tier off AgentAvatar: agentManager.agent.avatar.type === 'expressive' is the check every branch below is written against.
publishMicrophoneStream() takes a MediaStream and publishes its audio track to the session. Getting the stream is the browser's job, and the permission prompt is yours to time.
const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
await agentManager.publishMicrophoneStream(micStream);
unpublishMicrophoneStream() is the counterpart, and is what to call to mute the user for the rest of the session:
await agentManager.unpublishMicrophoneStream();
replaceMicrophoneTrack() swaps the live track without unpublishing it, which is what a device picker wants. The publication survives the swap — its LiveKit publication id (SID) and SSRC stay the same, though the MediaStreamTrack id changes — so the server sees continuous audio rather than a stop and a restart.
It rejects with a plain Error for four reasons: the room is not connected, the track is not an audio track, a publish is already in flight, or nothing is published to replace. Only the last is worth falling back on, and the message is what distinguishes it — so do not republish from a bare catch.
async function useInputDevice(deviceId: string) {
const stream = await navigator.mediaDevices.getUserMedia({ audio: { deviceId } });
const [track] = stream.getAudioTracks();
try {
await agentManager.replaceMicrophoneTrack(track);
} catch (error) {
if (!(error instanceof Error) || !error.message.includes('No microphone publication')) {
throw error; // Not connected, not audio, or a publish already in flight.
}
// Nothing published yet — publish instead of swapping.
await agentManager.publishMicrophoneStream(stream);
}
}
publishCameraStream() and unpublishCameraStream() are the same pair for video. Vision has to be enabled on the agent for the frames to be used — Agent.vision says whether it is, so check it before offering the control.
if (agentManager.agent.vision?.enabled) {
const cameraStream = await navigator.mediaDevices.getUserMedia({ video: true });
await agentManager.publishCameraStream(cameraStream);
}
Two methods, and they answer different questions.
getSttToken() fetches a short-lived token for D-ID's speech-to-text service, for an application that runs its own transcription. It is not an Expressive (V4) method: the request goes out whenever it is called, connected or not and whatever the avatar type, and the service decides whether to issue a token for the agent. It rejects with an HttpError when it does not.
setSttLanguage() switches the language the session transcribes in, in the middle of a conversation. It takes a language name or a BCP-47 code, and it is Expressive (V4) only, after connect().
await agentManager.setSttLanguage('en-US');
await agentManager.setSttLanguage('Spanish');
A room that has dropped since connect() reports a StreamError through onError and the promise still resolves — so the failure to watch for is on the callback, not on the await.
Interrupting is a property of the stream, not of the avatar tier: it needs a fluent stream, which means any Expressive (V4) agent, or a Clips (V3) agent built on a Pro avatar that asked for it — fluent is off unless you set it. Two different questions decide whether to show a stop button and whether to enable it.
true on a fluent stream once connected. Ask it once the connection is up, to decide whether the control belongs on screen.false while a blocking client tool call is outstanding, because the agent is suspended waiting for it, and back to true when the call finishes.interrupt() itself never throws. On every avatar type it returns without doing anything when interrupting is not available, when it is not allowed right now, and — on Talks (V2) and Clips (V3) agents — when the stream is not fluent or no video is playing.
import { type AgentManagerCallbacks } from '@d-id/client-sdk';
const callbacks: AgentManagerCallbacks = {
onSrcObjectReady(value) {
videoElement.srcObject = value;
},
onConnectionStateChange(state) {
if (state === 'connected') {
setStopButtonVisible(agentManager.isInterruptAvailable());
}
},
onInterruptibleChange(interruptible) {
setStopButtonEnabled(interruptible);
},
};
stopButton.onclick = () => agentManager.interrupt();
// The user typed over the answer instead of pressing the button. Expressive (V4) agents
// drop `text` interrupts, because the orchestrator does not cancel the answer in flight.
composer.onInput = () => agentManager.interrupt({ type: 'text' });
The interrupted message is marked interrupted in the next onNewMessage — but only when an interrupt was actually sent. A call that finds nothing to interrupt leaves the transcript untouched and fires no callback, so do not use the callback as an acknowledgement.
sendDataChannelMessage() sends a JSON payload to the agent on one of the topics in DataChannelTopic — the presentation topic tells a deck to change slide, for instance. Expressive (V4) agents only, after connect().
import { DataChannelTopic } from '@d-id/client-sdk';
await agentManager.sendDataChannelMessage(DataChannelTopic.Presentation, {
type: 'navigate',
slide: 3,
});
// The topic's plain string value is accepted too.
await agentManager.sendDataChannelMessage('did.presentation', { type: 'navigate', slide: 4 });
As with setSttLanguage(), a dropped room surfaces as a StreamError on onError while the promise still resolves.
The methods do not all fail the same way, and the difference is deliberate: the ones that add something reject, and the ones that remove something succeed, so a teardown path never has to guard on the avatar type.
| Method | Talks (V2) / Clips (V3), or before connect() |
|---|---|
publishMicrophoneStream() |
rejects with a ValidationError |
replaceMicrophoneTrack() |
rejects with a ValidationError |
publishCameraStream() |
rejects with a ValidationError |
setSttLanguage() |
rejects with a ValidationError |
sendDataChannelMessage() |
rejects with a ValidationError |
registerClientTool() |
throws a ValidationError (synchronously, and before connect() too) |
unpublishMicrophoneStream() |
resolves, having done nothing |
unpublishCameraStream() |
resolves, having done nothing |
interrupt() |
returns, having done nothing |
getSttToken() |
works — it is not an Expressive-only method |
if (agentManager.agent.avatar.type === 'expressive') {
const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
await agentManager.publishMicrophoneStream(micStream);
}
// No guard needed: this is a no-op wherever there is nothing published.
await agentManager.unpublishMicrophoneStream();