> ## Documentation Index
> Fetch the complete documentation index at: https://assemblyai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Message Sequence
This page is the canonical reference for the streaming WebSocket protocol: every message the client sends, every message the server emits, and the order they appear in. For the field-level schema of each message, see the [Streaming API reference](/docs/streaming/api-spec/streaming-websocket).
## Sequence diagram
```mermaid theme={null}
sequenceDiagram
participant C as Client
participant S as Server
C->>S: WebSocket connect (token, speech_model, sample_rate, ...)
S->>C: Begin
loop while streaming
C->>S: binary audio frames (raw PCM, ~50 ms)
S->>C: SpeechStarted
S->>C: Turn (end_of_turn: false) xN (partials)
S->>C: Turn (end_of_turn: true)
end
C->>S: Terminate
S-->>C: in-flight Turn(s)
S->>C: Termination
S->>C: close (1000)
```
## Messages at a glance
**Client → server**
| Message | Frame type | Purpose |
| --------------------- | ----------- | ---------------------------------------------------------------------- |
| Audio | **Binary** | Raw audio bytes (PCM, no JSON or base64 wrapper). Send \~50 ms chunks. |
| `UpdateConfiguration` | Text (JSON) | Change transcription settings mid-session. |
| `ForceEndpoint` | Text (JSON) | Immediately end the current turn. |
| `KeepAlive` | Text (JSON) | Reset the inactivity timer (only needed with `inactivity_timeout`). |
| `Terminate` | Text (JSON) | End the session. |
**Server → client**
| Message | When | Purpose |
| ----------------- | ------------------------------------ | ------------------------------------------------------------------------------- |
| `Begin` | Once, on connect | Session ID, expiration time, and the configuration the server actually applied. |
| `SpeechStarted` | Per turn (Universal-3.5 Pro only) | Speech detected; precedes the turn's first `Turn` message. |
| `Turn` | Repeatedly | Partial and final transcripts; see the walkthrough below. |
| `SpeakerRevision` | If `speaker_labels=true` | Revised speaker labels for earlier turns. |
| `Heartbeat` | Every 5s if `session_heartbeat=true` | Liveness ping with audio-received and real-time-factor stats. |
| `Termination` | Once, last message | Session totals; nothing follows it. |
| `Error` | On failure, before close | Error code and detail; the connection then closes. |
## Session initialization
When the session begins, you receive a `Begin` message with the session ID, expiration time, and the configuration the server applied to your session.
```json theme={null}
{
"type": "Begin",
"id": "3207b601-2054-48df-ba77-8784dfcf9fb8",
"expires_at": 1772570132,
"configuration": {
"model": "universal-3-5-pro",
"mode": "balanced",
"api_version": "2025-05-12"
}
}
```
`expires_at` is a Unix timestamp (seconds). When it is reached, the server closes the session with error `3008`.
Always check that `configuration.model` matches the `speech_model` you requested. Unrecognized or misspelled query parameters are ignored rather than rejected, so this echo is the fastest way to catch a typo (for example, `speechModel` instead of `speech_model`).
## Sending audio
Audio is sent as **binary WebSocket frames** containing audio bytes in the encoding and sample rate you set at connection time (default: 16 kHz, 16-bit mono PCM). Do not wrap audio in JSON and do not base64-encode it. A text frame that isn't valid JSON is rejected and closes the session.
Compressed audio is supported via Opus and AAC. Opus uses roughly 8× less bandwidth than 16 kHz PCM.
* `encoding=ogg_opus` — an Ogg-encapsulated Opus byte stream (the format produced by ffmpeg, gstreamer, opusenc, and browser MediaRecorder). Binary frames can be arbitrary chunks of the stream; no page or packet alignment is required.
* `encoding=opus` — raw Opus packets without a container. Each binary frame must contain exactly one Opus packet.
* `encoding=aac` — an AAC stream in ADTS framing. Raw AAC packets and MP4/M4A containers are not supported. Binary frames can be arbitrary chunks of the stream; the server buffers partial frames.
For AAC and both Opus encodings, `sample_rate` is ignored because each stream describes its own sample rate.
For PCM, send audio in chunks of roughly 50 ms (800 samples at 16 kHz); for Opus, one 20 ms packet (or page) per frame is typical. You can send audio faster than real time (for example when streaming from a file), but throughput is throttled at \~1.25× real time. If more than five minutes of audio is buffered ahead of processing, the session closes with error `3007`.
```python theme={null}
import websocket
# audio_chunk is raw PCM bytes (e.g. 1600 bytes = 800 samples ≈ 50 ms at 16 kHz mono)
ws.send(audio_chunk, websocket.ABNF.OPCODE_BINARY)
```
```python theme={null}
# audio_iterator yields raw PCM byte chunks; the SDK handles binary framing.
client.stream(audio_iterator)
```
```javascript theme={null}
// audioChunk is a Buffer/Uint8Array of raw PCM bytes
ws.send(audioChunk);
```
```javascript theme={null}
// Pipe any readable byte stream into transcriber.stream(); the SDK handles framing.
Readable.toWeb(audioSource).pipeTo(transcriber.stream());
```
## Speech started
Immediately before the **first `Turn` message of each turn**, the server sends a `SpeechStarted` message. The `timestamp` field is the start of the turn, in milliseconds relative to the beginning of the audio stream. The `confidence` field is the average word confidence of the initial transcript.
```json theme={null}
{
"type": "SpeechStarted",
"timestamp": 1216,
"confidence": 0.987654
}
```
Universal Streaming skips `SpeechStarted` and goes straight to the first `Turn`.
## Partial transcript
As the speaker is talking, the server emits one or more `Turn` messages with `end_of_turn: false`. These are partial transcripts.
```json expandable theme={null}
{
"turn_order": 0,
"turn_is_formatted": false,
"end_of_turn": false,
"transcript": "My name is",
"end_of_turn_confidence": 0,
"words": [
{
"start": 1216,
"end": 1627,
"text": "My",
"confidence": 0.956314,
"word_is_final": false
},
{
"start": 1668,
"end": 2490,
"text": "name",
"confidence": 0.999393,
"word_is_final": false
},
{
"start": 2531,
"end": 3067,
"text": "is",
"confidence": 0.753325,
"word_is_final": false
}
],
"utterance": "",
"type": "Turn"
}
```
The cadence and shape of partials depends on the model. See [Universal-3.5 Pro Streaming](/docs/streaming/getting-started/transcribe-streaming-audio) and [Universal Streaming](/docs/streaming/getting-started/transcribe-streaming-audio) for the details of how each model produces partials.
## End of turn
When the turn ends, the server emits a `Turn` message with `end_of_turn: true` and `turn_is_formatted: true`. This is the last message for this `turn_order` and the transcript is fully formatted.
```json expandable theme={null}
{
"turn_order": 0,
"turn_is_formatted": true,
"end_of_turn": true,
"transcript": "My name is Sonny.",
"end_of_turn_confidence": 1,
"words": [
{
"start": 1216,
"end": 1635,
"text": "My",
"confidence": 0.956583,
"word_is_final": true
},
{
"start": 1676,
"end": 2515,
"text": "name",
"confidence": 0.999199,
"word_is_final": true
},
{
"start": 2556,
"end": 2975,
"text": "is",
"confidence": 0.999535,
"word_is_final": true
},
{
"start": 3016,
"end": 4155,
"text": "Sonny.",
"confidence": 0.316031,
"word_is_final": true
}
],
"utterance": "My name is Sonny.",
"type": "Turn"
}
```
**Universal Streaming differs.** By default (`format_turns=false`), the final arrives with `turn_is_formatted: false`. With `format_turns=true`, you receive two `end_of_turn: true` messages for the same `turn_order`: the unformatted final first, then a formatted final right after. Treat a turn as complete only when both `end_of_turn` and `turn_is_formatted` are `true`, or you'll process every turn twice.
Default (`format_turns=false`) — single unformatted final:
```json expandable theme={null}
{
"turn_order": 0,
"turn_is_formatted": false,
"end_of_turn": true,
"transcript": "my name is sonny",
"end_of_turn_confidence": 0.812345,
"words": [
{ "start": 1216, "end": 1635, "text": "my", "confidence": 0.956583, "word_is_final": true },
{ "start": 1676, "end": 2515, "text": "name", "confidence": 0.999199, "word_is_final": true },
{ "start": 2556, "end": 2975, "text": "is", "confidence": 0.999535, "word_is_final": true },
{ "start": 3016, "end": 4155, "text": "sonny", "confidence": 0.316031, "word_is_final": true }
],
"type": "Turn"
}
```
With `format_turns=true` — the unformatted final above, then a formatted final right after:
```json expandable theme={null}
{
"turn_order": 0,
"turn_is_formatted": true,
"end_of_turn": true,
"transcript": "My name is Sonny.",
"end_of_turn_confidence": 0.812345,
"words": [
{ "start": 1216, "end": 1635, "text": "My", "confidence": 0.956583, "word_is_final": true },
{ "start": 1676, "end": 2515, "text": "name", "confidence": 0.999199, "word_is_final": true },
{ "start": 2556, "end": 2975, "text": "is", "confidence": 0.999535, "word_is_final": true },
{ "start": 3016, "end": 4155, "text": "Sonny.", "confidence": 0.316031, "word_is_final": true }
],
"type": "Turn"
}
```
## Forcing an endpoint
To end the current turn immediately, for example when your own VAD or a push-to-talk button decides the user is done, send a `ForceEndpoint` message:
```python theme={null}
ws.send(json.dumps({"type": "ForceEndpoint"}))
```
```python theme={null}
client.force_endpoint()
```
```javascript theme={null}
ws.send(JSON.stringify({ type: "ForceEndpoint" }));
```
```javascript theme={null}
transcriber.forceEndpoint();
```
The server responds with the `end_of_turn: true` message(s) for the current turn right away, without waiting for silence or punctuation.
## Updating configuration mid-session
You can change transcription settings at any point after `Begin` without reconnecting. Only the fields you include are changed:
```python theme={null}
ws.send(json.dumps({
"type": "UpdateConfiguration",
"end_of_turn_confidence_threshold": 0.8,
"min_end_of_turn_silence_when_confident": 400,
"max_turn_silence": 1500,
"keyterms_prompt": ["Keanu Reeves", "AssemblyAI"],
}))
```
```python theme={null}
client.update_configuration(
end_of_turn_confidence_threshold=0.8,
min_end_of_turn_silence_when_confident=400,
max_turn_silence=1500,
keyterms_prompt=["Keanu Reeves", "AssemblyAI"],
)
```
```javascript theme={null}
ws.send(JSON.stringify({
type: "UpdateConfiguration",
end_of_turn_confidence_threshold: 0.8,
min_end_of_turn_silence_when_confident: 400,
max_turn_silence: 1500,
keyterms_prompt: ["Keanu Reeves", "AssemblyAI"],
}));
```
```javascript theme={null}
transcriber.updateConfiguration({
endOfTurnConfidenceThreshold: 0.8,
minEndOfTurnSilenceWhenConfident: 400,
maxTurnSilence: 1500,
keytermsPrompt: ["Keanu Reeves", "AssemblyAI"],
});
```
There is no acknowledgement message; the new settings apply to audio processed after the update. They do not retroactively affect the turn in progress. Some fields are model-specific (for example `mode`, `prompt`, and `agent_context` are Universal-3.5 Pro only). See [Updating configuration mid-stream](/docs/streaming/updating-configuration-mid-stream) for the full list.
## Keep alive
**`KeepAlive` messages are not required.** By default, sessions remain open until [explicitly terminated](#session-termination) or until the 3-hour maximum session duration is reached.
`KeepAlive` is only relevant if you have configured the `inactivity_timeout` connection parameter, which closes the session after a period of no audio or messages being sent. If you are using `inactivity_timeout` and want to keep the session open during periods where no audio is being sent, send a `KeepAlive` message to reset the inactivity timer:
```python theme={null}
ws.send(json.dumps({"type": "KeepAlive"}))
```
```python theme={null}
client.keep_alive()
```
```javascript theme={null}
ws.send(JSON.stringify({ type: "KeepAlive" }));
```
```javascript theme={null}
transcriber.keepAlive();
```
If the inactivity timeout elapses, the session closes with error `3006` and the message `Session terminated due to inactivity: No messages received for N seconds`.
**Ordering guarantees.** `turn_order` is monotonically increasing, and all messages for a turn are delivered before the first message of the next turn. Within a turn, each `Turn` message supersedes the previous one. Render the latest `transcript`; do not append. A turn is complete on the message where both `end_of_turn` and `turn_is_formatted` are `true`.
## Heartbeat
When you set `session_heartbeat=true` at connection time, or toggle it on with `UpdateConfiguration`, the server sends a `Heartbeat` message every 5 seconds of wall-clock time for the life of the session, during both speech and silence. Use it to confirm the session is alive and to watch how the client's audio delivery compares to real time.
```json theme={null}
{
"type": "Heartbeat",
"total_audio_received_ms": 45000,
"total_duration_ms": 45205,
"realtime_factor": 0.9964,
"max_speech_probability": 0.999954
}
```
* `total_audio_received_ms` is the total audio received since the session started.
* `total_duration_ms` is the wall-clock time since the session started.
* `realtime_factor` is the ratio of audio received to wall-clock time over the window since the previous heartbeat. It sits around 1.0 when the client streams in real time and drops below 1.0 when the client is under-delivering audio.
* `max_speech_probability` is the maximum voice-activity speech probability observed since the previous heartbeat. It runs near 1.0 during speech and near 0.0 during silence.
## Session termination
To end a session, send a `Terminate` message:
```python theme={null}
ws.send(json.dumps({"type": "Terminate"}))
```
```python theme={null}
client.disconnect(terminate=True)
```
```javascript theme={null}
ws.send(JSON.stringify({ type: "Terminate" }));
```
```javascript theme={null}
await transcriber.close();
```
After you send `Terminate`, **keep reading from the WebSocket until you receive the `Termination` message**. The server first flushes any in-flight messages, which can include:
* the final (and formatted) `Turn` for audio you already sent;
* on Universal Streaming, a closing `Turn` with an empty `transcript` for an unfinished turn;
* a `SpeakerRevision` if `speaker_labels=true`. The end-of-session refinement adds approximately 400 ms of latency at session close. See [Revised speaker labels](/docs/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for the message schema and consumption guidance.
Closing the socket as soon as you send `Terminate` silently discards your last transcript.
The `Termination` message is always the last message:
```json theme={null}
{
"type": "Termination",
"audio_duration_seconds": 13,
"session_duration_seconds": 14
}
```
* `audio_duration_seconds` is the total audio processed in the session.
* `session_duration_seconds` is the total wall-clock time the connection was open. **This is the duration the session is billed on.**
After `Termination`, no further messages are sent and the server closes the connection with code `1000`.
Always terminate sessions explicitly. Streaming is [billed per session](/docs/billing-and-pricing#streaming-speech-to-text-billing). Sessions that are not terminated remain open and continue to accrue charges until the server auto-closes them after 3 hours (error code `3008`). See [Common errors](/docs/streaming/common-session-errors-and-closures) for more details.
## Errors
If the session fails at any point, the server sends an `Error` message as a text frame and then closes the connection:
```json theme={null}
{
"type": "Error",
"error_code": 3007,
"error": "Audio transmission rate exceeded: too much audio buffered"
}
```
See [Common errors](/docs/streaming/common-session-errors-and-closures) for the full list of error codes and how to handle reconnection.