Model Capabilities

Speech to Text

Transcribe audio files into text with a single API call, or stream audio in real time over WebSocket. The API supports 12 audio formats, word-level timestamps, multichannel transcription, and text formatting.

Quick Start

Transcribe an audio file with a single API call:

curl -X POST https://api.x.ai/v1/stt \
  -H "Authorization: Bearer $XAI_API_KEY" \
  -F model=grok-voice-transcribe-2.0 \
  -F format=true \
  -F language=en \
  -F "keyterm=Understand The Universe" \
  -F file=@audio.mp3

import os
import requests

response = requests.post(
    "https://api.x.ai/v1/stt",
    headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},
    files={"file": ("audio.mp3", open("audio.mp3", "rb"), "audio/mpeg")},
    data=[
        ("model", "grok-voice-transcribe-2.0"),
        ("format", "true"),
        ("language", "en"),
        ("keyterm", "Understand The Universe"),
    ],
)
response.raise_for_status()

result = response.json()
print(result["text"])
print(f"Duration: {result['duration']}s")
for word in result.get("words", []):
    print(f"  {word['start']:.2f}s - {word['end']:.2f}s: {word['text']}")

import fs from "fs";

const formData = new FormData();
formData.append("model", "grok-voice-transcribe-2.0");
formData.append("format", "true");
formData.append("language", "en");
formData.append("keyterm", "Understand The Universe");
formData.append("file", new Blob([fs.readFileSync("audio.mp3")]), "audio.mp3");

const response = await fetch("https://api.x.ai/v1/stt", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.XAI_API_KEY}`,
  },
  body: formData,
});

if (!response.ok) throw new Error(`STT error ${response.status}`);

const result = await response.json();
console.log(result.text);
console.log(`Duration: ${result.duration}s`);
for (const word of result.words ?? []) {
  console.log(`  ${word.start.toFixed(2)}s - ${word.end.toFixed(2)}s: ${word.text}`);
}

Note: The file parameter must be provided after all other parameters in the multipart form.

Get API Key →

Live Voice Demos

Model Selection

Pass model on the REST form or as a WebSocket query parameter.

Supported Languages

The language parameter enables formatting for the following languages. The model transcribes speech in any of these languages regardless of the language parameter — setting it enables formatting of numbers, currencies, and units into their written form.

Request Body

The request uses multipart/form-data. Either file or url must be provided.

† Either file or url must be provided.

Option fields should precede file in the multipart body — for streamable uploads, fields sent after file may be ignored.

Example with text formatting

curl -X POST https://api.x.ai/v1/stt \
  -H "Authorization: Bearer $XAI_API_KEY" \
  -F model=grok-voice-transcribe-2.0 \
  -F format=true \
  -F language=en \
  -F "keyterm=Understand The Universe" \
  -F file=@meeting.mp3

The file parameter must be provided after all other parameters in the multipart form.

Response

The response includes the full transcript, audio duration, and word-level timestamps.

{
  "text": "The balance is $167,983.15.",
  "language": "en",
  "duration": 3.45,
  "words": [
    { "text": "The", "start": 0.24, "end": 0.48 },
    { "text": "balance", "start": 0.48, "end": 0.96 },
    { "text": "is", "start": 0.96, "end": 1.12 },
    { "text": "$167,983.15.", "start": 1.12, "end": 3.20 }
  ]
}

Supported Audio Formats

Container formats (auto-detected)

Raw formats (require audio_format and sample_rate)

Limits

• Max file size: 500 MB
• Channels: Mono, stereo, or up to 8 channels (with multichannel=true)
• Sample rates: 8000, 16000, 22050, 24000, 44100, 48000 Hz

Streaming Speech-to-Text (WebSocket)

For real-time transcription, use the WebSocket API at wss://api.x.ai/v1/stt. The client streams raw audio as binary WebSocket frames and receives JSON transcript events as the audio is processed.

Endpoint: wss://api.x.ai/v1/stt

Configuration is done via URL query parameters — no setup message required. Audio is sent as raw binary frames (no base64 encoding).

[!NOTE]

Never expose your API key in client-side code. Always proxy WebSocket connections through your backend.

Query Parameters

Server Events

The transcript.partial event uses is_final and speech_final to convey three states:

Client Messages

• Binary frames — raw audio in the specified encoding (streamed in real-time-paced chunks, e.g. 100 ms). With encoding=opus, each binary frame must contain exactly one Opus packet — see Opus Streaming.
• {"type": "finalize"} — force the current utterance to finalize as speech_final immediately for PTT
• {"type": "audio.done"} — signal end of audio, triggers transcript.done

transcript.done tells the server no more audio will be sent and to flush the remaining transcript and close the websocket.

Push-to-talk example (finalize on button release, then continue the session):

{"type": "Finalize"}

Per-channel finalize (multichannel only — e.g. agent on channel 0):

{"type": "Finalize", "channel": 0}

Opus Streaming

Set encoding=opus to stream compressed audio instead of raw PCM — roughly 4 KB/s versus 48 KB/s for PCM16 at 24 kHz, with no client-side resampling.

How it works:

• Send exactly one raw Opus packet per binary WebSocket frame. Opus packets don't mark their own boundaries — the WebSocket framing does — so never concatenate packets or split one across frames.
• Omit sample_rate — Opus packets don't carry one, and the parameter is ignored for this encoding. Encode at whatever rate your audio source produces.
• multichannel is not supported with Opus — mono only.
• Send raw packets, not containers. To transcribe an Ogg-Opus or WebM file, use the batch endpoint instead — it auto-detects containers.
• If a frame can't be decoded, the server sends an error event and closes the session. Misframed packets often decode as noise rather than an error, so double-check that your encoder emits one whole packet per frame.

Example URL:

wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&encoding=opus&interim_results=true

Typical use case: Dictation and live transcription from mobile or other bandwidth-constrained clients, where streaming raw PCM is wasteful. Most platform audio APIs and WebRTC stacks produce Opus packets natively.

Multichannel Streaming

When multichannel=true and channels ≥ 2, the server transcribes each audio channel independently. Send interleaved multichannel PCM (e.g. L,R,L,R,… for stereo) as binary frames, and the server de-interleaves and processes each channel in parallel.

How it works:

• transcript.created is sent once (session-level — no channel_index).
• transcript.partial events include a channel_index field (0-based) identifying the source channel. Events from different channels arrive interleaved.
• transcript.done is sent once per channel after audio.done, each with its own channel_index.
• Client Finalize without channel finalizes every channel at once; {"type": "Finalize", "channel": N} finalizes only channel N (0-based).
• Chunk sizes should account for all channels — e.g. for stereo PCM16 at 16 kHz, 100 ms = 6,400 bytes (3,200 per channel × 2 channels).

Example URL:

wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&multichannel=true&channels=2&interim_results=true

Typical use case: Call center recordings with agent on channel 0 and customer on channel 1, enabling per-speaker transcription without requiring speaker diarization.

Smart Turn

Smart Turn uses a lightweight ML model to predict whether the speaker has finished their thought during silence pauses, reducing false endpointing on mid-sentence pauses (e.g. while dictating numbers or thinking between clauses).

How it works:

• When enabled via smart_turn=<threshold>, the model evaluates accumulated audio at each VAD silence boundary.
• If the end-of-turn confidence exceeds the threshold, speech_final=true fires normally.
• If confidence is below the threshold, the event is demoted to chunk_final (is_final=true, speech_final=false) — the transcript text is locked but the utterance continues.
• Every transcript.partial event includes an end_of_turn_confidence field (0.0–1.0) when Smart Turn is enabled.
• During active speech, end_of_turn_confidence is 0.0 (the model only runs at silence boundaries).

Silence timeout (smart_turn_timeout):

When Smart Turn is enabled, the model has full control over when speech_final fires. To prevent sessions from hanging during extended silence (e.g. the user walks away), set smart_turn_timeout to a maximum silence duration in milliseconds (1–5000). If the model keeps predicting "not done" for longer than this duration, speech_final fires anyway as a safety net.

wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&interim_results=true&smart_turn=0.7&smart_turn_timeout=3000

Without smart_turn_timeout, the model has unlimited control — speech_final only fires when confidence exceeds the threshold.

Example event with Smart Turn enabled:

{
  "type": "transcript.partial",
  "text": "I will buy two of those, please.",
  "words": [...],
  "is_final": true,
  "speech_final": true,
  "start": 0.0,
  "duration": 2.4,
  "end_of_turn_confidence": 0.983
}

Typical use case: Voice assistants and conversational AI where you want to avoid cutting the user off mid-sentence. Without Smart Turn, a brief pause while dictating a phone number or thinking between clauses triggers speech_final. With Smart Turn, the model holds off until it detects a natural turn ending (confidence 0.95+) while correctly suppressing during mid-sentence pauses (confidence ~0.005).

Full Example

import asyncio
import json
import os

import websockets

API_KEY = os.environ["XAI_API_KEY"]
WS_URL = "wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&interim_results=true&language=en&keyterm=Understand+The+Universe"

async def transcribe_stream(audio_file: str):
    headers = {"Authorization": f"Bearer {API_KEY}"}

    async with websockets.connect(WS_URL, additional_headers=headers) as ws:
        # Wait for server ready signal
        msg = json.loads(await ws.recv())
        assert msg["type"] == "transcript.created"
        print("Server ready")

        # Read raw PCM from a WAV file (skip 44-byte header)
        with open(audio_file, "rb") as f:
            f.read(44)  # Skip WAV header
            chunk_size = 16000 * 2 // 10  # 100ms of PCM16 at 16kHz

            while chunk := f.read(chunk_size):
                await ws.send(chunk)  # Send raw binary — no base64
                await asyncio.sleep(0.1)

        # Signal end of audio
        await ws.send(json.dumps({"type": "audio.done"}))

        # Collect events until transcript.done
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "transcript.partial":
                prefix = "FINAL" if event["is_final"] else "partial"
                print(f"[{prefix}] {event['text']}")
            elif event["type"] == "transcript.done":
                print(f"\nFull transcript: {event['text']}")
                print(f"Duration: {event['duration']}s")
                break

asyncio.run(transcribe_stream("audio.wav"))

import fs from "fs";
import WebSocket from "ws";

const apiKey = process.env.XAI_API_KEY;
const url = "wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&interim_results=true&language=en&keyterm=Understand+The+Universe";

const ws = new WebSocket(url, { headers: { Authorization: `Bearer ${apiKey}` } });

ws.on("open", () => console.log("Connected"));

ws.on("message", (data) => {
  const event = JSON.parse(data);
  switch (event.type) {
    case "transcript.created":
      console.log("Server ready — streaming audio...");
      // Read WAV file, skip 44-byte header, send 100ms chunks
      const audio = fs.readFileSync("audio.wav").slice(44);
      const chunkSize = 3200; // 100ms at 16kHz, 16-bit
      let offset = 0;
      const interval = setInterval(() => {
        if (offset >= audio.length) {
          clearInterval(interval);
          ws.send(JSON.stringify({ type: "audio.done" }));
          return;
        }
        ws.send(audio.slice(offset, offset + chunkSize));
        offset += chunkSize;
      }, 100);
      break;
    case "transcript.partial":
      const prefix = event.is_final ? "FINAL" : "partial";
      console.log(`[${prefix}] ${event.text}`);
      break;
    case "transcript.done":
      console.log(`\nFull transcript: ${event.text}`);
      console.log(`Duration: ${event.duration}s`);
      ws.close();
      break;
  }
});

Use Cases

• Live captions — Real-time subtitles for video calls, meetings, and live streams
• Voice assistants — Transcribe user speech for natural language understanding pipelines
• Call centers — Real-time agent assistance with multichannel per-speaker transcription
• Accessibility — Live transcription for hearing-impaired users
• Voice commands — Low-latency speech-to-action for hands-free interfaces

Tips for Streaming STT

• Use 16 kHz sample rate with PCM encoding (sample_rate=16000&encoding=pcm) — this is the model's native rate and avoids resampling on the server
• Enable interim_results for responsive UX — show transcription as the user speaks
• Use language=en to enable text formatting — numbers and currencies are written in their standard form
• Send 100 ms audio chunks (3,200 bytes at 16 kHz PCM16) for a good balance of latency and efficiency
• Use encoding=opus on bandwidth-constrained clients — ~4 KB/s versus 48 KB/s for raw PCM at 24 kHz. See Opus Streaming
• Wait for transcript.created before sending audio — the server needs to initialize its ASR backend

Error Handling

Related

• Voice Overview — Overview of all xAI voice capabilities
• Text to Speech — Convert text to speech
• API Reference — Speech to text — Full REST endpoint specification
• API Reference — Streaming — WebSocket streaming specification