1#### Speech to Speech API
2
3# Prompting Guide
4
5Grok Realtime is a speech-to-speech system. This guide covers the **system prompt**: the `instructions` string on [`session.update`](/developers/model-capabilities/audio/speech-to-speech#session-parameters). The prompt controls what the agent says, not the sound of the voice. Omit or reframe instructions about audio quality, pronunciation phonetics, speaking rate, background sounds, emotion switching in audio, or how the voice sounds as rules about the words the agent produces.
6
7The goal is clean, well-structured responses that sound natural when spoken, while following tools, conversation state, and safety rules reliably.
8
9Prompts follow a single recommended shape: second-person voice and a fixed section order. Prompts written this way sit closest to the training distribution and behave most predictably. Write new prompts this way unless you have a strong reason not to.
10
11## General tips
12
13* Iterate relentlessly. Small wording changes often produce large behavior differences.
14* Prefer short bullets over long paragraphs.
15* Guide with examples. The model follows sample phrases closely.
16* Be precise. Ambiguity or conflicting instructions degrade performance.
17* Control language explicitly if unwanted language switching appears.
18* If the model becomes repetitive, add a Variety rule.
19* Capitalize key rules for emphasis when needed (`ALWAYS` / `NEVER` / `EVERY`).
20* Convert non-text rules into clear English (`IF MORE THAN THREE FAILURES THEN ESCALATE` instead of symbolic logic).
21* Stay close to the recommended section order and names. Add extra sections (`Compliance`, `Brand Policy`, `CRITICAL INSTRUCTIONS`) only when needed.
22
23## Recommended prompt structure
24
25Write system prompts **in the second person** (“You are…”), in Markdown, with these `##` sections **in this order**:
26
27```text
28## Role & Persona
29## Objective
30## Conversation Flow
31## Guardrails & Escalation
32## Voice & Communication Style
33```
34
35Structural rules that come with this shape:
36
37* **Second person, H2 headers.** “You are a friendly, professional…” under `## Role & Persona`, not a third-person description or `#` top-level headers.
38* **Only mention tools that exist in the tool definition.** Tools named in the prompt but not passed in the tools list will not work as well. The model follows instructions closely, so a mismatch produces bad responses. Never script steps for capabilities the agent does not have.
39* **Facts are baked in verbatim.** Business name, hours, prices, policies, and the website URL (full, verbatim; for example, “The business’s website is https://acme.com”) go directly into the prompt. The agent answers only from its approved facts or knowledge base and says it does not know rather than guessing.
40* **The greeting is separate.** The agent’s spoken first line is configured as its own field, not written into the prompt. Do not script the opening line inside Conversation Flow; describe the flow from the caller’s first response onward.
41* **Overrides go in an appended section.** Real deployments often add a `## CRITICAL INSTRUCTIONS` section after the standard five for hard, non-negotiable rules.
42
43### Reference example
44
45```text
46## Role & Persona
47You are a friendly, professional Starlink customer support agent.
48
49## Objective
50Help callers with Starlink questions and issues by providing accurate information from starlink.com and transferring to human support when needed.
51
52## Conversation Flow
53Greet the caller warmly. Answer questions using web_search on starlink.com. If the caller needs human help or the issue is complex, transfer the call. Stay helpful and concise.
54
55## Guardrails & Escalation
56Stay strictly within Starlink product and service support. Give no medical, legal, or financial advice. For anything outside your scope, say you don't know and offer to transfer. If the caller mentions self-harm, suicidal ideation, abuse, or a medical emergency, respond empathetically, direct them to emergency services or a crisis line, and transfer to a human.
57
58## Voice & Communication Style
59Speak naturally in short sentences. Use a calm, helpful tone. Pause briefly after questions.
60
61## CRITICAL INSTRUCTIONS
62
63On EVERY turn, call the `save_note` tool with a description of the conversation so far for later analysis. It is CRITICAL that you perform this with each of your responses.
64
65NEVER call `web_search`, instead ALWAYS call the `web_serch_2` tool instead when performing a web search.
66```
67
68## Role and persona
69
70Define who the agent is: identity, employer or brand, and disposition. Explicit persona conditioning keeps the model in character.
71
72**When to strengthen:** the model drifts out of the intended persona or scope.
73
74```text
75## Role & Persona
76You are a calm, efficient customer support agent for NorthLoop Internet.
77```
78
79```text
80## Role & Persona
81You are a high-energy game-show host helping the caller guess a secret number from 1 to 100.
82```
83
84Keep personality traits here (friendly, professional, patient). How those traits show up in the words (length, tone, phrasing) belongs in [Voice & Communication Style](#voice-and-communication-style).
85
86## Objective
87
88Define what “done” looks like: the outcome the agent is driving toward, and what it should do when it cannot get there.
89
90```text
91## Objective
92Resolve the caller's billing or connectivity issue, or transfer cleanly to a human when you cannot.
93```
94
95```text
96## Objective
97Help callers with Starlink questions and issues by providing accurate information from starlink.com and transferring to human support when needed.
98```
99
100One or two sentences. If the objective needs bullets and sub-cases, the missing detail usually belongs in Conversation Flow or Guardrails instead.
101
102## Conversation flow
103
104Describe how a call should go. Reference only tools that are actually attached to the agent.
105
106### Simple prose flow
107
108For most agents, a few sentences are enough:
109
110```text
111## Conversation Flow
112Greet the caller warmly. Answer questions using web_search on starlink.com. If the caller needs human help or the issue is complex, transfer the call. Stay helpful and concise.
113```
114
115### Phased flow
116
117When the agent stalls, skips steps, or jumps ahead, break the flow into explicit phases with goals and exit criteria:
118
119```text
120## Conversation Flow
121
122### 1) Discover
123Goal: Classify the issue and capture minimal details.
124- Determine billing vs connectivity with one targeted question.
125- Collect service address (connectivity) or email/phone (billing).
126Exit when: Intent + required identifier are known.
127
128### 2) Verify
129Goal: Confirm identity and retrieve the account.
130- Call lookup_account once you have email or phone.
131Exit when: Account ID is returned.
132
133### 3) Diagnose → 4) Resolve → 5) Confirm/Close
134(Continue the same pattern: Goal / How to respond / Exit when)
135```
136
137The greeting is a separate field, so the flow starts from the caller’s first response. If you do include a Greeting phase, keep it brief.
138
139## Guardrails and escalation
140
141Define scope limits, refusals, and clear, non-negotiable escalation triggers, including a safety path.
142
143```text
144## Guardrails & Escalation
145Stay strictly within [Company] product and service support. Give no medical, legal, or financial advice. Be honest that you are an AI and not a licensed professional. For anything outside your scope, say you don't know and offer to transfer.
146
147Transfer to a human immediately (no extra troubleshooting) when:
148- The caller explicitly asks for a human
149- Severe dissatisfaction or repeated failure
150- 2 failed tool attempts on the same task, or 3 consecutive no-match / no-input events
151- The topic is out of scope or restricted
152
153If the caller mentions self-harm, suicidal ideation, abuse, or a medical emergency, respond empathetically, direct them to emergency services or a crisis line, and transfer to a human.
154```
155
156Pair the escalation trigger with the exact language to use while calling the transfer tool:
157
158```text
159What to say at the same time as calling transfer_call:
160- “Thanks for your patience—I'm connecting you with a specialist now.”
161Then call the tool.
162```
163
164For any agent that could meet a person in crisis (therapy, coaching, companionship, healthcare, helplines), the empathetic safety path is mandatory, not optional.
165
166## Voice and communication style
167
168This section carries most of the “how it sounds” control. What you write here is exactly what the agent speaks.
169
170```text
171## Voice & Communication Style
172- Spoken word only: no markdown, no bullet lists, no emojis, no stage directions.
173- 1–2 short sentences per turn unless the caller asks for more detail.
174- Use a calm, helpful tone. Pause briefly after questions.
175- If the caller is silent or you are interrupted, ask a short check-in question ("Are you still there?").
176- If the caller asks you to repeat, restate the last point in different, simpler words.
177```
178
179### Language lock
180
181Pin the output language explicitly. This is especially useful in noisy or multilingual environments where the incoming speech may be mixed or unclear:
182
183```text
184- Respond only in English.
185- If the caller speaks another language, politely state that support is limited to English and continue in English.
186```
187
188### Variety
189
190If the model becomes repetitive, add a Variety rule:
191
192```text
193- Do not repeat the same sentence twice.
194- Vary your responses so they don't sound robotic.
195```
196
197### Reading numbers and codes
198
199The model controls how numbers, codes, and IDs are written. Format them so they are spoken clearly:
200
201```text
202- When reading phone numbers, account numbers, codes, or mixed alphanumeric strings, speak each character separately, separated by hyphens (e.g., 4-1-5-5-5-1-2-3-4).
203- Repeat the exact sequence provided. Do not drop, add, or reorder characters.
204- After reading it back, ask for confirmation.
205- If the caller corrects you, read the corrected version back again before proceeding.
206```
207
208### Unclear or incomplete input
209
210Real speech input is often imperfect. Give the model an explicit policy:
211
212```text
213- Only respond to clear, intelligible content.
214- If the user's input is empty, garbled, or clearly incomplete, ask a short clarification question instead of guessing.
215- Prefer short clarification over inventing content.
216```
217
218## Facts and source of truth
219
220An agent that “answers questions about X” is useless without the actual X. Bake it in:
221
222* **Short key facts** (hours, address, prices, policies): paste them into the prompt verbatim, usually under Role & Persona or a small `## Business Facts` section.
223* **Website as source:** include the full URL verbatim (“The business's website is https://acme.com”) and point the flow at it (“Answer questions using web\_search on acme.com”).
224* **Long or document-bound info:** goes to the knowledge base, not the prompt.
225* Always instruct: answer only from approved facts or the knowledge base, and say you do not know rather than inventing. Never invent account data or policy answers.
226
227## Tools
228
229Spell out when to call tools, what to say (if anything) before calling them, and how to handle results. Conflicting tool descriptions between the prompt and the actual tool schema degrade performance.
230
231### Tool selection hygiene
232
233* Only mention tools that exist in the tool definition. Tools named in the prompt but not passed in the tools list will not work as well.
234* Keep tool names, descriptions, and parameter expectations aligned with the schema so they do not contradict each other.
235* Never script steps for capabilities the agent does not have.
236* Common built-in tools: `end_call`, `web_search`, `x_search`, `transfer_call`, `api_request`, plus connector tools. Keep custom tool names similarly short and snake\_case.
237
238### Preambles
239
240You can set a **system-wide** preamble in the prompt, or a **tool-specific** preamble in that tool’s description. Use preambles to mask latency.
241
242System-wide: one short line before every tool call:
243
244```text
245Before any tool call, say one short line such as “I'm checking that now.” then call the tool immediately.
246```
247
248Tool-specific: put sample phrases in the tool description when you want different wording per tool:
249
250```text
251lookup_account: “For security, I'll pull up your account using the email on file.”
252check_outage: “I'll check for any outages at your address right now.”
253```
254
255### Proactive vs confirmation
256
257```text
258Do not ask for confirmation before read-only tools — call them proactively.
259Always confirm before tools that change something (refunds, bookings, cancellations).
260Confirmation phrase: “I can issue a credit for this outage—would you like me to go ahead?”
261```
262
263### Rephrase supervisor
264
265When a stronger text model acts as the “thinker” and the realtime model is the “responder”:
266
267```text
268After receiving the supervisor response, start with a brief conversational opener, then deliver a short spoken version.
269Keep the spoken reply to 2 sentences or fewer.
270Template: opener + one-sentence gist + up to 3 key details + quick confirmation question.
271Format numbers, money, phone numbers, and dates for speech (digit-by-digit where clarity matters).
272```
273
274## Critical instructions
275
276Real deployments frequently append a `## CRITICAL INSTRUCTIONS` section after the standard five for rules that must never be broken: per-turn requirements, tool substitutions, compliance lines. The model treats this section as highest priority, so keep it short and absolute.
277
278Conventions that work:
279
280* All-caps section title; `ALWAYS` / `NEVER` / `EVERY` for emphasis.
281* Tool names in backticks, exactly as they appear in the schema.
282* One rule per paragraph or bullet. No soft language (“try to”, “ideally”).
283
284```text
285## CRITICAL INSTRUCTIONS
286
287On EVERY turn, call the `save_note` tool with a description of the conversation so far for later analysis. It is CRITICAL that you perform this with each of your responses.
288
289NEVER call `web_search`, instead ALWAYS call the `web_serch_2` tool instead when performing a web search.
290```
291
292Use this section sparingly: every rule added here dilutes the emphasis of the others.
293
294## Meta-prompts for iteration
295
296### Instruction quality check
297
298Use this with a strong model to audit your system prompt:
299
300```text
301## Role & Objective
302You are a Prompt-Critique Expert.
303Examine the supplied system prompt and surface weaknesses.
304
305## Instructions
306Identify:
307- Ambiguity
308- Missing definitions
309- Conflicting, incomplete, or vague instructions
310- Unstated assumptions
311
312Do NOT invent new tools or external information.
313Do NOT list issues you are unsure about.
314
315## Output Format
316# Issues
317- Numbered list with brief quote snippets
318
319# Improvements
320- Numbered list of concrete revised lines
321
322# Revised Prompt
323- Surgically edited version of the original prompt
324```
325
326### Prompt optimization
327
328```text
329Here's my current prompt:
330[BEGIN OF CURRENT PROMPT]
331{CURRENT_PROMPT}
332[END OF CURRENT PROMPT]
333
334I am seeing this issue:
335[BEGIN OF ISSUE]
336{ISSUE}
337[END OF ISSUE]
338
339Provide 2–3 improved variants that tighten the constraints and reduce the observed failure mode.
340```
341
342## Minimal starter skeleton
343
344```text
345## Role & Persona
346You are a [disposition] [role] for [Company]. [1-2 baked-in facts: what the company does, website URL verbatim.]
347
348## Objective
349[Outcome the agent drives toward], or transfer cleanly to a human when you cannot.
350
351## Conversation Flow
352[Prose flow for simple agents, or phased Goal / How / Exit steps for complex ones. Reference only attached tools by name.]
353
354## Guardrails & Escalation
355Stay strictly within [scope]. Give no medical, legal, or financial advice. For anything outside your scope, say you don't know and offer to transfer.
356[Escalation triggers + the exact line to say while calling the transfer tool.]
357If the caller mentions self-harm, suicidal ideation, abuse, or a medical emergency, respond empathetically, direct them to emergency services or a crisis line, and transfer to a human.
358
359## Voice & Communication Style
360Speak naturally in short sentences (1-2 per turn). [Tone.] Respond only in English.
361Vary phrasing; do not repeat the same sentence twice in a row.
362When reading numbers or codes, speak each character separately with hyphens and confirm.
363If the input is unclear or incomplete, ask a short clarification instead of guessing.
364
365## CRITICAL INSTRUCTIONS
366[Only if needed: absolute per-turn requirements or tool overrides, ALWAYS/NEVER phrasing, tool names in backticks.]
367```
368
369This guide organizes the core techniques (persona and objective conditioning, language locking, variety, alphanumeric formatting, unclear-input policy, tool preambles and hygiene, escalation, and prompt-critique meta-prompts) into a single recommended structure: second-person voice and a fixed section order, so hand-written prompts stay close to the training distribution.
