2 2
3> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.3> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.
4 4
5These recipes support the [computer use guide](https://developers.openai.com/api/docs/guides/tools-computer-use). Use the sections you need to connect the tool to your environment or expose an existing browser or desktop interface.5Use these recipes to connect the [computer use tool](https://developers.openai.com/api/docs/guides/tools-computer-use) to a browser or desktop you manage. They cover environment setup, action handling, screenshots, and user consent.
6
7For a first hands-on exercise, [try the booking demo](https://developers.openai.com/api/docs/guides/tools-computer-use#see-and-approve-changes) in Python or Node. Select the local rehearsal to fill a fictional reservation, approve submission, and check the saved result in a real local browser. Rehearsal uses scripted choices and needs no API key. It tests the sample's own driver. To adapt it, follow [Check a custom driver locally](https://developers.openai.com/api/docs/guides/tools-computer-use-sdk-drivers#check-a-custom-driver-locally) and add checks for your environment.
8
9If you use the SDK helpers, the [driver reference](https://developers.openai.com/api/docs/guides/tools-computer-use-sdk-drivers) covers driver methods, approvals, session cleanup, and failure handling. The direct API recipes below show how to manage the action loop yourself.
10
11## Choose a hosted environment
12
13See [hosted environment options](https://developers.openai.com/api/docs/guides/tools-computer-use#connect-a-hosted-environment) for partner guides.
14
15## Watch and control a local desktop
16
17Run this on your own computer with a local Docker engine and Docker Compose. It starts a Linux desktop with an editor. You can see and control the desktop in your browser. The initial exercise uses `ComputerToolset.execute` with fixed actions and needs no API key. The optional live run lets the Responses computer tool choose actions.
18
191. Create an empty directory. Save these two files there.
20
21Dockerfile
22
23```dockerfile
24FROM debian:bookworm-slim
25ENV DEBIAN_FRONTEND=noninteractive
26ENV DISPLAY=:99
27
28RUN apt-get update && apt-get install -y --no-install-recommends \
29 xfce4 \
30 dbus-x11 \
31 xvfb \
32 xdotool \
33 imagemagick \
34 x11-utils \
35 firefox-esr \
36 && rm -rf /var/lib/apt/lists/* \
37 && useradd --create-home --shell /bin/sh myuser
38
39USER myuser
40WORKDIR /home/myuser
41
42USER root
43RUN apt-get update && apt-get install -y --no-install-recommends \
44 mousepad novnc websockify x11vnc \
45 && rm -rf /var/lib/apt/lists/*
46USER myuser
47RUN touch /home/myuser/dispatch.txt
48CMD set -eu; \
49 Xvfb :99 -screen 0 1280x800x24 -nolisten tcp & \
50 display_pid=$!; \
51 until xdpyinfo -display :99 >/dev/null 2>&1; do \
52 kill -0 "$display_pid"; sleep 0.2; \
53 done; \
54 dbus-run-session -- startxfce4 & \
55 xdotool search --sync --onlyvisible --class xfce4-panel >/dev/null; \
56 mousepad /home/myuser/dispatch.txt & \
57 exec x11vnc -display :99 -listen 0.0.0.0 -forever -shared -nopw -rfbport 5900
58```
59
60
61compose.yaml
62
63```plain
64services:
65 desktop:
66 build:
67 context: .
68 image: cua-visual-desktop-demo
69 container_name: cua-visual-desktop
70 init: true
71 networks: [desktop]
72 cap_drop: [ALL]
73 security_opt: [no-new-privileges:true]
74 pids_limit: 256
75 mem_limit: 2g
76 cpus: 2
77 shm_size: 512m
78 viewer:
79 image: cua-visual-desktop-demo
80 pull_policy: never
81 command: ["websockify", "--web=/usr/share/novnc", "6080", "desktop:5900"]
82 depends_on: [desktop]
83 init: true
84 ports:
85 - "127.0.0.1:6080:6080"
86 networks: [desktop, viewer]
87 cap_drop: [ALL]
88 security_opt: [no-new-privileges:true]
89 pids_limit: 64
90 mem_limit: 512m
91 cpus: 1
92networks:
93 desktop:
94 internal: true
95 viewer: {}
96```
97
98
992. Start the desktop from that directory:
100
101```bash
102docker compose up --build -d
103```
104
105Open [the local desktop](http://127.0.0.1:6080/vnc.html?autoconnect=1&resize=scale). Wait until `dispatch.txt` appears in the editor. This is the live Linux desktop, so you can type in the window yourself.
106
1073. Install one SDK on the host:
108
109```bash
110# Python 3.11+
111python3 -m venv .venv
112.venv/bin/python -m pip install openai==3.31.0
113
114# Or Node.js 22.6+
115npm install openai@7.33.0
116```
117
118On Windows, use `py -3` instead of `python3` and `.venv\Scripts\python.exe` instead of `.venv/bin/python`.
119
120Save the matching sample as `dispatch.py` or `dispatch.mjs` in the same directory.
121
122Write and verify a dispatch note
123
124```javascript
125import { execFile } from "node:child_process";
126import { promisify } from "node:util";
127import { setTimeout as delay } from "node:timers/promises";
128import OpenAI, { ComputerToolset } from "openai";
129
130const note = "Dispatch #1042: ready for pickup.";
131const exec = promisify(execFile);
132
133async function desktop(...args) {
134 const { stdout } = await exec(
135 "docker",
136 ["exec", "cua-visual-desktop", ...args],
137 {
138 encoding: "buffer",
139 timeout: 10_000,
140 maxBuffer: 4 * 1024 * 1024,
141 }
142 );
143 return stdout;
144}
145
146class EditorDriver {
147 async screenshot() {
148 const image = await desktop("import", "-window", "root", "png:-");
149 return {
150 type: "computer_screenshot",
151 image_url: "data:image/png;base64," + image.toString("base64"),
152 };
153 }
154
155 async focusEditor() {
156 const windows = await desktop(
157 "xdotool",
158 "search",
159 "--onlyvisible",
160 "--class",
161 "mousepad"
162 );
163 const window = windows.toString().trim().split("\n").at(-1);
164 if (!window) throw new Error("The dispatch editor is not open.");
165 await desktop("xdotool", "windowactivate", "--sync", window);
166 }
167
168 async type(action) {
169 if (action.text !== note)
170 throw new Error("Only the dispatch note is allowed.");
171 await this.focusEditor();
172 await desktop(
173 "xdotool",
174 "type",
175 "--clearmodifiers",
176 "--delay",
177 "25",
178 "--",
179 action.text
180 );
181 }
182
183 async keypress(action) {
184 const keys = action.keys.join("+");
185 if (keys !== "CTRL+A" && keys !== "CTRL+S") {
186 throw new Error("Only Select All and Save are allowed.");
187 }
188 await this.focusEditor();
189 await desktop("xdotool", "key", "--clearmodifiers", keys.toLowerCase());
190 }
191
192 refuse(action) {
193 throw new Error(`This exercise does not allow ${action.type}.`);
194 }
195
196 click = this.refuse;
197 double_click = this.refuse;
198 drag = this.refuse;
199 move = this.refuse;
200 scroll = this.refuse;
201 wait = this.refuse;
202}
203
204const tools = new ComputerToolset(new EditorDriver());
205try {
206 if (process.argv.includes("--live")) {
207 const existing = (await desktop("cat", "/home/myuser/dispatch.txt"))
208 .toString()
209 .trim();
210 if (existing === note) {
211 throw new Error(
212 "Clear the note in the editor and save before a live run."
213 );
214 }
215 const client = new OpenAI({ timeout: 30_000, maxRetries: 0 });
216 await tools.run(
217 client,
218 {
219 model: "gpt-6.1-sol",
220 store: false,
221 input:
222 `The open editor has dispatch.txt. Replace its content with exactly "${note}". ` +
223 "Use screenshot, CTRL+A, type, CTRL+S only. The driver focuses the editor.",
224 },
225 { maxSteps: 6 }
226 );
227 } else {
228 await tools.execute({
229 id: "cu_dispatch",
230 call_id: "call_dispatch",
231 type: "computer_call",
232 status: "completed",
233 pending_safety_checks: [],
234 actions: [
235 { type: "keypress", keys: ["CTRL", "A"] },
236 { type: "type", text: note },
237 { type: "keypress", keys: ["CTRL", "S"] },
238 ],
239 });
240 }
241 // Saving is asynchronous; check the file independently of the screenshot.
242 let saved = "";
243 for (let attempt = 0; attempt < 20; attempt += 1) {
244 saved = (await desktop("cat", "/home/myuser/dispatch.txt"))
245 .toString()
246 .trim();
247 if (saved === note) break;
248 await delay(100);
249 }
250 if (saved !== note)
251 throw new Error(`The editor did not save the note: ${saved}`);
252 console.log("Verified:", saved);
253} finally {
254 tools.close();
255}
256```
257
258```python
259import base64
260import subprocess
261import sys
262import time
263
264from openai import ComputerToolset, OpenAI
265from openai.types.responses import ResponseComputerToolCall, computer_action
266
267NOTE = "Dispatch #1042: ready for pickup."
268
269
270def desktop(*args):
271 return subprocess.check_output(
272 ["docker", "exec", "cua-visual-desktop", *args], timeout=10
273 )
274
275
276class EditorDriver:
277 def screenshot(self):
278 image = desktop("import", "-window", "root", "png:-")
279 return {
280 "type": "computer_screenshot",
281 "image_url": "data:image/png;base64," + base64.b64encode(image).decode(),
282 }
283
284 def focus_editor(self):
285 window = desktop(
286 "xdotool", "search", "--onlyvisible", "--class", "mousepad"
287 ).splitlines()[-1]
288 desktop("xdotool", "windowactivate", "--sync", window.decode())
289
290 def type(self, action):
291 if action.text != NOTE:
292 raise ValueError("Only the dispatch note is allowed.")
293 self.focus_editor()
294 desktop(
295 "xdotool", "type", "--clearmodifiers", "--delay", "25", "--", action.text
296 )
297
298 def keypress(self, action):
299 keys = tuple(action.keys)
300 if keys not in (("CTRL", "A"), ("CTRL", "S")):
301 raise ValueError("Only Select All and Save are allowed.")
302 self.focus_editor()
303 desktop("xdotool", "key", "--clearmodifiers", "ctrl+" + keys[1].lower())
304
305 def refuse(self, action):
306 raise ValueError(f"This exercise does not allow {action.type}.")
307
308 click = double_click = drag = move = scroll = wait = refuse
309
310
311tools = ComputerToolset(EditorDriver())
312try:
313 if "--live" in sys.argv:
314 if desktop("cat", "/home/myuser/dispatch.txt").decode().strip() == NOTE:
315 raise RuntimeError(
316 "Clear the note in the editor and save before a live run."
317 )
318 with OpenAI(timeout=30, max_retries=0) as client:
319 tools.run(
320 client,
321 {
322 "model": "gpt-6.1-sol",
323 "store": False,
324 "input": (
325 f'The open editor has dispatch.txt. Replace its content with exactly "{NOTE}". '
326 "Use screenshot, CTRL+A, type, CTRL+S only. The driver focuses the editor."
327 ),
328 },
329 max_steps=6,
330 )
331 else:
332 tools.execute(
333 ResponseComputerToolCall(
334 id="cu_dispatch",
335 call_id="call_dispatch",
336 type="computer_call",
337 status="completed",
338 pending_safety_checks=[],
339 actions=[
340 computer_action.Keypress(type="keypress", keys=["CTRL", "A"]),
341 computer_action.Type(type="type", text=NOTE),
342 computer_action.Keypress(type="keypress", keys=["CTRL", "S"]),
343 ],
344 )
345 )
346 # Saving is asynchronous; check the file independently of the screenshot.
347 for _ in range(20):
348 saved = desktop("cat", "/home/myuser/dispatch.txt").decode().strip()
349 if saved == NOTE:
350 break
351 time.sleep(0.1)
352 if saved != NOTE:
353 raise RuntimeError(f"The editor did not save the note: {saved!r}")
354 print("Verified:", saved)
355finally:
356 tools.close()
357```
358
359
3604. Keep the viewer open and run the sample:
361
362```bash
363.venv/bin/python dispatch.py
364# Or:
365node dispatch.mjs
366```
367
368The SDK selects the editor's text, types a fictional note, saves it, and captures the desktop. It independently reads the guest file and prints `Verified: Dispatch #1042: ready for pickup.` The text should also appear in the viewer. The driver focuses the editor before every action and refuses inputs beyond the fixed note, Select All, and Save. Wait for the command to finish before typing in the viewer again.
369
370
371
372### Run the same task with the Responses API
373
374
375
376Set `OPENAI_API_KEY` on the host using the [API key setup instructions](https://developers.openai.com/api/docs/quickstart#create-and-export-an-api-key). Use a project with access to the [computer tool](https://developers.openai.com/api/docs/guides/tools-computer-use). Clear the editor and save the empty file before testing the live path. The sample refuses to run if the note is already saved, so a previous run cannot pass the check for it.
377
378```bash
379.venv/bin/python dispatch.py --live
380# Or:
381node dispatch.mjs --live
382```
383
384This run uses at most six Responses requests and verifies the file in the same way. It stops if the model requests a safety check or an input this driver does not support. API usage is billed to your project. Only the host SDK reads the API key; screenshots and prompts go to OpenAI.
385
386
387
388
389
3905. Stop and remove the guest after either run, including a failed run:
391
392```bash
393docker compose down
394```
395
396The helper closes when the script exits; the guest stays open so you can inspect the result. `docker compose down` removes the guest, viewer, and their networks. It leaves the local build image and files in your working directory.
397
398Use a local Docker engine on a trusted single-user host. The viewer has no password and listens at `127.0.0.1:6080` on the Docker host. Other local users can access it. Only the viewer connects to both networks; it serves the desktop at the fixed address `desktop:5900`. The guest stays on the internal network with no host files, credentials, or Docker socket mounted. Keep those networks and the port binding for this exercise. For a deployed service, authenticate each session and restrict outbound network access.
399
400If the page will not connect, run `docker compose logs desktop viewer`. If the guest name is in use, stop only your previous demo; do not reuse someone else's desktop. If port 6080 is occupied, free the port or choose a different host port in `compose.yaml` and open that port in your browser. Keep the `127.0.0.1` binding. If the editor is closed or the command fails, run `docker compose down`, start again, and wait for the editor before retrying.
6 401
7## Prepare an environment402## Prepare an environment
8 403
66 461
67 462
68 463
69### Set up a local virtual machine464### Set up a caller-managed Docker container
70 465
71 466
72 467
73For a desktop application, provide a VM or container and translate the returned actions into operating system input events.468This recipe uses a container you manage and translates returned actions into operating system input events. It is separate from the [isolated desktop example](https://developers.openai.com/api/docs/guides/tools-computer-use-sdk-drivers#use-an-isolated-desktop).
74 469
75#### Create a Docker image470#### Create a Docker image
76 471
77The following Dockerfile starts an Ubuntu desktop with `Xvfb`, `x11vnc`, and Firefox:472Save this Dockerfile in an empty directory. It installs a Debian desktop with `Xvfb` and Firefox. The container runs as an ordinary user. The action and screenshot helpers below use `docker exec`, so this recipe does not start a VNC server or publish any ports.
78 473
79Dockerfile474Dockerfile
80 475
81```dockerfile476```dockerfile
82FROM ubuntu:22.04477FROM debian:bookworm-slim
83ENV DEBIAN_FRONTEND=noninteractive478ENV DEBIAN_FRONTEND=noninteractive
479ENV DISPLAY=:99
84 480
85RUN apt-get update && apt-get install -y \481RUN apt-get update && apt-get install -y --no-install-recommends \
86 xfce4 \482 xfce4 \
87 xfce4-goodies \483 dbus-x11 \
88 x11vnc \
89 xvfb \484 xvfb \
90 xdotool \485 xdotool \
91 imagemagick \486 imagemagick \
92 x11-apps \487 x11-utils \
93 sudo \
94 software-properties-common \
95 firefox-esr \488 firefox-esr \
96 && apt-get remove -y light-locker xfce4-screensaver xfce4-power-manager || true \489 && rm -rf /var/lib/apt/lists/* \
97 && apt-get clean && rm -rf /var/lib/apt/lists/*490 && useradd --create-home --shell /bin/sh myuser
98 491
99RUN useradd -ms /bin/bash myuser \
100 && echo "myuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
101USER myuser492USER myuser
102WORKDIR /home/myuser493WORKDIR /home/myuser
103 494
104RUN x11vnc -storepasswd secret /home/myuser/.vncpass495CMD set -eu; \
105 496 Xvfb :99 -screen 0 1280x800x24 -nolisten tcp & \
106EXPOSE 5900497 display_pid=$!; \
107CMD ["/bin/sh", "-c", "\498 until xdpyinfo -display :99 >/dev/null 2>&1; do \
108 Xvfb :99 -screen 0 1280x800x24 >/dev/null 2>&1 & \499 kill -0 "$display_pid"; sleep 0.2; \
109 x11vnc -display :99 -forever -rfbauth /home/myuser/.vncpass -listen 0.0.0.0 -rfbport 5900 >/dev/null 2>&1 & \500 done; \
110 export DISPLAY=:99 && \501 exec dbus-run-session -- startxfce4
111 startxfce4 >/dev/null 2>&1 & \
112 sleep 2 && echo 'Container running!' && \
113 tail -f /dev/null \
114"]
115```502```
116 503
117 504
121docker build -t cua-image .508docker build -t cua-image .
122```509```
123 510
124Run the container:511Run the container without a network:
512
513```bash
514docker run --rm -d --name cua-image \
515 --network none \
516 --cap-drop ALL --security-opt no-new-privileges \
517 --pids-limit 256 --memory 2g --cpus 2 --shm-size 512m \
518 cua-image
519```
520
521Check that the desktop is ready and save its current screen on the host:
125 522
126```bash523```bash
127docker run --rm -it --name cua-image -p 5900:5900 -e DISPLAY=:99 cua-image524docker exec cua-image xdpyinfo -display :99
525docker exec cua-image xdotool search --sync --onlyvisible --class xfce4-panel
526docker exec cua-image import -window root png:- > desktop.png
128```527```
129 528
130Create a helper for shelling into the container:529The second command waits until the desktop panel appears. Open `desktop.png` to view the desktop. Run the screenshot command again after actions to view the latest screen. If the display is still starting, wait a moment and repeat the commands. For a complete Python or Node task, use the [isolated desktop example](https://developers.openai.com/api/docs/guides/tools-computer-use-sdk-drivers#use-an-isolated-desktop).
530
531The container can use its own loopback interface but cannot reach external sites, other containers, or the host network. Do not mount your personal files, credentials, or the Docker socket. For internet tasks, use a VM or an environment with egress rules that allow only the sites required by the task.
532
533Create a helper that passes the command arguments directly to the container:
131 534
132Execute commands on the container535Execute commands on the container
133 536
174import subprocess577import subprocess
175 578
176 579
177def docker_exec(cmd: str, container_name: str, decode: bool = True):580def docker_exec(container_name, executable, args, *, decode=True, env=None):
178 safe_cmd = cmd.replace('"', '\\"')581 environment_args = [
179 docker_cmd = f'docker exec {container_name} sh -c "{safe_cmd}"'582 part
180 output = subprocess.check_output(docker_cmd, shell=True)583 for name, value in (env or {}).items()
181 if decode:584 for part in ("--env", f"{name}={value}")
182 return output.decode("utf-8", errors="ignore")585 ]
183 return output586 output = subprocess.check_output(
587 [
588 "docker",
589 "exec",
590 *environment_args,
591 container_name,
592 executable,
593 *map(str, args),
594 ]
595 )
596 return output.decode("utf-8", errors="ignore") if decode else output
184 597
185 598
186class VM:599class VM:
193```606```
194 607
195 608
609When you finish, stop the container:
610
611```bash
612docker stop cua-image
613```
614
615`--rm` discards the files inside the container when it stops. Screenshots you saved on the host remain until you remove them. This setup is for local experiments on a trusted host. For a deployed service, use per-session authentication, isolated hosts or VMs, resource and egress policies, and audited cleanup.
616
196 617
197 618
198 619
826 await dockerExec(1247 await dockerExec(
827 vm.containerName,1248 vm.containerName,
828 "xdotool",1249 "xdotool",
829 ["type", "--delay", 0, action.text],1250 ["type", "--delay", 0, "--", action.text],
830 { env: { DISPLAY: vm.display } }1251 { env: { DISPLAY: vm.display } }
831 );1252 );
832 break;1253 break;
865 reject_modifiers(action)1286 reject_modifiers(action)
866 button = normalize_xdotool_button(getattr(action, "button", "left"))1287 button = normalize_xdotool_button(getattr(action, "button", "left"))
867 docker_exec(1288 docker_exec(
868 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y} click {button}",
869 vm.container_name,1289 vm.container_name,
1290 "xdotool",
1291 ["mousemove", action.x, action.y, "click", button],
1292 env={"DISPLAY": vm.display},
870 )1293 )
871 case "double_click":1294 case "double_click":
872 reject_modifiers(action)1295 reject_modifiers(action)
873 docker_exec(1296 docker_exec(
874 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y} click --repeat 2 1",
875 vm.container_name,1297 vm.container_name,
1298 "xdotool",
1299 ["mousemove", action.x, action.y, "click", "--repeat", 2, 1],
1300 env={"DISPLAY": vm.display},
876 )1301 )
877 case "drag":1302 case "drag":
878 reject_modifiers(action)1303 reject_modifiers(action)
881 raise ValueError("drag action requires at least two path points")1306 raise ValueError("drag action requires at least two path points")
882 start_x, start_y = path[0]1307 start_x, start_y = path[0]
883 docker_exec(1308 docker_exec(
884 f"DISPLAY={vm.display} xdotool mousemove {start_x} {start_y} mousedown 1",
885 vm.container_name,1309 vm.container_name,
1310 "xdotool",
1311 ["mousemove", start_x, start_y, "mousedown", 1],
1312 env={"DISPLAY": vm.display},
886 )1313 )
887 for x, y in path[1:]:1314 for x, y in path[1:]:
888 docker_exec(1315 docker_exec(
889 f"DISPLAY={vm.display} xdotool mousemove {x} {y}",
890 vm.container_name,1316 vm.container_name,
1317 "xdotool",
1318 ["mousemove", x, y],
1319 env={"DISPLAY": vm.display},
891 )1320 )
892 docker_exec(1321 docker_exec(
893 f"DISPLAY={vm.display} xdotool mouseup 1",
894 vm.container_name,1322 vm.container_name,
1323 "xdotool",
1324 ["mouseup", 1],
1325 env={"DISPLAY": vm.display},
895 )1326 )
896 case "move":1327 case "move":
897 reject_modifiers(action)1328 reject_modifiers(action)
898 docker_exec(1329 docker_exec(
899 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y}",
900 vm.container_name,1330 vm.container_name,
1331 "xdotool",
1332 ["mousemove", action.x, action.y],
1333 env={"DISPLAY": vm.display},
901 )1334 )
902 case "scroll":1335 case "scroll":
903 reject_modifiers(action)1336 reject_modifiers(action)
907 )1340 )
908 1341
909 docker_exec(1342 docker_exec(
910 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y}",
911 vm.container_name,1343 vm.container_name,
1344 "xdotool",
1345 ["mousemove", action.x, action.y],
1346 env={"DISPLAY": vm.display},
912 )1347 )
913 for button in buttons:1348 for button in buttons:
914 docker_exec(1349 docker_exec(
915 f"DISPLAY={vm.display} xdotool click {button}",
916 vm.container_name,1350 vm.container_name,
1351 "xdotool",
1352 ["click", button],
1353 env={"DISPLAY": vm.display},
917 )1354 )
918 case "keypress":1355 case "keypress":
919 keys = "+".join(normalize_xdotool_key(key) for key in action.keys)1356 keys = "+".join(normalize_xdotool_key(key) for key in action.keys)
920 docker_exec(1357 docker_exec(
921 f"DISPLAY={vm.display} xdotool key '{keys}'",
922 vm.container_name,1358 vm.container_name,
1359 "xdotool",
1360 ["key", keys],
1361 env={"DISPLAY": vm.display},
923 )1362 )
924 case "type":1363 case "type":
925 docker_exec(1364 docker_exec(
926 f"DISPLAY={vm.display} xdotool type --delay 0 '{action.text}'",
927 vm.container_name,1365 vm.container_name,
1366 "xdotool",
1367 ["type", "--delay", 0, "--", action.text],
1368 env={"DISPLAY": vm.display},
928 )1369 )
929 case "wait":1370 case "wait":
930 time.sleep(2)1371 time.sleep(2)
1284 await dockerExec(1725 await dockerExec(
1285 vm.containerName,1726 vm.containerName,
1286 "xdotool",1727 "xdotool",
1287 ["type", "--delay", 0, action.text],1728 ["type", "--delay", 0, "--", action.text],
1288 { env: { DISPLAY: vm.display } }1729 { env: { DISPLAY: vm.display } }
1289 );1730 );
1290 break;1731 break;
1315 try:1756 try:
1316 for key in normalized_keys:1757 for key in normalized_keys:
1317 docker_exec(1758 docker_exec(
1318 f"DISPLAY={vm.display} xdotool keydown '{key}'",
1319 vm.container_name,1759 vm.container_name,
1760 "xdotool",
1761 ["keydown", key],
1762 env={"DISPLAY": vm.display},
1320 )1763 )
1321 pressed_keys.append(key)1764 pressed_keys.append(key)
1322 1765
1324 finally:1767 finally:
1325 for key in reversed(pressed_keys):1768 for key in reversed(pressed_keys):
1326 docker_exec(1769 docker_exec(
1327 f"DISPLAY={vm.display} xdotool keyup '{key}'",
1328 vm.container_name,1770 vm.container_name,
1771 "xdotool",
1772 ["keyup", key],
1773 env={"DISPLAY": vm.display},
1329 )1774 )
1330 1775
1331 1776
1338 vm,1783 vm,
1339 getattr(action, "keys", None),1784 getattr(action, "keys", None),
1340 lambda: docker_exec(1785 lambda: docker_exec(
1341 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y} click {button}",
1342 vm.container_name,1786 vm.container_name,
1787 "xdotool",
1788 ["mousemove", action.x, action.y, "click", button],
1789 env={"DISPLAY": vm.display},
1343 ),1790 ),
1344 )1791 )
1345 case "double_click":1792 case "double_click":
1347 vm,1794 vm,
1348 getattr(action, "keys", None),1795 getattr(action, "keys", None),
1349 lambda: docker_exec(1796 lambda: docker_exec(
1350 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y} click --repeat 2 1",
1351 vm.container_name,1797 vm.container_name,
1798 "xdotool",
1799 ["mousemove", action.x, action.y, "click", "--repeat", 2, 1],
1800 env={"DISPLAY": vm.display},
1352 ),1801 ),
1353 )1802 )
1354 case "drag":1803 case "drag":
1359 def do_drag():1808 def do_drag():
1360 start_x, start_y = path[0]1809 start_x, start_y = path[0]
1361 docker_exec(1810 docker_exec(
1362 f"DISPLAY={vm.display} xdotool mousemove {start_x} {start_y} mousedown 1",
1363 vm.container_name,1811 vm.container_name,
1812 "xdotool",
1813 ["mousemove", start_x, start_y, "mousedown", 1],
1814 env={"DISPLAY": vm.display},
1364 )1815 )
1365 for x, y in path[1:]:1816 for x, y in path[1:]:
1366 docker_exec(1817 docker_exec(
1367 f"DISPLAY={vm.display} xdotool mousemove {x} {y}",
1368 vm.container_name,1818 vm.container_name,
1819 "xdotool",
1820 ["mousemove", x, y],
1821 env={"DISPLAY": vm.display},
1369 )1822 )
1370 docker_exec(1823 docker_exec(
1371 f"DISPLAY={vm.display} xdotool mouseup 1",
1372 vm.container_name,1824 vm.container_name,
1825 "xdotool",
1826 ["mouseup", 1],
1827 env={"DISPLAY": vm.display},
1373 )1828 )
1374 1829
1375 with_modifiers(vm, getattr(action, "keys", None), do_drag)1830 with_modifiers(vm, getattr(action, "keys", None), do_drag)
1378 vm,1833 vm,
1379 getattr(action, "keys", None),1834 getattr(action, "keys", None),
1380 lambda: docker_exec(1835 lambda: docker_exec(
1381 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y}",
1382 vm.container_name,1836 vm.container_name,
1837 "xdotool",
1838 ["mousemove", action.x, action.y],
1839 env={"DISPLAY": vm.display},
1383 ),1840 ),
1384 )1841 )
1385 case "scroll":1842 case "scroll":
1390 1847
1391 def do_scroll():1848 def do_scroll():
1392 docker_exec(1849 docker_exec(
1393 f"DISPLAY={vm.display} xdotool mousemove {action.x} {action.y}",
1394 vm.container_name,1850 vm.container_name,
1851 "xdotool",
1852 ["mousemove", action.x, action.y],
1853 env={"DISPLAY": vm.display},
1395 )1854 )
1396 for button in buttons:1855 for button in buttons:
1397 docker_exec(1856 docker_exec(
1398 f"DISPLAY={vm.display} xdotool click {button}",
1399 vm.container_name,1857 vm.container_name,
1858 "xdotool",
1859 ["click", button],
1860 env={"DISPLAY": vm.display},
1400 )1861 )
1401 1862
1402 with_modifiers(vm, getattr(action, "keys", None), do_scroll)1863 with_modifiers(vm, getattr(action, "keys", None), do_scroll)
1403 case "keypress":1864 case "keypress":
1404 keys = "+".join(normalize_xdotool_key(key) for key in action.keys)1865 keys = "+".join(normalize_xdotool_key(key) for key in action.keys)
1405 docker_exec(1866 docker_exec(
1406 f"DISPLAY={vm.display} xdotool key '{keys}'",
1407 vm.container_name,1867 vm.container_name,
1868 "xdotool",
1869 ["key", keys],
1870 env={"DISPLAY": vm.display},
1408 )1871 )
1409 case "type":1872 case "type":
1410 docker_exec(1873 docker_exec(
1411 f"DISPLAY={vm.display} xdotool type --delay 0 '{action.text}'",
1412 vm.container_name,1874 vm.container_name,
1875 "xdotool",
1876 ["type", "--delay", 0, "--", action.text],
1877 env={"DISPLAY": vm.display},
1413 )1878 )
1414 case "wait":1879 case "wait":
1415 time.sleep(2)1880 time.sleep(2)
1826```python2291```python
1827def capture_screenshot(vm):2292def capture_screenshot(vm):
1828 return docker_exec(2293 return docker_exec(
1829 f"export DISPLAY={vm.display} && import -window root png:-",
1830 vm.container_name,2294 vm.container_name,
2295 "import",
2296 ["-window", "root", "png:-"],
2297 env={"DISPLAY": vm.display},
1831 decode=False,2298 decode=False,
1832 )2299 )
1833```2300```
