Computer#

Warning

The computer use interface is experimental and has serious security implications. Please use with caution and see Anthropic’s documentation on computer use for additional guidance.

See How to Automate GUIs with Computer Use for practical recipes: prerequisites, backend selection, web vs. native automation, and the observe-act-verify loop.

Tool for computer interaction for X11 or macOS environments, including screen capture, keyboard, and mouse control.

The computer tool provides direct interaction with the desktop environment. Similar to Anthropic’s computer use demo, but integrated with gptme’s architecture.

Features

Keyboard input simulation
Mouse control (movement, clicks, dragging)
Screen capture with automatic scaling
Cursor position tracking

Installation

On Linux, requires X11 and xdotool:

# On Debian/Ubuntu
sudo apt install xdotool

# On Arch Linux
sudo pacman -S xdotool

On macOS, uses native screencapture and external tool cliclick:

brew install cliclick

You need to give your terminal both screen recording and accessibility permissions in System Preferences.

Configuration

The tool uses these environment variables:

DISPLAY: X11 display to use (default: “:1”, Linux only)
WIDTH: Screen width (default: 1024)
HEIGHT: Screen height (default: 768)
GPTME_COMPUTER_CONFIRM_SENSITIVE: Pre-execution gate for sensitive actions (type, key, left_click_drag, fill_element). Values: - unset / “0”: gate disabled (default, back-compatible) - “1”: gate enabled; interactive sessions prompt the user, non-interactive sessions block - “auto-allow”: gate enabled but approves silently (useful in automated scripts)

Usage

The tool supports these actions:

Keyboard:

key: Send key sequence (e.g., “Return”, “Control_L+c”)
type: Type text with realistic delays

Mouse:

mouse_move: Move mouse to coordinates

left_click: Click left mouse button

right_click: Click right mouse button

middle_click: Click middle mouse button

double_click: Double click left mouse button

triple_click: Triple click at position (selects all text in most native inputs; use with coordinate to click a field)

left_click_drag: Click and drag to coordinates

Screen:

screenshot: Take and view a screenshot

cursor_position: Get current mouse position

wait_for_change: Poll until screen changes, then return one screenshot

Window management:

window_focus: Wait for a window matching a name pattern to appear and focus it

Accessibility (cross-platform):

accessibility_tree: Dump the native accessibility tree for all visible apps. On Linux uses AT-SPI2 (role names like “push button”, “entry”). On macOS uses System Events via AppleScript (role names like “AXButton”, “AXTextField”).

click_accessible_element: Find and click an element by role and name (text=’role:name’). Linux example: text=’push button:Submit’ macOS example: text=’AXButton:Submit’

The tool automatically handles screen resolution scaling to ensure optimal performance with LLM vision capabilities.

Tips for Complex Operations

For complex operations involving multiple keypresses, you can use semicolon-separated sequences with key:

Examples

Filling a login form: t:username;kp:tab;t:password;kp:return
Switching applications: cmd+tab on macOS, alt+Tab on Linux
(macOS) Opening Spotlight and searching: cmd+space;t:firefox;return

Using a single sequence for complex operations ensures proper timing and recognition of keyboard shortcuts.

Instructions

You can interact with the computer through the `computer` Python function.
Works on both Linux (X11) and macOS.

### When to use the computer tool

Use computer for GUI interactions that cannot be done through the shell: clicking
elements in running applications, typing into GUI windows, taking screenshots to
verify visual state, and keyboard shortcuts in desktop apps. Prefer the shell or
tmux over computer for anything that has a CLI equivalent. Use computer when the
task requires direct screen interaction — for example, operating a browser UI,
a desktop app, or an interactive installer that has no headless mode.

The key input syntax works consistently across platforms with:

Available actions:
- key: Send key sequence using a unified syntax:
  - Type text: "t:Hello World"
  - Press key: "return", "esc", "tab"
  - Key combination: "ctrl+c", "cmd+space"
  - Chain commands: "cmd+space;t:firefox;return"
- type: Type text with realistic delays (legacy method)
- mouse_move: Move mouse to coordinates
- left_click, right_click, middle_click, double_click: Mouse clicks
- left_click_drag: Click and drag to coordinates
- scroll: Scroll the mouse wheel at coordinates (text="up"/"down"/"left"/"right")
- screenshot: Take and view a screenshot
- cursor_position: Get current mouse position
- wait_for_change: Wait until the screen changes, then return a single screenshot.
  Loops internally until ≥1% of pixels differ from the initial capture, or the
  timeout (text="<seconds>", default 10) elapses. Returns one screenshot regardless
  of how many internal polls were needed — avoids stacking redundant screenshots in
  the conversation context. Use after triggering an action that produces a visual
  response (page load, dialog open, animation finish).
- window_focus: Wait for a window whose title contains text=<pattern> to appear,
  then focus it. On Linux/X11 this uses xdotool --sync so no screenshot polling
  is needed. Use after opening a new application to avoid guessing where to click.
- accessibility_tree: Dump the native accessibility tree for all open applications.
  On Linux (AT-SPI2): role names like 'push button', 'entry', 'check box'.
    Requires: pip install pyatspi (and AT-SPI2 accessibility stack).
  On macOS (System Events): role names like 'AXButton', 'AXTextField', 'AXCheckBox'.
    Requires Accessibility permission for the terminal in System Preferences.
  Use this to discover element names and roles before using click_accessible_element.
- click_accessible_element: Find and click an element by role and name without
  needing screen coordinates. Use text='role:name' where role is the platform role
  name and name is a substring of the element's accessible name. Examples:
    Linux:  computer('click_accessible_element', text='push button:Submit')
    macOS:  computer('click_accessible_element', text='AXButton:Submit')

### Accessibility-first for native apps

Prefer click_accessible_element over coordinate-based clicks for native apps:

  computer("accessibility_tree")                               # inspect available elements
  # Linux:
  computer("click_accessible_element", text="entry:Username")  # fill username field
  computer("type", text="user@example.com")
  computer("click_accessible_element", text="push button:Log In")
  # macOS:
  computer("click_accessible_element", text="AXTextField:Username")
  computer("type", text="user@example.com")
  computer("click_accessible_element", text="AXButton:Log In")

This is more robust than coordinate guessing: element names don't shift when
window size or position changes. Use coordinate-based clicks only when the app
lacks accessibility support (e.g. electron apps, games, canvas-based UIs).

### Efficient action-verify loops

Prefer ``act_and_observe()`` over separate ``computer()`` + ``wait_for_change``:

  act_and_observe("left_click", coordinate=(760, 540))  # trigger action, see result

This combines the action and observation into one call, preventing the conversation
from accumulating multiple nearly-identical screenshots during transitions.
Only call ``screenshot()`` directly when you need the current state without waiting.

### Opening new windows without guessing their position

Prefer window_focus over clicking at a guessed coordinate after launching a window:

  computer("key", text="ctrl+alt+t")         # open terminal
  computer("window_focus", text="Terminal")   # wait for it, then focus it
  computer("type", text="echo hello")         # type into the now-focused window

This avoids the delay/click-at-random pattern that fails when window position
varies across sessions or virtual displays.

Note: Key names are automatically mapped between platforms.
Common modifiers (ctrl, alt, cmd/super, shift) work consistently across platforms.

### Observation helpers (structured-first policy)

Higher-level helpers are available that implement the structured-first observation policy:

- ``observe_web(url, screenshot_too=False)`` — observe a web page using ARIA snapshots first
  (no vision tokens), with automatic fallback to a browser screenshot, then desktop screenshot.
  Pass ``screenshot_too=True`` to get both an ARIA snapshot AND a screenshot side by side.
- ``observe_desktop()`` — thin wrapper around ``computer('screenshot')`` that signals intent
  clearly for native apps and non-browser surfaces.
- ``act_and_observe(action, text=None, coordinate=None, timeout=3.0)`` — perform a desktop
  action **and** automatically observe the result. Combines ``computer(action, ...)`` with
  ``wait_for_change`` in one call — the complete "act then look" loop without separate
  screenshot calls. Use this for tight interaction loops where you want to see the screen
  after every click, keypress, or scroll.
- ``fill_native(coordinate, text)`` — replace text in a native (non-browser) text field in one
  call. Triple-clicks to select all existing text, then types the replacement. The native
  equivalent of ``fill_element(selector, value)`` for DOM targets. Use when the target is a
  native app input (terminal, dialog, form) rather than a web page.
- ``computer_task(task, timeout=300, model=None)`` — run a multi-step computer-use task
  in a **context-isolated subagent** and block until done. All screenshots and intermediate
  steps are kept inside the subagent's own context — the caller's context stays lean. Use
  this for long, multi-step automations (filling forms, navigating multi-page flows, running
  GUI apps) where piling dozens of screenshots into the current context would be wasteful.
  Returns a status dict with ``status`` and ``result`` keys.

These helpers are preferred over calling ``computer("screenshot")`` directly when observing
web pages, because ARIA snapshots avoid costly vision tokens and give a DOM-addressable tree.

Examples

User	Take a screenshot of the desktop
Assistant	I'll capture the screen using the screenshot tool. ipython computer("screenshot")
System	Viewing image...

User	Type "Hello, World!" into the active window
Assistant	I'll type the text with realistic delays. ipython computer("type", text="Hello, World!")
System	Typed text: Hello, World!

User	Move the mouse to coordinates (100, 200) and click
Assistant	I'll move the mouse and perform a left click. ipython computer("mouse_move", coordinate=(100, 200))
System	Moved mouse to 100,200 ipython computer("left_click")
System	Performed left_click

User	Get the current mouse position
Assistant	I'll get the cursor position. ipython computer("cursor_position")
System	Cursor position: X=512,Y=384

User	Double-click at current position
Assistant	I'll perform a double-click. ipython computer("double_click")
System	Performed double_click

User

Scroll down in the page at (512, 400)

Assistant

I'll scroll down at those coordinates.ipython
computer("scroll", coordinate=(512, 400), text="down")

System

Scrolled down at 512,400

User

Click the Submit button then wait for the result page to load

Assistant

I'll use act_and_observe to click Submit and automatically get a screenshot once the screen settles.ipython
act_and_observe("left_click", coordinate=(760, 540))

System

Screen changed (23.4% pixels differ)
Viewing image...

User	Open a terminal and run a command
Assistant	I'll open a terminal with a keyboard shortcut, then use act_and_observe for window_focus so the shell prompt has time to appear before I type. ipython computer("key", text="ctrl+alt+t")
System	Sent key sequence: ctrl+alt+t ipython act_and_observe("window_focus", text="Terminal")
System	Screen changed (18.7% pixels differ) Viewing image... ipython act_and_observe("type", text="ls -la" + chr(10))
System	Screen changed (12.3% pixels differ) Viewing image...

User

Read the content of https://news.ycombinator.com

Assistant

I'll use observe_web to get a structured ARIA snapshot of the page — faster and cheaper than a screenshot.ipython
observe_web("https://news.ycombinator.com")

System

[ARIA snapshot of Hacker News front page...]

User	Check what's on my desktop right now
Assistant	I'll capture a screenshot of the desktop. ipython observe_desktop()
System	Viewing image...

User

Navigate to https://example.com and verify both the text content and visual layout

Assistant

I'll use observe_web with screenshot_too=True to get both the ARIA snapshot and a screenshot.ipython
observe_web("https://example.com", screenshot_too=True)

System

[ARIA snapshot + screenshot of example.com]

User

Open Firefox, go to https://x.com/compose/tweet, type "Hello from gptme!" and submit it — without filling up my context with screenshots

Assistant

I'll delegate this to computer_task() so all the intermediate screenshots stay in a subagent context rather than here.ipython
computer_task("Open Firefox, navigate to https://x.com/compose/tweet, type 'Hello from gptme!', and click the Tweet button.", timeout=120)

System

{"status": "success", "result": "Tweet submitted successfully. Firefox opened, x.com/compose/tweet loaded, typed the message, clicked Tweet. Confirmed tweet posted.", "agent_id": "computer-task-a1b2c3d4"}

User

Replace the text in the URL bar of the browser window with "https://example.com"

Assistant

I'll use fill_native to triple-click the URL bar to select all text, then type the new URL.ipython
fill_native((760, 45), "https://example.com")

System

Performed triple_click
Typed text: https://example.com

User

Fill the "Username" field at (300, 200) in this native login dialog

Assistant

I'll use fill_native to replace whatever is in the field with the username.ipython
fill_native((300, 200), "alice@example.com")

System

Performed triple_click
Typed text: alice@example.com

User	Open a new browser tab
Assistant	I'll open a new browser tab. ipython computer("key", text="ctrl+t")
System	Sent key sequence: ctrl+t

class gptme.tools.computer.ComboOperation

class gptme.tools.computer.KeyOperation

class gptme.tools.computer.ScreenRecording

Handle for an in-progress screen recording.

Returned by start_recording(). Call .stop() to finish the recording and get the output path. Also usable as a context manager:

with start_recording("session.mp4") as rec:
    # ... do things on screen ...
    pass  # recording stops here
print(rec.output_path)  # path to the MP4

output_path: Destination file path (set at construction time).

__init__(process: Popen, output_path: Path, stderr: IO[bytes] | None = None) → None

stop() → Path

Stop the recording. Safe to call more than once.

Returns:: Path to the completed video file.

class gptme.tools.computer.TextOperation

gptme.tools.computer.act_and_observe(action: Action, text: str | None = None, coordinate: tuple[int, int] | None = None, timeout: float = 3.0, settle_time: float = 0.2) → list[Message]

Perform a desktop action then automatically observe the result.

Implements the “act → look” half of the computer-use loop in one call, eliminating the separate computer('wait_for_change') step after every interaction. The screen is polled until it settles (up to timeout seconds), then a single screenshot is returned — exactly the same behaviour as computer('wait_for_change') but wired directly after the requested action.

For observation-only actions ("screenshot", "cursor_position", "accessibility_tree", "wait_for_change") the call is passed through unchanged: no extra screenshot is appended.

Parameters:

action – Desktop action to perform — same values as computer().
text – Text to type or key sequence (forwarded to computer()).
coordinate – Mouse coordinates (forwarded to computer()).
timeout – Seconds to wait for a screen change after the action (default 3 s).
settle_time – After detecting the first screen change, keep polling until the screen stops changing for settle_time consecutive seconds (default 0.2 s). This handles multi-phase UI transitions — e.g. a terminal frame appearing first and the shell prompt rendering shortly after — so the returned screenshot always shows the final settled state rather than a transient intermediate frame. Set to 0.0 to get the original behaviour (return on first change).

Returns:

For state-changing actions: zero or one action-output message (if the action itself produces output) plus a screenshot of the settled screen after the change.
For observation-only actions: just the output of computer().

Return type:

List of Message objects

Example (from IPython in a computer-use session):

# Click a button and see the screen update — one call, no polling
msgs = act_and_observe("left_click", coordinate=(760, 540))

# Type text and immediately verify what appeared
msgs = act_and_observe("type", text="hello world")

# Open a terminal and wait for the shell prompt (multi-phase transition)
# act_and_observe uses settle_time=0.2 by default: frame appears first,
# then the shell prompt, then 0.2s of quiet → returned screenshot shows prompt
msgs = act_and_observe("window_focus", text="Terminal")

# Observation-only actions are passed through unchanged
msgs = act_and_observe("screenshot")  # same as [computer("screenshot")]

gptme.tools.computer.computer(action: Action, text: str | None = None, coordinate: tuple[int, int] | None = None) → Message | None

Perform computer interactions in X11 or macOS environments.

Parameters:

action – The type of action to perform
text – Text to type or key sequence to send
coordinate – X,Y coordinates for mouse actions

gptme.tools.computer.computer_task(task: str, timeout: int = 300, model: str | None = None) → dict

Run a computer-use task in a context-isolated subagent.

Spawns a child agent with the computer-use profile and blocks until it completes (or times out). All screenshots and intermediate steps stay inside the subagent’s own context, so the caller’s context remains lean — this is the “context-efficient tool-use loop until goal is achieved” pattern described in gptme/gptme#216.

Use this instead of issuing a long chain of computer() + act_and_observe() calls directly when the task has many steps, or when you don’t want dozens of screenshots piling up in the current context.

Parameters:

task – Natural-language description of what to accomplish.
timeout – Maximum seconds to wait before giving up (default 300 = 5 min).
model – Optional model override for the subagent.

Returns:

Status mapping with keys:

status: "success" / "failure" / "clarification_needed" / "timeout"
result: text summary from the subagent
agent_id: subagent identifier — pass to subagent_read_log() for the full transcript
conversation: conversation name for the audit CLI (gptme-util computer audit-log CONVERSATION)
logdir: absolute path to the subagent’s conversation directory (str)

"clarification_needed" is returned if the subagent needs more information before it can complete the task. "timeout" is returned when the wall-clock deadline is reached before the subagent finishes. The worker thread may still wind down in the background, but callers immediately see the terminal timeout result.

Return type:

dict

Example (from IPython in a gptme session):

# Compose a tweet without piling screenshots into this context
result = computer_task(
    "Open Firefox, navigate to https://x.com/compose/tweet, "
    "type 'Hello from gptme!', and click Tweet.",
    timeout=120,
)
print(result["status"], result["result"])

# Audit what the subagent actually did (computer-use actions only)
import subprocess
subprocess.run(["gptme-util", "computer", "audit-log", result["conversation"]])

# Read the full step-by-step transcript
from gptme.tools.subagent import subagent_read_log
print(subagent_read_log(result["agent_id"]))

gptme.tools.computer.fill_native(coordinate: tuple[int, int], text: str) → list[Message]

Fill a native (non-browser) text field by clicking, selecting all, and typing.

Use this to replace the content of a native text input with new text — equivalent to fill_element(selector, value) for web targets, but for native desktop/X11/macOS apps.

The sequence is: triple-click to select all existing text, then type the replacement text. No separate ctrl+a step is needed.

Note

Triple-click selects all text in most single-line native inputs, but may select only a word or line in terminals and multi-line text widgets. Verify the field type before using this helper in those contexts.

Note

A screenshot is always captured after the fill so callers can observe the field state and detect partial or failed replacements.

Parameters:

coordinate – X,Y coordinates of the text field (in API space).
text – Replacement text to type into the field.

Returns:

List of Message objects — usually a single screenshot message showing the field state after the fill.

Example (from IPython in a computer-use session):

# Replace the URL bar text in a browser window
msgs = fill_native((400, 50), "https://example.com")

# Fill a login field in a native app
msgs = fill_native((300, 200), "username@example.com")

gptme.tools.computer.observe_desktop() → Message | None

Observe the current desktop state via screenshot.

Thin wrapper around computer('screenshot') that makes the structured-first / screenshot-fallback policy explicit: call this when there is no URL to snapshot (native apps, the raw desktop, or any non-browser surface).

Returns:: Screenshot Message, or None if capture failed.

Example (from IPython in a computer-use session):

msg = observe_desktop()
# Equivalent to computer("screenshot"), but signals intent clearly.

gptme.tools.computer.observe_web(url: str, screenshot_too: bool = False) → list[Message]

Observe a web page: structured ARIA snapshot first, screenshot as fallback.

Implements the structured-first observation policy: prefer accessibility snapshots for web targets — they avoid vision-token cost and give a DOM-addressable tree. Use screenshot_too=True when you need pixel-level visual confirmation alongside the structured snapshot (e.g. to verify layout or canvas content).

Falls back to a browser screenshot, then to a desktop screenshot, if Playwright is not available.

Parameters:

url – Page URL to observe.
screenshot_too – If True, also take a screenshot even when a snapshot succeeded.

Returns:

List of Message objects (snapshot and/or screenshots). Always returns at least one message; if all observation paths fail, returns a single system message explaining what failed and how to fix it.

Example (from IPython in a computer-use session):

msgs = observe_web("https://news.ycombinator.com")
# Returns one Message containing the ARIA snapshot text.

msgs = observe_web("https://example.com", screenshot_too=True)
# Returns snapshot Message + screenshot Message side-by-side.

gptme.tools.computer.record_screen(output: str | Path | None = None, duration: float = 10.0, fps: int = 10, display: str | None = None) → Path

Record the screen for a fixed duration and return the output path.

Synchronous wrapper around start_recording() / ScreenRecording.stop(). Blocks for duration seconds.

Parameters:

output – Destination path for the MP4 file. Defaults to a timestamped file in the system temp directory.
duration – How many seconds to record (default 10).
fps – Frames per second (default 10).
display – X11 display string (Linux only). Defaults to $DISPLAY.

Returns:

Path to the finished MP4 file.

Raises:

RuntimeError – If ffmpeg is not found or recording fails to start.

Example (from IPython in a computer-use session):

path = record_screen("tweet-demo.mp4", duration=30)
print(f"Recording saved to {path}")

gptme.tools.computer.start_recording(output: str | Path | None = None, fps: int = 10, display: str | None = None) → ScreenRecording

Start recording the screen to an MP4 file.

Uses ffmpeg with x11grab (Linux) or avfoundation (macOS). Returns a ScreenRecording handle — call .stop() to finish or use it as a context manager.

Parameters:

output – Destination path for the MP4 file. Defaults to a timestamped file in the system temp directory.
fps – Frames per second (default 10 — suitable for UI demos; increase to 24+ for smooth game recordings).
display – X11 display string (Linux only). Defaults to $DISPLAY.

Returns:

ScreenRecording handle. Call .stop() when done.

Raises:

RuntimeError – If ffmpeg is not found or recording fails to start.

Example (from IPython in a computer-use session):

rec = start_recording("tweet-demo.mp4")
# ... interact with the browser ...
rec.stop()  # saves tweet-demo.mp4

# Or as a context manager:
with start_recording("demo.mp4") as rec:
    computer_task("open Firefox and navigate to https://example.com")
print(rec.output_path)