Desktop accessibility automation agents: the part nobody documents is that they have to be checkpointed

A program that drives desktop apps the way a screen reader reads them: through the OS accessibility tree (UI Automation on Windows, AXUIElement on macOS, AT-SPI2 on Linux) instead of pixels. The agent finds elements by role and name, fires actions through accessibility patterns or synthesised input, and reads state back from the same tree. When a large language model is in the loop, the agent talks to the OS through an MCP server (Terminator's is one) so the model issues structured tool calls instead of OCR'ing screenshots. The model never sees a pixel that is not in the tree, and the OS never sees an action that is not bound to a structural selector.

Computer use agents (the Claude computer-use mode, Anthropic's reference loop, Gemini's computer-use API) read pixels and emit click(x,y) plus type(text). They work on any UI but are sensitive to DPI, theme, scroll position, and resolution; the same step is a different click on a different monitor. Accessibility-tree agents read role+name+AutomationId and emit click(role:Button|name:Save). The selector is stable across DPI, scroll, theme, and even most localisation, because the OS is doing the lookup. The trade-off is that surfaces the OS does not classify (game canvases, custom-rendered controls, Office documents at the cell level) are invisible. A real production agent does both, and falls through from the tree to vision when the tree is silent.

It means the agent writes its working state to disk after every step, so a crash, a network glitch, or a Ctrl-C does not reset the run to step 1. In Terminator's MCP, the meta-tool execute_sequence calls save_workflow_state (server_sequence.rs:216) after a step stores a tool result into env. The state file is at ~/Library/Application Support/mediar/workflows/<folder>/state.json on macOS, %LOCALAPPDATA%\mediar\workflows\<folder>\state.json on Windows, and ~/.local/share/mediar/workflows/<folder>/state.json on Linux. To resume, you re-run execute_sequence on the same workflow URL with start_from_step:"<step_id>" set to the last failed step. load_workflow_state (line 261) reads the file and re-injects env into the execution context before step 1 of the resumed range runs. Most desktop automation frameworks (PyAutoGUI, AutoHotkey, raw UIA, raw AX) have no analogue; they are scripts, not agents.

From the workflow URL. extract_workflow_folder_from_url (server_sequence.rs:50) takes a string like file:///Users/matt/.../workflows/github-demo/src/terminator.ts, normalises slashes, finds the substring /workflows/, and returns the next path component ("github-demo"). That folder name becomes the leaf directory under .../mediar/workflows/. The convention is that a workflow lives in its own folder under a workflows/ directory, and the folder name is the workflow's identity. Two workflows in two different folders never collide on state, even with the same step IDs. There is also a workflow_id parameter on execute_sequence kept for backward compatibility, but the URL-based path takes precedence (line 200). If neither is set, no state file is written and the workflow runs unchecked but unrecoverable.

A flat JSON object with five keys: last_updated (RFC3339 timestamp), last_step_id (the step that just finished), last_step_index (the 0-based index of that step), workflow_id, workflow_file (basename of the workflow URL), and env (the live environment object the steps read and write). The env object is the one the steps use as both input (variables substituted into selectors and arguments via {{key}}) and output (a step that returns set_env: {key: value} writes back into it). On resume, the loader (line 261) reads the file, takes the env field, and inserts it under "env" in the new execution context. From the perspective of the resumed step, nothing else has happened in the world. From the perspective of the OS, the desktop is wherever the user left it.

Only steps whose result is stored into env. In execute_sequence, a step that supplies a step id and returns a tool result (or that uses run_command with a set_env return) writes the result to env at server_sequence.rs:1466 and immediately calls save_workflow_state with the current step index. A read-only step that does not store anything (a get_window_tree without an id, a validate_element with no follow-up env write) does not bump the state file. This is intentional: the state file is the variables that the next step depends on, not a transcript of every tool call. A separate execution_logger captures the full transcript for debugging.

Three habits. First, give every step an id, because start_from_step:"<id>" is the resume handle. Second, store anything that took non-trivial work to compute (a parsed account number, an OAuth token captured from a browser run_command, a file path the agent just wrote) into env, so it survives. Third, write your steps as idempotent restarts: a click on a Submit button that was already submitted should be safe, or the resumed run will double-submit. With those three, a 20-step Outlook automation that fails on the network round trip at step 14 resumes at step 14 with the env from step 13 intact, instead of replaying steps 1 through 13 against a UI that may already be in a different state. Without those three, you rerun the whole thing every failure.

Three layers. Layer one is 38 #[tool] declarations in crates/terminator-mcp-agent/src/server.rs (read, click, type, key, drag, scroll, validate, wait, highlight, navigate, screenshot, run_command, execute_sequence, and so on). Layer two is the seven-mode click_element router (Selector, Index over UiTree/Ocr/Omniparser/Gemini/Dom, Coordinates) that lets the agent fall through grounding sources when the AX tree is silent. Layer three is execute_sequence, the meta-tool that wraps the other 37 into a step list with state persistence, jump conditions, fallback IDs, and resumability. A real agent uses all three: tools to read and act, the click router to ground when the tree is incomplete, and execute_sequence to make the run survivable. The page you are reading is about layer three, because that is the one the existing playbooks miss.

Because the failure modes that matter are crash, hang, and reboot. An in-memory store dies with the process. A database is one more dependency to install and one more place where the credentials must be valid. A JSON file written through tokio::fs::write to an OS-standard data directory (data_local_dir from the dirs crate) survives a kill -9 of the MCP server, a Windows update reboot, and a laptop sleep. If the user wants to inspect the state, they cat the file. If they want to migrate the workflow to another machine, they copy the folder. The cost is no transactions and no concurrent writers; the convention is that one workflow URL maps to one folder, and only one execute_sequence is in flight against that folder at a time. For our customers' agents, that constraint has not bound.

No. The state-persistence path uses platform-agnostic Rust (dirs::data_local_dir, tokio::fs, serde_json) and works on Windows, macOS, and Linux. The accessibility-tree backends underneath are platform-specific (UI Automation COM on Windows, AXUIElement on macOS, AT-SPI2 on Linux), but the agent loop is the same. macOS support exists at the core Rust level (the terminator-rs crate); the Node and Python packages currently ship Windows binaries only. Most of our users target Windows because the line-of-business apps that resist browser-side automation are Windows-native, but the cross-platform path is real.

Two commands. claude mcp add terminator 'npx -y terminator-mcp-agent@latest' to register the server with Claude Code (Cursor, VS Code, Windsurf take an equivalent MCP entry). Then in a chat: "create a workflow file at ~/workflows/notepad-demo/terminator.ts that opens Notepad, types a sentence, then errors on purpose; run it with execute_sequence; show me the state.json that was written; then resume from after the type step." The agent will use execute_sequence with the file:// URL, the runtime will write ~/Library/Application Support/mediar/workflows/notepad-demo/state.json (macOS) or %LOCALAPPDATA%\mediar\workflows\notepad-demo\state.json (Windows) after each successful step, and on the resume call it will load env from that file and start at the step you specified. The full source is at github.com/mediar-ai/terminator.