Accessibility APIs vs deterministic scripts. The word ‘deterministic’ is doing too much work.

What ‘deterministic scripts’ usually means in practice

The phrase shows up most in three contexts. The first is AutoHotkey and AutoIt scripts on Windows: short imperative programs that move the mouse to a coordinate, click, send keystrokes, and sleep between steps. The second is PyAutoGUI in Python: the same idea with a different syntax, usually paired with locateOnScreen for template-matching against a saved screenshot. The third is the recorded-macro lane: the user runs a recorder, performs the workflow once, and the tool captures the sequence of mouse moves and keystrokes for replay. Older RPA tools, Sikuli, Pulover's Macro Creator, and the legacy desktop-test suites built on UI Recorder all sit here.

What unites all three is that the script encodes the state of one specific machine at one specific moment. The position of a button, the time the previous action took, the appearance of an icon: all baked into the script as literal values. The script is ‘deterministic’ in the sense that two runs on that same machine produce the same syscalls. On a different machine the same syscalls land in different places.

Same task, written both ways

Save an Excel workbook with a specific filename, the canonical hello-world for desktop automation in office workflows. The coord-script version is shorter to write and feels more direct. The selector version is shorter to keep alive. Toggle between them.

What a wait actually compiles to

The single most expensive line in a deterministic script is Sleep, 2000. It is expensive twice: once because it always waits the full 2000ms even when the screen is ready in 200ms, and once because it does not wait long enough when the screen takes 2300ms. The accessibility-API equivalent is a polled predicate. The Rust source for the canonical implementation is short enough to read in one breath: tab to it below.

# the deterministic-script equivalent. canonical autohotkey shape.
# pyautogui's pyautogui.PAUSE has the same effect for every action.
# autoit's Sleep() is identical. recorded macros bake this in.

Sleep, 2000        ; wait for some duration we picked on our machine
MouseMove, 412, 300
Click

# the script does not check whether the element exists.
# does not check whether it is visible.
# does not check whether it is enabled.
# does not check whether anything is focused.
# the OS dispatches the click at (412, 300) into whatever window
# happens to be in foreground after the 2000ms.
#
# the "input" is deterministic: 2000ms, then click at (412, 300).
# the "outcome" is whatever the OS happens to be showing at that point.

The right side is roughly the body of Locator::wait_for in terminator/src/locator.rs lines 170 to 233. The poll interval is fixed at 100ms (line 186). The four typed conditions are WaitCondition::Exists, Visible, Enabled, Focused (lines 13 to 22). The loop returns the moment the predicate holds, returns a typed AutomationError::Timeout at the budget, and otherwise does nothing. There is no fudge factor for ‘the screen might still be animating’, because the predicate is the answer to that question.

What replay looks like the next day

The interesting place to look at outcome-determinism is at replay time. The coord-based script makes a contract with the OS: dispatch these synthetic input events at these coordinates after this delay. The OS keeps that contract. The contract just does not say anything about what the target will be.

The selector-based script makes a different contract: resolve this structural identity in the tree, wait until it is visible, then invoke its default action. The contract is over the OS's structural state, which is what the user was actually pointing at.

The first sequence has no ‘the wrong window is foregrounded’ arrow because the OS does not need one; it just dispatches. The second sequence has no ‘the wrong window is foregrounded’ arrow because the selector scope-clause (process:excel) excludes everything that is not Excel before resolution starts.

The defense recorders need (and what the coord recorders skip)

Even semantic selectors can drift if a recorder is not careful. A button labelled ‘3 hours ago’ on a Slack message becomes ‘yesterday’ tomorrow, ‘last week’ in a week, and ‘last month’ eventually. A recorder that captured name:3 hours ago into a selector chain produces a deterministic-looking script that matches nothing tomorrow. The fix is straightforward and lives in real source: filter the names the recorder is allowed to encode.

The filter is defined in terminator-workflow-recorder/src/events.rs lines 67 to 81 (contains_relative_time), with the empty-name and null-like-value filter at lines 8 to 36 (is_empty_string, NULL_LIKE_VALUES). A coord-based recorder does not need this filter because it does not encode meaning in the first place. It captures (412, 300) and replays (412, 300). The replay outcome happens to be wrong; the recorder cannot tell. Outcome-determinism is not free in either approach: it costs a list of relative-time substrings on the AX side, and on the coord side it costs everything the script does.

When to use which, written out

The honest rule is short. The accessibility tree is the right default for everything that exposes a tree. Coordinate input is the right answer for the residual targets where the tree is empty or single-node. Most production scripts mix the two; the question is what is your default and where do you fall back, not which one to pick exclusively.

Why the framing matters for AI agents

A growing category of computer-use agents (ChatGPT Agents, Anthropic computer use, BrowserUse) follows the deterministic-script pattern with one substitution: an LLM picks the coordinates from a screenshot instead of the author hardcoding them. The shape is the same: the agent decides where to click, then synthesizes a click at that location through the OS's HID input layer. The outcome is bound to whatever is at those pixels at that moment, and to whatever the model picked given the image it saw.

An agent built on accessibility APIs uses the model differently. The model picks a selector (role:Button && name:Save), the framework resolves it through the OS tree, and the framework calls the element's pattern. The model never needs to ground from pixels because the tree is text. Per-action latency is bounded by IPC instead of inference, which is the source of the roughly 100x speed delta Terminator claims at llms.txt line 243. The shape difference matters more than the speed: a model that picks selectors and gets typed errors back is in a closed loop the framework can recover from. A model that picks pixels and gets ‘the click happened’ back is in an open loop.