Accessibility selectors vs screenshot automation: one is a query, the other is a guess

The same task, both ways

Here is "click the Save button" written against a screenshot library and against a selector. Look less at the line count and more at the comment block at the bottom of each side. That is where the real cost lives.

import pyautogui, time

def click_save():
    # the button is identified by a cropped screenshot
    loc = pyautogui.locateOnScreen("save_button.png", confidence=0.9)
    if loc is None:
        # retry once in case the toolbar is still animating
        time.sleep(0.5)
        loc = pyautogui.locateOnScreen("save_button.png", confidence=0.9)
    if loc is None:
        raise RuntimeError("Save button not found on screen")
    pyautogui.click(pyautogui.center(loc))

# save_button.png has to be re-cropped on every theme,
# DPI scale, locale, or toolbar change. There is no query.

The screenshot version is not wrong. It works, today, on this machine, in this theme, at this resolution. The problem is that every one of those qualifiers is load-bearing. The croppedsave_button.pngis a snapshot of a moment, and the moment keeps ending. A selector is not a snapshot. It is a description that stays true as long as the button still exists and is still called Save.

A selector is a query. A coordinate is not.

This is the distinction that most write-ups on the topic skip. They land on "accessibility selectors are more stable than pixels" and stop, as if a selector were just a sturdier string. It is not a string. It is a query, and a screenshot approach has no query language at all.

A screenshot tool gives you two primitives:locateOnScreen(image)andclick(x, y). Every other concept you need has to be hand-built on top of those. "The third row" becomes pixel arithmetic. "The field next to the Email label" becomes a cropped region and a hope that the layout never shifts. "A button that is visible and not disabled" cannot be expressed at all, because a picture does not carry an enabled flag.

A selector carries all of that as first-class syntax. You can compose by role and name, filter by visibility, scope to a process, walk to a parent, require a descendant, pick the nth match, and combine clauses with boolean operators. The accessibility tree already knows what every node is; the selector engine just lets you ask.

Real selector strings Terminator parses

0 ways to name an element, zero ways to name a pixel

This is the part you can check yourself. Terminator's selector engine lives incrates/terminator/src/selector.rs. TheSelectorenum at the top of that file defines 25 variants. They group into five kinds of question you can ask about an element:

• Identity. role:, name:, id:, nativeid:, classname:, text:. What the element is and what it is called.
• Spatial. rightof:, leftof:, above:, below:, near:. An element described by its position relative to another element, computed from the tree at runtime.
• Structure. has: (a Playwright-style descendant requirement), .. (parent navigation), >> (chain through the hierarchy), and nth: for ordinal selection.
• Filters. visible: and process:. State and scope a picture cannot record.
• Boolean. &&, ||, and !, with parentheses. The test file boolean_selector_tests.rs exercises real expressions like (role:button && name:Submit) || (role:link && name:Cancel) and !role:button && visible:true.

Now count the equivalents on the screenshot side. A template-match library exposes one way to name a target: a cropped image. A vision model adds one more: a natural-language description that it grounds into a bounding box. Neither is composable. You cannot AND two template images. You cannot ask a picture for the element to the left of another element without writing the geometry yourself. The asymmetry is not a matter of polish. One side is a language and the other side is a pair of primitives.

Watch one selector narrow the tree

A selector resolves by progressively filtering nodes. Each clause shrinks the candidate set until one element is left. A screenshot never narrows: it carries the whole window into every match attempt.

How each side resolves one click

The two paths below do not just differ in reliability. They differ in how many moving parts sit between "I want to click Save" and an actual click. The selector path is one request and one response. The screenshot path adds a capture, an inference step, and a coordinate that is only ever a best guess.

The red step at the end of the screenshot path is the one that bites. Once you have a coordinate, you have committed to it. If the window scrolled, if a notification pushed the layout down, if the inference was off by twelve pixels, the click lands on the wrong thing and the script keeps going as if it succeeded. The selector path never produces a loose coordinate to be wrong about: it hands you an element, and the click goes through the operating system's own accessibility action for that element.

Where a screenshot is still the right call

Selector-first does not mean selector-only. There are real surfaces where the accessibility tree has nothing useful in it, and on those surfaces a screenshot plus OCR or a vision model is the honest answer. The mistake is using screenshots as the default and falling back to selectors, instead of the other way around.

The dividing line is whether the element exists in the tree at all. A button in a normal AppKit, WinUI, WPF, or WinForms app has a node with a role and a name. A button drawn by a game engine onto a Metal or Direct3D surface does not: the whole window is one opaque node. The first is a selector's job. The second is genuinely a screenshot's job. Confusing the two, in either direction, is where automations go brittle.

How Terminator draws that line

Terminator is a desktop automation framework with an API shaped like Playwright, except it targets the whole operating system instead of just the browser. It drives apps through the native accessibility APIs: UI Automation on Windows, the Accessibility API on macOS. Element lookup is selector-first by default, which is why the selector engine in selector.rs is as expressive as it is. OCR and vision detection ship in the box, but they are the fallback for surfaces the tree cannot describe, not the primary mechanism.

It is a developer framework, not a consumer app. You pull in the Rust crate terminator-rs, the Python bindings, or the Node package, or you wire the MCP server into Claude Code, Cursor, or VS Code so an AI assistant can drive real desktop apps as a tool. Whichever entry point you pick, the selector is the unit of work, and the selector is a query. The full prefix list is documented in the selector cheat sheet in the open-source repository, and the core crate is published as terminator-rs on crates.io.