Structural locators vs pixel matching for computer use

Two definitions, side by side

Strip away the framework names and the two approaches store fundamentally different things. One stores a picture. One stores a question.

What each one actually stores, and when it resolves

The reliability gap is not about accuracy on a static screen. On a frozen screenshot a good template match and a locator both find the button. The gap opens the moment anything between capture and action changes. Toggle between what the two approaches hold in memory.

The line that makes it a locator, not a snapshot

The reason a structural locator is resilient is not a clever matching heuristic. It is that the locator deliberately resolves nothing until you act. Here is the struct, verbatim in spirit from crates/terminator/src/locator.rs:

// crates/terminator/src/locator.rs
pub struct Locator {
    engine: Arc<dyn AccessibilityEngine>,
    selector: Selector,
    timeout: Duration,        // default for this locator instance
    root: Option<UIElement>,
}

// One-time search by default. Opt into polling by setting a timeout.
const DEFAULT_LOCATOR_TIMEOUT: Duration = Duration::from_secs(0);

There is no field for a found element. Every action method re-runs the query against the live tree: wait() calls the engine’s find_element again, all() calls find_elements again, and wait_for() loops on a 100 millisecond poll, re-checking the element against one of four conditions until it holds:

pub enum WaitCondition {
    Exists,
    Visible,
    Enabled,
    Focused,
}

// wait_for(): poll_interval = Duration::from_millis(100)
// loop: validate() -> check condition -> sleep 100ms -> repeat
// on timeout: AutomationError::Timeout, with the selector in the message

That is the uncopyable part. A pixel matcher cannot adopt this behavior because it has nothing to re-resolve: its target is a crop, and a crop is just as stale on the second poll as the first. The locator can poll because its target is a description, and the description is still true after the button moves.

What it looks like in code

The public API is shaped like Playwright, so the lazy-resolution model reads the way you expect. You build a locator, optionally wait for a condition, then act. Each call resolves fresh. This is from examples/notepad.py and the Node test suite:

# Python: build a locator, resolve it now, act on it
editor   = desktop.open_application("notepad.exe")
add_tab  = await editor.locator("name:Add New Tab").first()
add_tab.click()

document = await editor.locator("role:Document").first()
document.type_text("hello from terminator!")

// Node: wait for a condition before acting
const win = await desktop
  .locator("role:window")
  .waitFor("exists", 5000);   // re-resolves until it exists or 5s passes

Notice there is no screenshot in that loop and no coordinate anywhere. The agent never says “click at (412, 238)”. It says “find the thing named Save and invoke it,” and the framework re-answers that on the spot.

Four numbers that define the behavior

These come straight from the source, not a benchmark. They are the dials that govern how a locator resolves.

The failure mode is the real difference

Both approaches succeed on a clean run. What separates them is what happens when the target is not where it was. A pixel matcher returns its best-correlation coordinate no matter how poor the match, so a missed target becomes a click into empty space that corrupts the next several steps silently. A locator that cannot resolve raises a typed, located error and stops.

The timeout message carries the selector string, so when an agent stalls you know exactly which element it could not find. That is the difference between a five-minute fix and an hour of bisecting a run that drifted three steps after the actual failure.

Where pixel matching still wins

A structural locator needs structure to query. On surfaces that have none, the locator is the worse tool and you should reach for pixels. A game rendered to a GPU canvas, a remote-desktop viewport delivered as a single video stream, the document area of Figma or Photoshop, a custom-drawn chart: the accessibility tree hands back one opaque node with the window bounds and nothing actionable inside. There is no role and no name to match, so a vision model or a pixel detector is the only thing carrying signal there.

The honest answer is not “locators always, pixels never.” It is: use a locator wherever the tree has structure, which is the large majority of native and well-built desktop apps, and fall back to pixels only on the opaque surfaces. The mistake the brittle agents make is using pixels everywhere because pixels are the only thing a pure-screenshot loop can see. For how to merge both signals into one tool result when you do need them together, see the accessibility tree vs pixel breakdown.