Accessibility tree vs pixel for computer use. The framing is wrong.

The dichotomy survives only at the API level

Anthropic’s computer_20251124 tool returns clicks at pixel coordinates after the model reads a base64 PNG. Microsoft UIAutomation, Apple AXUIElement, Linux AT-SPI all return tree nodes with roles and names. Each contract is internally consistent, and an agent that lives inside one contract gets a clean mental model. The price is that real desktop sessions cross contracts every few minutes. A user opens Slack to read a message (Electron, AX returns nothing), Cmd-Tabs to Notes to jot a reply (Cocoa, AX returns role-and-name on every line), opens Figma to grab a screenshot of a frame (canvas, only pixels carry signal), pastes it into a doc (rich text, AX). An agent built around one regime treats half the session as noise.

Every comparison article admits this with a closing paragraph like “combine both for best results.” What they skip is the part the architect needs: where does the combination happen, what does the wire format look like, and how does the model pick a source for one click without restructuring the prompt? The next sections are about that mechanism.

The tree-only view, on the surface where the tree breaks

Toggle to see what the agent reads when you ship only the UIA tree on a Slack window, and what the same agent reads when the server clusters the UIA snapshot with an OmniParser pass and an OCR pass on the same screenshot. The first view is the one most production agents render today. The second is the view that lets the model click #u12 when the tree has a name and #p7 when it does not.

The mechanism in five lines of Rust

The shape that makes the “both at once” framing actually work is a five-variant enum and a union-find pass. The enum:

// crates/terminator/src/tree_formatter.rs, around line 40
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ElementSource {
    Uia,        // #u - Accessibility tree
    Dom,        // #d - Browser DOM
    Ocr,        // #o - OCR text
    Omniparser, // #p - Omniparser vision
    Gemini,     // #g - Gemini vision
}

The pass: every element from every source lands in one Vec<UnifiedElement> with a screen-coordinate bounding box and a source tag. The clustering rule:

// around line 450
fn should_cluster(b1: (f64, f64, f64, f64),
                  b2: (f64, f64, f64, f64)) -> bool {
    let smaller_dim = f64::min(
        f64::min(b1.2, b1.3), f64::min(b2.2, b2.3));
    let threshold = smaller_dim * 1.5;
    min_edge_distance(b1, b2) < threshold
}

A union-find walk over all pairs of elements groups every set that shares a region. Sources sit inside the same group when their bounds overlap or sit closer than 1.5 times the smaller box dimension. The result is a list of clusters sorted in reading order. The MCP server emits one clustered_tree string and one index map. The model picks indices off the string and the resolver routes them through the right backend.

That TLDR is the first comment in the platform shim that backs the #u source. The clustering pass does not know it. The model does not see it. The accessibility-tree branch silently misses every non-main window unless the platform shim works around an OS-level walker default. That is the kind of failure that does not survive in a one-source agent.

What the model sees, one round-trip at a time

One MCP call returns a clustered tree. The model reads it. The next call clicks one prefixed index. The MCP server snapshots all caches in parallel, clusters, and writes a single result. The wire diagram below shows the call shape; nothing about the agent loop changes between a tree-friendly app and a tree-hostile one.

When each source actually carries the signal

The model does not need a prompt that tells it which surfaces are AX-friendly and which are not. It reads a cluster, sees which sources are present, and picks one. But for a human designing the agent, the per-surface picture is useful. The five-source split lines up roughly with four classes of UI:

The lines blur. Office for Mac uses AppKit on the chrome and a Chromium webview for the cloud-doc surface, so a single task crosses regimes. Cursor renders its file tree with Chromium AX but the editor itself with a Monaco canvas. Some Photoshop dialogs are native AX while the document area is a Metal canvas. None of that needs custom code in the agent if the clustering happens server-side: the model just sees more clusters tagged with #p in one part of the window and #u in another.

The same choice with no model in the loop

Everything above assumes an agent: a model reads a clustered tree and clicks a prefixed index. Plenty of desktop automation has no model in it at all. RPA jobs, QA suites, scheduled scripts, a build step that drives an installer. The accessibility-tree-versus-pixel question is identical for those, and so is the answer, but the machinery is lighter. There is no union-find clustering and no five-source merge to run, because there is no model to feed. You pick the source yourself, at author time.

In that mode the selector grammar is the whole API. A deterministic script calls desktop.locator("role:Button && name:Save") then .invoke() directly. Tree first, because role plus name resolves in a COM round trip and survives DPI, theme, and locale shifts. Pixel and OCR stay as the fallback for exactly the surfaces a model would also fall through on: canvases, games, and remote-desktop viewports where the tree is one opaque node. The only thing that changes between an agent and a script is who chooses the source. An agent reads the clustered output and picks an index; a script commits the selector when you write it, and a miss surfaces as a typed ElementNotFoundError instead of a hallucinated coordinate. Terminator exposes both paths from one Rust core: the terminator-rs crate and the terminator-py binding for scripted automation, and the terminator-mcp-agent server for the agent loop.

Four numbers that define the shape

Four numbers from the source code, not benchmarks. They define the surface of the system and the price of crossing it.

Why the binary framing keeps coming back

Two reasons. The first is research-paper inertia. Computer-use benchmarks publish numbers on one regime at a time so the result is comparable across labs; that framing leaks into how the press writes about the field. The second is that pipeline engineering for a hybrid is genuinely harder than pipeline engineering for one regime. You need a per-source bounds cache, a union-find over visible elements every tree call, a prefix-aware resolver in every action tool, and a tool result format that survives clusters with eight entries on a busy screen. None of it is hard individually; together it adds a hundred or two hundred lines of orchestration code per agent. Most published agents skip that and pick one regime, which is why the dichotomy in articles outlives the dichotomy in practice.

Terminator’s contribution to this conversation is not the insight that hybrid is better; that part is widely agreed. The contribution is one open-source repo where the orchestration code is written down. If the framing on your roadmap doc is still “pick a regime,” the next page to read is tree_formatter.rs. The next thing to look at is the prefixed-index resolver in server.rs around line 1697.