Desktop automation and the accessibility tree: what one node costs to capture

The tree is real, and so is its cost

Here is the mental model most people carry away from a tutorial: the accessibility tree exists, somewhere, fully formed, and a library hands it to you the way a browser hands you the DOM. That picture is wrong in a way that matters.

On Windows, the tree lives inside each running application's UI Automation provider. Your automation process does not share memory with it. When you ask for a node's name, that is a COM call that crosses a process boundary, gets serviced by the target app, and returns one string. Ask for its role, its value, its bounding rectangle, whether it is enabled: each one is a separate round trip. A single node with six interesting properties is six cross-process calls.

Now multiply. A bare Notepad window has a couple hundred nodes. A Chrome window with a real web app inside it has many thousands. A framework that captures every property on every node is making tens of thousands of cross-process calls for one tree dump, and the user watches their machine hitch while it happens. The naive implementation is not slightly slow. It is unusable inside an agent loop that re-reads the tree after every action.

So the real engineering question for desktop automation is not "how do I get the accessibility tree." It is "how little of it can I get away with capturing, and how do I capture even that without freezing the host." The rest of this page is Terminator's answer, read straight out of the source.

What one node actually carries

Terminator's representation of a node is the UIElementAttributes struct in crates/terminator/src/element.rs (lines 309 to 343). It defines roughly 17 fields. Below is the whole set. The orange fields are the ones the default Fast mode loads for every node. The amber fields are loaded conditionally (state flags, structural counts, the focusable-gated bounds). The plain fields are only filled when you explicitly ask for Complete or Smart mode.

One node is not just attributes. The UINode wrapper (lib.rs, lines 328 to 338) adds three more things: an id, a children vector, and a selector string. That selector is the full chained path from the root window down to this node, something like role:Window && name:Untitled >> role:Edit. It is built as the tree is walked, so every node in a captured tree is already addressable: copy its selector straight into desktop.locator() and you have a locator for it.

Why the default loads two fields, not seventeen

The choice of how much to capture is a single enum. PropertyLoadingMode in platforms/mod.rs (lines 57 to 64) has three variants, and the comments in the source describe each one exactly: Fast is "only load essential properties (role + name) - fastest," Complete is "load all properties for complete element data - slower but comprehensive," and Smart is "load specific properties based on element type - balanced approach." The default for tree building is Fast.

That default looks aggressive until you remember the cost model. Selector matching needs role, name, and id. It does not need a tooltip description or a value string for the layout container three levels up that you will never touch. Loading those anyway means one extra cross-process call per field, per node, across the whole tree. Fast mode is not cutting corners. It is declining to pay for data nothing will read.

The anchor: bounds only for what you can focus

Here is the single most telling line in the whole tree builder. In element.rs, line 334, the bounds field of a node is declared with this comment attached to it:

pub bounds: Option<(f64, f64, f64, f64)>,
// Only populated for keyboard-focusable elements

That comment is a design decision written down. The tree builder's get_configurable_attributes function checks element.is_keyboard_focusable() and only when that is true does it call element.bounds() to attach a rectangle. Asking for a node's bounding rectangle is itself a cross-process call, so this is not a cosmetic skip. It is the framework declining to pay for the pixel coordinates of things you will never click.

The logic underneath it is clean. The elements you act on (buttons, edit fields, checkboxes, list items, links) are precisely the ones the OS marks keyboard-focusable. A static text label, a decorative image, a layout group: not focusable, not clicked, no bounds. Then the tree formatter closes the loop. It assigns a numeric click index only to nodes that have bounds. So the indexed, clickable subset of a captured tree is the keyboard-focusable subset, by construction, with no separate filtering pass. One inline comment, and the whole capture-cost story is consistent.

Walking the tree without freezing the machine

Capturing less per node is half the job. The other half is the walk itself. build_ui_node_tree_configurable in tree_builder.rs does not just recurse blindly. It uses an explicit work queue, so a pathologically deep app cannot blow the call stack: recursion is capped at depth 100, and anything deeper gets pushed back onto the queue and processed iteratively.

The yield step is the one developers underestimate. The builder tracks how many elements it has processed, and every 50 of them (the yield_every_n_elements default), plus between large child batches, it calls thread::sleep for 1 millisecond. That tiny pause hands CPU back to the rest of the system so the host UI does not visibly hitch while a big tree is being captured. Children are processed in batches of 50 (batch_size), and the tree depth itself defaults to 50 levels, raised to 500 for browsers because web apps nest far deeper than native windows.

None of this is glamorous. It is the unglamorous machinery that turns "read the accessibility tree" from a slogan into a function that reliably returns in a few hundred milliseconds without making the user's desktop stutter. When a guide tells you the tree is fast, this is the code that earned the word.

What a captured tree actually looks like

Once the builder has run, the formatter renders the tree as a compact, indented, YAML-like block. Nodes that carry bounds get a #N click index; nodes without bounds get a plain dash. Here is a trimmed dump of a Notepad window captured in Fast mode.

Read the last two lines. 214 nodes captured, only 2 of them indexed, because only 2 are keyboard-focusable and therefore only 2 carry bounds. The MenuBar and the StatusBar text are in the tree (you can still read their names and roles) but they are not clickable targets, so the formatter does not waste an index on them. And the selector for node #1 is already written out, ready to paste into a locator. That is the Fast-mode philosophy made visible: capture what addresses and acts on elements, skip the rest.

The numbers, straight from the source

Seventeen fields defined, two loaded by default. That ratio is the whole argument. Everything else (the batching, the depth limits, the millisecond yield) exists to make even those two reads, multiplied across thousands of nodes, finish fast enough that an AI agent can re-capture the tree on every single step of a workflow without the human noticing.

Where the tree is thin, and what catches the fall

The accessibility tree is the right default for desktop automation, but it is honest to name where it gets thin. Three cases come up repeatedly.

Custom-drawn surfaces. A fullscreen game, a 3D modeller, a browser canvas (Figma's drawing area, Excalidraw, Miro): these paint their own pixels and expose a single opaque node with no useful children. The tree cannot help you there because the app never built one.

Shallow trees. Some apps implement accessibility poorly. Electron apps are known for flat, unhelpful trees. And a genuinely deep web view can exceed the depth limit, which is exactly why Terminator raises the default depth to 500 for browsers and why "element not found" often means "deeper than the depth you captured" rather than "not there."

Lying providers. A few targets return success on a tree action while doing nothing, most notoriously browser web views on macOS. The tree is not wrong about what exists; it is wrong about what acting on it will do.

Terminator's response is to treat the tree as the fast first layer of a stack, not the only layer. Its own description is "accessibility tree + DOM + OCR + vision AI for maximum reliability": the tree resolves the overwhelming majority of elements at CPU speed, DOM access through a Chrome extension covers web content the tree mangles, and OCR plus vision catch the custom-drawn surfaces. The tree is the default precisely because it is cheap and structured; the other layers exist because no single layer is complete.

Capturing it yourself

Everything above is one function call away. From the Node SDK, desktop.getWindowTree('notepad') returns a UINode you can walk or serialize; getWindowTreeResult additionally hands back the formatted indexed string and an index-to-bounds map for click-by-index. From an MCP agent in Claude Code, Cursor, or VS Code, the same capture is exposed as a tool: the agent asks for the tree, reads the structured nodes, and picks a selector, all running the same Rust tree builder this page describes.

claude mcp add terminator "npx -y terminator-mcp-agent@latest"