Computer use on native apps: what the accessibility tree actually looks like to the agent

What the agent actually receives

Most write-ups describe the accessibility tree in the abstract: structured metadata, element types, labels, parent-child links. All true, and none of it tells you what arrives in the model's context. Here is the concrete answer. When Terminator's get_window_tree tool runs on a focused native app, the response is this:

# get_window_tree on the focused Calculator process
# returns this, not the raw UIAutomation objects:

#1 [Window] Calculator (bounds: [760,300,322,500], focused)
  #2 [Group] Number pad (bounds: [760,540,322,260])
    #3 [Button] Seven (bounds: [760,540,107,65], focusable)
    #4 [Button] Eight (bounds: [867,540,107,65], focusable)
    #5 [Button] Nine  (bounds: [974,540,107,65], focusable)
  #6 [Text] Display is 0 (bounds: [760,460,322,80])
  #7 [Button] Equals (bounds: [974,735,107,65], focusable)
  - [Text] Calculator mode: Standard

That is the whole Calculator window. Indentation is tree depth. Lines that start with # are clickable: they have a bounding rectangle and a stable index. Lines that start with a dash are structure, containers and labels the agent reads for context but should not try to click. A model does not need a screenshot to know there is a Seven button at index 3. It is right there as text.

From a 40-property tree to a one-line element

The operating system tree is built for screen readers, so it is exhaustive. The serializer's job is to throw away everything a model cannot act on and keep the four things it can: where the element is, what kind it is, what it is called, and what state it is in. Watch the transformation.

The index is the whole trick

This is the part that does not show up in the usual explanations, and it is the reason an accessibility-tree agent is steadier than a screenshot agent. A vision-only computer use agent has to output a coordinate. To click Save it recognizes the button in the pixels, estimates its center, and emits something like (1218, 22). That number is wrong the moment the window moves, the DPI scale changes, or the theme shifts.

Terminator removes coordinates from the model's job entirely. When format_node assigns an element a #index, it also stores that index in a map called index_to_bounds, keyed by the number, valued by the element's role, name, bounds, and selector. The model reads the list, picks #7, and calls click_element in index mode. The framework looks 7 up, recovers the real element, and drives the click through the platform's native invoke path. The model produced an integer. It never needed to know a coordinate exists.

/// Format a UI tree as compact YAML with #index [ROLE] name format
///
/// Output format:
///   #1 [ROLE] name (bounds: [x,y,w,h], additional context)
///     #2 [ROLE] name (bounds: [x,y,w,h])
///       - ...
///
/// Elements with bounds get a clickable index first.
/// Elements without bounds use a dash prefix.

That doc comment sits directly above format_tree_as_compact_yaml in the source. The index is not a presentation detail. It is the contract between the model and the framework. The deeper version of this idea, where the same indexed list also covers OCR and vision sources, is in the seven-mode click_element router guide.

Reading one node

Every indexed line follows the same grammar. Once you can read one, you can read the whole window.

#6 [Button] Save (bounds: [1180,8,80,28], focusable, disabled)

  #6          1-based clickable index. This is what the agent emits.
  [Button]    role. Button, Edit, MenuItem, CheckBox, ComboBox, ...
  Save        name. The accessible label the OS already exposes.
  bounds      [x, y, width, height] in screen pixels. Present = clickable.
  focusable   state flags: focusable, focused, disabled, selected, toggled.

The presence of a bounds rectangle is what decides whether an element gets an index at all. An element with bounds is on screen and clickable, so it earns a #index. An element without bounds gets a dash. State flags such as disabled, focused, selected and toggled are appended only when they are true, which keeps each line short. A model can answer "is the checkbox already on?" without a screenshot because the word toggled either is or is not on the line.

The same window, two ways a model can see it

Why this matters for native apps specifically

Browser automation has the DOM. It is structured, queryable, and stable, which is why Playwright works as well as it does. Step outside the browser and that structure disappears. A native app has no DOM. What it has is the accessibility tree, the one structured description of its UI the operating system already maintains for screen readers. A computer use agent that reads the tree gets a DOM-like target for Notepad, Outlook, the Settings app, and a legacy line-of-business tool, all through one interface.

And because the browser itself publishes an accessibility tree, the same agent code reaches into a web page when it needs to. The tree does not stop at the browser window the way the DOM does. Terminator is shaped like Playwright on purpose, but the surface it targets is the whole operating system rather than one tab.

Where the tree is honest about its limits

The accessibility tree is the right primary input for native apps, but it is not complete. Some surfaces render their content as pixels with no accessible structure underneath. An agent built only on the tree will hit those surfaces and stop. A production agent keeps the tree as the default and falls through to a vision source when the tree returns nothing useful.

The honest framing is not "tree or vision". It is tree first because it is fast, structured, and cheap on tokens, then vision second for the surfaces above where the tree is a single opaque node. Terminator wires both behind the same click_element tool so the agent does not have to switch interfaces when it crosses that boundary.