Accessibility tree desktop agents: the browser is already a UIA element, that is what closes the gap

Every modern OS exposes a structured tree of every visible UI element so that screen readers and assistive tech can describe a window without reading pixels. On Windows that surface is UI Automation (UIA); on macOS it is AXUIElement; on Linux it is AT-SPI2. Each node carries a role (Button, Edit, MenuItem, Document), a name (Save, Subject, Cancel), enabled and focused state, and a stable bounding rectangle. A desktop agent that reads this tree gets the same description the OS gives a screen reader, in milliseconds, with no inference required. A pixel-driven agent has to reconstruct that same information from a screenshot every step, with a vision model, at thousands of visual tokens per window. The structured path is faster, cheaper, and survives DPI changes and theme changes that break pixel matching.

Because the browser itself is a node in the OS accessibility tree. On Windows a Chrome window appears as a UIA element with role:Document underneath the chrome.exe process; on macOS the equivalent AX role exists on the WebKit and Blink content area. The web page inside that document is already exposed as accessibility children that mirror the DOM's accessible nodes. So an agent that speaks role:Button name:Save against the OS tree can walk a native dialog, and an agent that speaks role:Button name:Subscribe against the OS tree can walk the page inside the active tab. There is no second tool, no second selector grammar, no boundary the agent has to be aware of. The tree spans both worlds because the OS already has to span both worlds for assistive tech.

Two reasons that come up in real workflows. First, the OS accessibility view of a page is structurally accurate but coarser than the DOM. It does not include CSS attributes, computed styles, the full hierarchy of generic divs, or the pseudo-state of hover and focus rings. Second, in-page JavaScript execution (filling a controlled React input, dispatching a synthetic event, reading shadow DOM) needs a real JS context. The Manifest V3 extension at crates/terminator/browser-extension/manifest.json (version 0.24.32) holds a WebSocket on 127.0.0.1:17373 and gives the agent a way to call execute_browser_script when the structural tree is not enough. The extension is an enrichment over the AX path, not a replacement for it. The agent decides which one to use per call, and the click target lands in the same screen-space coordinate either way.

DOM coordinates and OS coordinates are in different spaces. getBoundingClientRect returns CSS pixels relative to the viewport. UIAutomation returns physical pixels relative to the screen. On a Retina or 4K display window.devicePixelRatio is 2.0 or higher, so a CSS pixel rect at (320, 200) is at (640, 400) in the OS frame, plus the screen offset of the browser viewport. Terminator's capture_browser_dom_elements at crates/terminator-mcp-agent/src/server.rs:838 reconciles them in two steps. It first locates the UIA element with role:Document for the active tab and reads its bounds(); the (x, y) of that element is the viewport offset in screen space. Then for every DOM element it serializes, it multiplies the CSS pixel rect by window.devicePixelRatio (server.rs:911-914). After those two scaling steps a click computed from a DOM rect lands in the same place as a click computed from a UIA element. Without that reconciliation, the agent drifts on every laptop with non-100% scaling.

Yes for the tree itself. AXUIElement on macOS exposes the same role / name / bounds shape, with values like AXButton, AXTextField, AXWindow that map to the same selector grammar Terminator uses on Windows. AT-SPI2 on Linux is consistent for GTK and Qt apps; coverage of Electron and Chromium on Linux varies by distribution. The Manifest V3 browser extension that fills in the in-page enrichment is platform-agnostic because Chrome is the same on all three, but the macOS port of the MCP server has a smaller surface than the Windows build at this point. If you are building a native-only agent on macOS or Linux, the AX path is solid; the cross-OS browser extension path on macOS still has rough edges worth pinging us about before you commit.

Three concrete failure surfaces. First, fullscreen DirectX, OpenGL, or Metal surfaces (games, some IDE editors, the Figma drawing canvas) have an empty or single-node AX subtree because the app paints pixels directly. Second, sandboxed RDP and VM viewers do not bridge the AX tree across the host boundary; you see one opaque element where the remote screen is. Third, custom-drawn controls in legacy line-of-business apps that never implemented a UIA provider show up as Pane or generic Custom elements with no useful name. The right architecture is to keep the tree as the default path because it is fast and tiny-model-friendly, and stack a vision-grounded fallback for those three cases. Terminator's click_element MCP tool exposes seven grounding modes for exactly this reason; the AX selector path is mode one, the OCR / Omniparser / Gemini / DOM / coordinate paths are the fallbacks.

On a moderately complex window (a Notepad++ session, an Outlook compose pane, a populated browser tab) the cached UIA subtree fetch in tree_builder.rs:388 with TreeScope::Subtree comes back in tens of milliseconds because the cache pre-fetches every property in one IPC call (ControlType, Name, BoundingRectangle, IsEnabled, IsKeyboardFocusable, HasKeyboardFocus, AutomationId at lines 404-412). Walking the same window node-by-node with separate IPC calls is one to two orders of magnitude slower; this is why the cached path exists. A screenshot of the same window costs roughly the same wall time on the capture side, but the perception cost on the model side is many thousands of visual tokens to read versus a few thousand text tokens for the tree. The tree wins on every dimension except the three failure surfaces above.

No. Vision still wins for fullscreen rendered surfaces, for sanity-checking that the agent is on the right screen at all, and for any task where the visual layout itself is the data (a slide layout, a chart, a design canvas). The right framing is that the accessibility tree is the default tool the agent reaches for, screenshots are the escalation when the tree is silent, and a production agent stack stitches both. The thing this page is arguing against is the symmetric mistake: building a screenshot-only agent for the desktop. That stack burns visual tokens to read text the OS already gave you for free, and it inherits every brittleness of pixel matching across DPI, theme, language, and resolution.

Install the MCP agent: claude mcp add terminator npx -y terminator-mcp-agent@latest. That registers the tool surface (get_window_tree, click_element, type_into_element, press_key, capture_screenshot, execute_browser_script) with your assistant. The defaults connect to the OS UIA / AX tree out of the box; the browser extension is opt-in. From inside Claude Code or Cursor, ask the agent to run get_window_tree on the focused window and inspect the JSON it returns; that is the same input shape any model is going to see. Once you can read the tree, the click and type tools accept role+name selectors directly. Repo at github.com/mediar-ai/terminator, source for everything in this article is in crates/terminator-mcp-agent/src/server.rs.