Accessibility tree vs screenshot desktop automation: it is a router, not a binary choice

The accessibility tree is faster, and the gap is structural, not implementation-specific. An AX-tree click on Windows resolves to IUIAutomationElement.FindFirst followed by IUIAutomationInvokePattern.Invoke, two COM cross-process calls that complete in single-digit milliseconds. A screenshot click has to capture the framebuffer (xcap on Windows pulls roughly 8 MB of RGBA at 1920x1080), encode to PNG, base64 the bytes, post the payload to a vision backend (OmniParser, Gemini, or local OCR), wait for the model to return bounding boxes, scale the coordinates back into screen space, then send a synthetic mouse event. Terminator's llms.txt at line 243 puts the gap at 100x ('CPU speed, not LLM inference'). The dominant cost on the screenshot path is the network round-trip and model inference, not the screenshot itself.

When the target surface has no accessibility provider, or has one that lies. Three categories: full-screen DirectX or OpenGL games where the OS sees one HWND containing a frame buffer; canvas-rendered design surfaces (Figma's drawing area, Photopea, web-based 3D tools) where every tool's hit region lives inside one canvas element; legacy ActiveX or custom-painted line-of-business widgets that draw their own pixels and never registered a UIA provider. In all three the AX tree returns a single opaque element and the only addressable thing is pixel coordinates. Terminator's click_element exposes vision_type='omniparser' and vision_type='gemini' for exactly these cases. Everywhere else the tree is the right default.

Yes, that is the design. In Terminator's MCP server, click_element is one tool with three modes (selector, index, coordinates) and the index mode takes a vision_type parameter that selects which grounding source to dereference. The enum is declared at crates/terminator-mcp-agent/src/utils.rs line 1062 with five variants: UiTree (the default at utils.rs:835), Ocr, Omniparser, Gemini, and Dom. The dispatch lives in server.rs starting at line 2609 as one match arm. The caller picks the source per click. The same agent can use ui_tree on a sidebar button and gemini on a canvas in the next call without re-instantiating anything.

Three honest reasons. First, the AX tree on macOS Chrome and Safari silently no-ops on AXPress for many web views, so any production desktop engine carries a hardcoded browser bypass list that falls back to synthetic input. Second, vision models like Gemini 2.0 Flash and OmniParser see things the tree does not: a numeric label drawn inside a canvas, an icon with no accessible name, a tooltip that has not yet shown. Third, vendor-built apps occasionally ship with the UIA provider disabled (older Office variants, kiosk shells, locked-down line-of-business apps), and the only entry point is the pixel grid. Production frameworks treat screenshot as a tagged fallback, not the default; that is the difference between a router and a single-source agent.

Yes, twice. First, the captured pixel coordinates of an element shift when the user changes DPI scaling: a button at (412, 300) on a 100% scale 1920x1080 monitor lands at (515, 375) when the same window is dragged to a 200% scale 4K display. Second, vision models trained on one resolution scale degrade when given screenshots at a different resolution. Terminator's omniparser path at server.rs:1141 to 1147 carries explicit DPI scaling (`x = window_x + (box_2d[0] * inv_scale / dpi_scale_w)`) to convert physical-pixel bounding boxes back to logical screen coordinates. The accessibility tree is DPI-invariant by design: the OS reports element bounds in logical units already, and selectors carry semantic identity (role, AutomationId, name) that survives DPI, theme, and resolution changes.

Yes, and the gap is well-defined. Custom controls that draw their own pixels and skip the UIA provider interface show up in the tree as a single opaque element with no children: many games, some terminal emulators, certain 3D modellers, and legacy ActiveX widgets fall in this bucket. Tooltips and hover states are inconsistently exposed; the tree may not see a callout that has not been activated. Custom-rendered icons inside a canvas have no accessible name, and even when the tree sees the canvas element, it cannot tell you that the third icon from the top is the trash button. Vision models can. The right design treats screenshot-and-vision as a labelled fallback, not the default. That is what vision_type=omniparser and vision_type=gemini exist for.

The caller picks one. There is no automatic selection; vision_type defaults to UiTree at utils.rs line 835 (`self.vision_type.unwrap_or(VisionType::UiTree)`). If you call get_window_tree first, the response carries indexed entries from the AX tree and the natural follow-up is click_element with vision_type=ui_tree. If you call capture_screenshot followed by an OmniParser or Gemini parse, the response carries indexed entries from the vision pass and the follow-up is click_element with vision_type=omniparser or vision_type=gemini. The server keeps four separate index maps in memory (uia_bounds, ocr_bounds, omniparser_items, vision_items, dom_bounds) keyed by the tool that populated them, so the dispatch is unambiguous.

Yes. The Anthropic Computer Use beta operates over screenshots: the model receives a screenshot, emits pyautogui-style click(x, y) and type(text) actions, and waits for the next screenshot. This is the binary you might be reading about as 'screenshot desktop automation'. Pixel grounding is general (it works on anything visible) but slow (every action pays inference latency) and DPI-sensitive (the model has to learn the current resolution). The accessibility tree is narrower in surface coverage but deterministic on what it sees and ~100x faster per action. The honest combination: route the bulk of clicks through the tree to keep cost and latency down; route the residual through vision to cover the tree's blind spots. Terminator's terminator-computer-use crate is exactly that combination wired up.

You get a typed error, not a silent miss. The match arm at server.rs:2610 looks up the index in self.uia_bounds and returns McpError::internal_error with the message 'UI tree index N not found. Call get_window_tree first.' if the lookup fails. The same shape holds for vision_type=ocr (looks up ocr_bounds), vision_type=omniparser (omniparser_items), vision_type=gemini (vision_items), and vision_type=dom (dom_bounds). Each vision type has its own index map populated by its own producer tool. The error tells you exactly which producer to call. That is part of why the router design beats a single-source agent: the failure mode is observable instead of 'we tried the tree, fell back to vision, and silently aimed at the wrong pixel'.

It is doing screenshot grounding, then dispatching the click through synthetic input at the resolved coordinates. The screenshot path takes a window or monitor capture, base64-encodes it, posts it to https://app.mediar.ai/api/vision/parse with the Gemini prompt, receives normalized bounding boxes, scales them back into screen space, and finally calls click_at_coordinates_with_type. The mouse event itself uses the same SendInput synthetic-input path that PyAutoGUI uses; Terminator does not pretend to invoke a UIA pattern when there is none to invoke. The honesty is in the labelling: vision_type=gemini means 'I am asking Gemini to find the element for me and then clicking the pixels'. vision_type=ui_tree means 'I am asking UIA to find the element and invoking the pattern'. Two different operations, one tool surface.