Accessibility API for computer use agents: seven click paths, one MCP tool.

Because the accessibility tree is not a complete description of every pixel that matters. Office documents render their content inside a single canvas element with no AX children for individual cells, paragraphs, or shapes. Electron apps wrap a Chromium renderer and often expose only the outer window through UIA, leaving the actual UI invisible to the AX tree until the chrome.accessibility flag is on. Custom-rendered controls in games, IDEs, and design tools paint pixels directly. An agent that can only click by AX selector hits these surfaces and stops. The fix is not to give up on accessibility, it is to layer alternative grounding sources behind the same click tool so the agent can keep the AX path as default and fall through when the tree is silent.

Three modes plus five vision sources. ClickMode (Selector, Index, Coordinates) lives at crates/terminator-mcp-agent/src/utils.rs lines 728-736. VisionType (Ocr, Omniparser, Gemini, Dom, UiTree) lives at the same file lines 1062-1067. Selector mode resolves an AX selector through the platform engine. Index mode plus UiTree resolves a 1-based clickable index against a UIA tree captured by get_window_tree. Index plus Ocr resolves against the cache populated by include_ocr. Index plus Omniparser resolves against include_omniparser. Index plus Gemini resolves against include_gemini_vision. Index plus Dom resolves against the Chrome extension's DOM bounds. Coordinates is raw screen-space x and y. All seven dispatch through one click_element function at server.rs:2486.

Each grounding source caches bounds in its own map on the MCP server. server.rs holds five separate Mutex<HashMap<u32, ...>>: uia_bounds, ocr_bounds, omniparser_items, vision_items, dom_bounds. When you call get_window_tree with include_ocr or include_omniparser, the server runs the detector, assigns 1-based indices to each detected box, and stores the bounds in the matching map. The model sees lines like '#42 [Button] Save' or '#42 OCR: "Save" bounds [840,32,68,28]'. When the agent later calls click_element with index: 42 and vision_type: "omniparser", server.rs:2660 looks up entry 42 in omniparser_items, takes the box_2d, computes the center, and routes to desktop.click_at_coordinates_with_type. The agent has called back to a vision-grounded element with the same shape it would call back to an AX node.

Fall through. The recommended pattern is to call get_window_tree once with include_ocr: true, include_omniparser: true (or include_gemini_vision: true), and include_browser_dom: true if you are in a browser. The server returns a unified compact YAML with each source prefixed (#u for UIA, #o for OCR, #m for Omniparser, #v for Gemini vision, #d for DOM). The agent sees every grounded element across every source in one tree, picks the highest-confidence one for each click, and calls click_element with the matching vision_type. If the AX tree exposes the element, the agent uses index plus UiTree; if it doesn't, but OCR caught the label, the agent uses index plus Ocr; if nothing else worked but a vision model spotted an icon, the agent uses index plus Gemini. Coordinates is the last resort, used only when none of the indexed sources caught the element.

Two reasons, both about agent reliability. First, the index is a stable handle the agent can carry across reasoning steps without re-deriving it. The model is more accurate calling back to '#42 the omniparser-detected Save icon at [840,32,68,28]' than computing those coordinates fresh from a screenshot every turn. Second, the server is the one that converts the index to actual screen coordinates, so DPI scaling, window offset, and any image resize the detector applied are handled in one well-tested function. Compare that to the Gemini computer use loop in terminator-computer-use, where the model emits 0-999 normalized coordinates and convert_normalized_to_screen has to apply four sequential transforms (resize_scale, DPI, window x, window y) for each click. Index dispatch skips the math; the bounds were captured at real screen coordinates from the start.

Wire Terminator's MCP server into the agent's tool list. The agent now sees one click_element tool instead of a screenshot-coordinate tool. Its first call is get_window_tree with include_ocr: true and include_omniparser: true. The compact YAML comes back with up to five interleaved grounding sources, indexed. From that point on, every click_element call carries either a selector, an index plus vision_type, or coordinates. The agent's reasoning becomes 'this Save button is in the AX tree as #88, click that' or 'the icon I want is not in the AX tree but Omniparser caught it as #156, click that with vision_type omniparser'. The agent never has to choose at boot time whether to be an accessibility agent or a vision agent. It is both, per click.

AX tree first, DOM second (in browsers), Omniparser or Gemini-vision third, OCR fourth, raw coordinates last. The order is rank-by-stability. AX selectors and AX indices reference elements the OS itself classified, so they are robust across re-renders, DPI changes, and window resizes. DOM indices come from the live Chromium tree, also stable as long as the page hasn't navigated. Omniparser and Gemini-vision boxes are stable for as long as the captured screenshot is, then need re-grounding. OCR is the least stable because text recognition jitters around character boundaries. Raw coordinates are the most fragile because nothing on the screen is anchored to them; a window move invalidates the click. The same click_element call accepts all of them, so an agent can encode this priority in its system prompt and still emit the same JSON shape.

Negligible. ClickMode::determine_mode at utils.rs:740 is a constant-time check on which arguments are present. The vision_type match in server.rs:2609 is one HashMap lookup keyed by index. The bottleneck for any of these paths is the platform-side click, not the dispatch. The interesting cost is the up-front call to get_window_tree with the include_* flags on, because OCR and Omniparser run on the captured screenshot before returning. That happens once per agent turn rather than once per click, and the result is cached and indexed for every click_element call until the next get_window_tree.

Use the MCP server with click_element when you are wiring an existing agent (Claude, custom OpenAI tool-use, your own harness) that already has its own reasoning loop. Use the bundled Desktop::gemini_computer_use loop in crates/terminator/src/computer_use/mod.rs:483 when you want a self-contained agent that runs end-to-end against the Gemini Computer Use API. The bundled loop is screenshot-only and uses normalized 0-999 coordinates throughout; it is the right choice if you want to ship a working agent in fifteen lines of Rust without designing your own tool layer. The MCP path is the right choice when the AX tree is the primary input and vision is a fallback. Both share the same desktop click implementation underneath.

Track it. Each grounding source has its own bounds map, and the indices are namespaced per source by the prefix on the line in get_window_tree's output (#u, #o, #m, #v, #d). Calling click_element with index: 42 and the wrong vision_type will either miss or land on a different element. In practice the agent does not re-derive this; it copies the prefix and the integer from the line it just read. The compact YAML's prefixing is what makes the seven-mode router safe to expose to an LLM without requiring it to memorize the cache topology.