Claude computer use, and the selector-based path the articles skip

Anthropic ships a tool type called computer (current revision computer_20251022). When enabled, your harness is responsible for taking a screenshot of the desktop and sending it to Claude alongside the tool definition. Claude returns a tool_use block whose input is an action like left_click with a coordinate pair in pixel space. Your code executes the click, takes another screenshot, sends both back, and the loop continues. Every action is one screenshot upload and one model round-trip. This is not a limitation of Claude, it is how the tool is defined: the model sees pixels, you execute pixels.

Two reasons, both mechanical. First, every action pays for one image token budget plus output tokens, and screenshots are not tiny even after downscaling. Anthropic's own computer use docs note that long tasks with frequent screenshots consume significant credits. Second, wall-clock latency: every action is one full inference pass, typically one to several seconds. A 40-action workflow becomes a coffee break. A deterministic selector-driven agent can do the same 40 steps in seconds because the model is not in the inner loop.

32 typed tools that speak accessibility-tree selectors, not coordinates. You can list them by opening crates/terminator-mcp-agent/src/server.rs at line 9953, where the dispatch_tool match block has one arm per tool. Examples: click_element takes a selector like role:Button && name:Save, type_into_element takes a selector plus text, navigate_browser drives the address bar, execute_browser_script runs JS inside the page, execute_sequence accepts a YAML of steps. The complete list at the time of writing: get_window_tree, get_applications_and_windows_list, click_element, type_into_element, press_key, press_key_global, validate_element, wait_for_element, activate_element, navigate_browser, execute_browser_script, open_application, scroll_element, mouse_drag, highlight_element, select_option, set_selected, capture_screenshot, invoke_element, set_value, execute_sequence, run_command, delay, stop_highlighting, stop_execution, gemini_computer_use, read_file, write_file, edit_file, copy_content, glob_files, grep_files.

Because the operating system already knows where every element is. Windows UI Automation and macOS Accessibility both expose a live tree where each element has a role (Button, Edit, Text, Window), a name, a value, bounds, and a parent chain. Terminator finds elements by matching that tree. role:Button && name:Save survives the button moving by 100 pixels, the window being resized, a theme change, or a DPI shift. A coordinate pair does not. The system prompt Terminator sends to Claude, in crates/terminator-mcp-agent/src/prompt.rs line 21, makes this explicit: always derive selectors strictly from the provided UI tree or DOM data; never guess or predict element attributes based on assumptions.

Yes. Terminator's MCP agent exposes both a capture_screenshot tool and a vision-model fallback called gemini_computer_use (server.rs line 10267) for cases where the accessibility tree is empty or lies. The normal flow is: Claude calls get_window_tree first, reads the tree, picks the selector it needs, calls click_element or type_into_element. If an element is truly invisible to accessibility, you can escalate to a visual path. The point is that the selector path is the default, not the exception, and most Windows and macOS apps expose enough tree to skip vision entirely.

One command: claude mcp add terminator "npx -y terminator-mcp-agent@latest" -s user. That registers the server at the user scope, runs it over stdio under Claude Code's supervision, and exposes the 32 tools inside the normal tool picker. For Cursor, VS Code, Windsurf, or any other MCP client, the same server binary works with a standard mcpServers JSON block pointing at the same npx command. Full instructions live in crates/terminator-mcp-agent/README.md on GitHub.

It replaces the default path for the actions where accessibility tree is faster and more reliable, which is most of them on Windows and macOS. Terminator's README is explicit about the goal: run 100x faster than the pixel-loop agents and hit above 95% success rate by keeping the model out of the inner loop. Anthropic's native computer use still matters for fully alien UIs (games, some Electron apps that do not expose their tree, canvas-heavy apps), which is why Terminator keeps a vision-model tool available as a fallback rather than pretending it is never needed.

Yes, and this is the detail worth looking at yourself. crates/terminator-mcp-agent/build.rs at line 31 defines extract_mcp_tools(), a build-time function that opens src/server.rs, scans for the let result = match tool_name line, and collects every subsequent "tool_name" =>. That list becomes the MCP_TOOLS environment variable via println!("cargo:rustc-env=MCP_TOOLS=..."). Then prompt.rs reads env!("MCP_TOOLS") at compile time and pastes it into the system instructions Claude receives. The practical consequence: the server cannot tell Claude about a tool that dispatch_tool does not handle, and Claude cannot see a stale tool list. They are the same list by construction.