Open source computer use agents in April 2026, and the math most survey posts skip

A computer use agent is a system where an AI model can see the state of a computer and drive its input devices — cursor, keyboard, scroll — toward a goal. Open source means the harness, the tools, and usually the system prompt are on GitHub under a permissive license; the model itself can be proprietary (Claude, Gemini) or open (UI-TARS, Qwen-VL). In April 2026 the space splits into three architectures: pure screenshot-and-vision (works on any OS, slow and expensive), pure accessibility-tree selectors (fast and robust, OS-specific), and hybrid systems that start with selectors and fall back to vision when the tree does not describe the element.

Two things in one binary. First, it exposes 32 selector-based tools over MCP, so a coding assistant like Claude or Cursor drives the desktop by role:Button && name:Save rather than pixel coordinates. Second, it also exposes a vision loop via the gemini_computer_use tool, and it ships the bridge code that connects the two worlds. That bridge is the convert_normalized_to_screen function at crates/terminator-computer-use/src/lib.rs line 336, which takes a model's 0-999 normalized coordinate and applies four transforms (divide by 1000, divide by resize_scale, divide by dpi_scale, add window offset) to produce a real pixel. Most hybrid agents keep that math private; Terminator ships it in public Rust.

None of them are optional on a real Windows machine. Step 1 exists because the model is trained on a normalized 0-999 grid, not your display. Step 2 exists because we have to resize the screenshot before uploading (Gemini Computer Use caps the long edge around 1024 pixels); the scale we used to shrink has to be undone to find where in the original capture the click belongs. Step 3 exists because screenshots live in physical pixels but UI Automation coordinates live in logical pixels, and on a 150% scaled display those differ by a factor of 1.5. Step 4 exists because up to that point we have been measuring inside the screenshotted window, and we have to add the window's top-left to land on the virtual desktop. Skip any one of them and your click is either off the button or off the screen.

Gemini emits keyboard input in one dialect (control+a, Meta+Shift+T) and Windows UIA accepts a different one ({Ctrl}a, {Win}{Shift}t). translate_gemini_keys in the same file (lines 212 to 313) walks each plus-separated token, maps modifiers to bracket forms, handles enter, tab, escape, backspace, delete, space, arrows, and the f1 through f24 range with an explicit 1..=24 match so a model hallucinating 'f99' fails loudly instead of silently. Single ASCII characters are only valid as the last segment of a combination. It is small code, but it is the kind of code a survey post never shows.

Open crates/terminator-mcp-agent/src/server.rs and jump to the dispatch_tool function (around line 9953 at the time of writing). There is a single match block on tool_name; each arm is one MCP tool. At the time of this page: get_window_tree, get_applications_and_windows_list, click_element, type_into_element, press_key, press_key_global, validate_element, wait_for_element, activate_element, navigate_browser, execute_browser_script, open_application, scroll_element, mouse_drag, highlight_element, select_option, set_selected, capture_screenshot, invoke_element, set_value, execute_sequence, run_command, delay, stop_highlighting, stop_execution, gemini_computer_use, read_file, write_file, edit_file, copy_content, glob_files, grep_files. The vision loop is one of the 32.

Today, yes. The README is explicit: Windows support is the stable surface; macOS and Linux are not currently supported for the automation features. The computer use crate itself compiles cross-platform, but the UIA invocations that execute the converted coordinates are Windows APIs. If you are on macOS and want an open source, accessibility-first agent, Fazm is the closest counterpart in the April 2026 space.

Prefer selectors whenever the accessibility tree describes what you want. Regular apps, browsers, native Windows dialogs — these all have a tree, and role + name selectors survive layout changes that break pixel coordinates. Prefer vision for surfaces that do not expose structure: canvas-based drawing apps, games, custom-rendered dashboards, WebGL content, remote desktops, anything where the 'button' is just painted pixels. A well-built hybrid agent tries the cheap path first and only pays for the screenshot round-trip when there is nothing structural to target.

To https://app.mediar.ai/api/vision/computer-use by default, overridable with the GEMINI_COMPUTER_USE_BACKEND_URL environment variable. The payload is a JSON body with three fields: image (base64 PNG), goal (the user task), previous_actions (history with their screenshots and success flags). The backend proxies to Gemini. Source: terminator-computer-use/src/lib.rs line 140, call_computer_use_backend, with a 300-second timeout.