Accessibility tree automation vs PyAutoGUI: the two clicks are not the same operation

Accessibility tree automation calls a method on a UI element. PyAutoGUI sends a mouse event at coordinates. On Windows, the tree path resolves to IUIAutomationInvokePattern::Invoke, a COM call that runs inside the target process and returns a binary success or failure. The PyAutoGUI path resolves to SendInput with MOUSEEVENTF_ABSOLUTE plus MOUSEEVENTF_MOVE plus MOUSEEVENTF_LEFTDOWN plus MOUSEEVENTF_LEFTUP, a kernel call that injects HID events at normalized screen coordinates. The two operations have different types. One asks the element to perform its default action; the other moves a virtual mouse and presses a virtual button at a screen location and hopes the right element is under it.

When the target does not expose a tree. That is the only honest case. Three real examples: a fullscreen game rendered through DirectX or OpenGL where the only thing on the screen is a frame buffer; a canvas-based design tool like Figma's drawing surface where each tool's hit region lives inside a single canvas element with no children; a sandboxed remote desktop or virtual machine viewer where the accessibility bridge does not cross the host boundary. In all three the AX or UIA tree is empty or single-node, and pixel coordinates are the only addressable thing. Everywhere else, picking PyAutoGUI is choosing the OS's worst-case input path on purpose.

It is faster, and the cause is not mysterious. SendInput on Windows posts events into the message queue, which the OS schedules with timing that depends on cursor smoothing settings, foreground priority, and any IME or low level keyboard hook installed by other software. A typical click is at minimum the time to move plus debounce plus down plus up, and most automation tools add a 10 to 50 ms safety pause around each event so the target can react. UIInvokePattern.invoke runs as a COM cross process call and returns when the target process acknowledges. There is no virtual cursor motion, no debounce window, no foreground gating. On Terminator's own claim of 100x faster than screenshot or pixel based agents, the source is the llms.txt at line 243: 'CPU speed, not LLM inference', which describes pattern invocation specifically.

Because (x, y) is a coordinate in screen space, and screen space changes. A button at (412, 300) on a 1920x1080 monitor at 100% scale is at (515, 375) on the same window dragged to a 200% scaled 4K monitor. PyAutoGUI's locateOnScreen function (the higher level alternative) takes a reference image and template matches against the current screenshot pixel by pixel; PyAutoGUI's own docs note that 'if a single pixel is a different color, locateOnScreen will not find the image'. The accessibility tree carries semantic identity (role, automation id, name) that survives DPI changes, theme changes, resolution changes, and most layout changes. The only thing that breaks the tree path is the developer renaming the control or removing the AutomationId, and that shows up as a typed ElementNotFoundError instead of a click on the wrong place.

Two things, both worth knowing. First, custom controls that draw their own pixels and do not implement the AX or UIA provider interface show up in the tree as a single opaque element. Some games, some terminal emulators, some 3D modellers, and some legacy ActiveX widgets fall in this bucket. Second, write actions through the AX bridge sometimes silently no-op on browser web views: AXPress and AXClick on macOS Chrome and Safari return success and do nothing, which is why production AX engines maintain hardcoded browser bypass lists and fall back to synthetic input on those apps. The tree is the right default; the synthetic input path is the right fallback; PyAutoGUI is the right tool when there is nothing else.

On Windows, Terminator's invoke() at crates/terminator/src/platforms/windows/element.rs lines 838 to 859 is twenty two lines of Rust. It calls get_pattern::<patterns::UIInvokePattern>() on the wrapped IUIAutomationElement, classifies the error into UnsupportedOperation or PlatformError, and calls invoke_pat.invoke(). No SendInput, no GetCursorPos, no screen metrics math. The fallback click() path lives in crates/terminator/src/platforms/windows/input.rs at function send_mouse_click starting line 38, and it is the canonical SendInput dance: GetCursorPos to save state, GetSystemMetrics for SM_CXSCREEN and SM_CYSCREEN, multiply by 65535 and divide by screen size to normalize, build three INPUT structs (move, down, up), call SendInput on each. Roughly eighty lines for the path that PyAutoGUI's whole click is conceptually a port of.

Yes, and that is the largest practical difference. Pattern invocation does not move the cursor or change keyboard focus. You can run an automation against a background Excel workbook while continuing to write code in another window. PyAutoGUI cannot do this; every click physically moves your cursor and every typewrite physically presses your keys. If your cursor was hovering a save button when the script fires, the script's click and your accidental human click queue together, and which one wins is undefined. Terminator markets this as 'does not take over your cursor or keyboard', and that statement is literally true for every action that resolves to a UIA pattern: invoke, toggle, expand_collapse, set_selected, and type_text against a value pattern edit control.

Several, with tradeoffs. pywinauto is the closest historical match on Windows: it walks UIA, exposes a Python-shaped API, and has been around for a decade. Its weakness is concurrency; it is fully synchronous and holds the GIL through every UIA call, so asyncio.gather cannot parallelize it. Terminator's Python binding (pip install terminator-py, import terminator) wraps the same Rust UIA core that drives the MCP server, and every awaitable releases the GIL onto a Tokio reactor, which means asyncio.gather across multiple apps actually overlaps. atomacos and pyobjc-framework-ApplicationServices cover the macOS AX side. None of them ship the synthetic input fallback list out of the box; you usually wire that in yourself when a target lies about supporting AXPress.

PyAutoGUI is not dead, and it is the right answer for a narrow set of problems: scripts that operate over genuinely opaque graphical targets (games, canvas apps, presentations, retro emulators), educational examples where a beginner needs to see a cursor move, automation across an OS or app combination where no AX bridge exists. The tutorial overrepresentation comes from PyAutoGUI being the easiest possible thing to demo in five lines of Python. The cost of that simplicity shows up in production: every script that reaches a thousand actions ends up either (a) growing a screenshot diff harness to detect when a pixel changed, or (b) being rewritten against an accessibility library because the screenshot harness still misses things on a different DPI.

The accessibility tree, with PyAutoGUI as the bottom of the fallback stack. An agent that takes screenshots and asks an LLM to output (x, y) coordinates is paying the LLM inference latency on every action, which is the path ChatGPT Agents and Claude computer use take. Terminator's own claim is roughly 100x faster on this loop because invoke() is a COM call that does not require a model in the loop at all; the model picks a selector, the framework resolves it, and the action fires. PyAutoGUI underneath catches the cases where the tree returns nothing, and OCR plus vision AI catches the cases where even pixel coordinates are not stable. The tree is not the only layer, but it is the layer that turns an agent loop from inference-bound to CPU-bound.