Why accessibility APIs beat OCR and pixel matching for OS-level automation

Axis 1 of 3

Latency: one COM call vs a pipeline of screen capture, inference, and synthetic input

Calling invoke() on a UIA element resolves to a single COM call into the IUIAutomationInvokePattern proxy that is already loaded inside the target process. The runtime returns when the target acknowledges. There is no frame buffer involved, no pixel math, no synthetic input event posted to the OS message queue.

The OCR path has a strict minimum number of stages, and you pay all of them on every click. Terminator's own implementation of the Windows OCR path is the cleanest evidence: it lives in crates/terminator/src/platforms/windows/engine.rs starting at line 720, and the stages are visible in the source.

The UIA-pattern alternative collapses every one of those stages into a single message: caller calls invoke_pat.invoke(), target app acknowledges. Terminator's llms.txt at line 243 frames the resulting performance gap as "100x faster (CPU speed, not LLM inference)" specifically against screenshot-based agents like ChatGPT Agents, Claude computer use, and BrowserUse. The multiplier is not the point. The shape of the cost curve is: pattern invocation is bounded by CPU, OCR-then-click is bounded by whatever ML model is doing the recognition.

Axis 2 of 3

Stability: AutomationId is set once in source, pixels are reconstructed every render

The standard argument for accessibility-API stability stops at DPI: if the user drags the window to a 4K monitor, an (x, y) click misses but a tree path does not. That argument is correct and undersold. There are at least five other shifts that move pixels without moving identity, and any one of them is enough to break a pixel-matched script:

• Subpixel anti-aliasing. ClearType on Windows, font smoothing on macOS. The same character at the same size renders to different pixels on different machines.
• Theme changes. Light vs dark, custom accent color, high-contrast mode. Every button repaints.
• Font substitution. If the user does not have the requested font installed, the OS picks a near-match with different glyph widths. The button is now five pixels wider.
• Scrollbar width and chrome. Different OS versions, different accessibility settings, different per-app window-decoration policies all push the content area by a few pixels.
• Animation. Pixel matching mid-fade matches the wrong frame. Half the time the script clicks too early; the other half it clicks the wrong place.

None of those shifts move AutomationId, ControlType, or the element's position in the UIA tree. The selector role:Button && id:save_btn resolves the same element on a 1080p ClearType light-theme box with Segoe UI 9pt as it does on a 4K dark-theme box with a Segoe UI substitute at 11pt mid-fade-in. That is the entire stability story.

The two failure modes that do break an accessibility selector are both honest: the developer renamed the control's AutomationId (which shows up as a typed ElementNotFoundError, not a silent-wrong-click), or the control is a custom-drawn widget that never implemented a UIA provider (in which case the tree is single-node and you legitimately need the OCR fallback below).

Axis 3 of 3 — the one nobody covers

i18n: your OCR is pinned to the user's installed language packs

This is the axis the existing guides on accessibility-vs-OCR all miss. Pull up Terminator's Windows OCR engine creation:

// crates/terminator/src/platforms/windows/engine.rs:763
let ocr_engine = WinOcrEngine::TryCreateFromUserProfileLanguages()
    .map_err(|e| {
        AutomationError::PlatformError(
            format!("Failed to create Windows OCR engine: {e}"),
        )
    })?;

That single Microsoft API call, Windows.Media.Ocr.OcrEngine.TryCreateFromUserProfileLanguages(), decides which languages your automation can read. The documentation is explicit: it iterates the language profiles installed in the user's settings and tries to build an OCR engine that supports them. If none of the installed languages are supported by the OCR runtime, it returns null and Terminator raises a PlatformError. There is no graceful fallback. There is no auto-download. The user's machine is the source of truth for which scripts are even readable.

The trap people fall into is selecting on the visible text instead. Terminator exposes a name: selector that matches the accessible name, and a text: selector that matches visible text. Both of those are translated. So is LocalizedControlType, the property Narrator reads aloud ("button" in English, "Schaltfläche" in German). Terminator's property mapping at crates/terminator/src/platforms/windows/utils.rs lines 165 to 175 keeps LocalizedControlType and the locale-independent ControlType as separate fields specifically so you can choose which one your selector hits.

The discipline that survives locale flips is simple: write selectors against role: (UIA ControlType enum, never translated) and id: (AutomationId, set once in source). Treat name: and text: as last resorts, only when the app developer did not give you an AutomationId. If your selector grammar starts with a string the user can see in their language, you have implicitly opted in to localization risk.

When OCR and pixel matching are actually the right call

None of the above means OCR is wrong. It means OCR is a fallback, not a default. Three honest cases where you reach for it:

1. The target renders through a frame buffer. A fullscreen game on DirectX or OpenGL, a 3D modeller, a custom CAD canvas. The window is one accessibility node and pixels are the only addressable thing.
2. The accessibility bridge does not cross the host boundary. A remote desktop session, a VM viewer, or a Citrix-streamed app. The host machine sees a single video surface where the guest's tree should be.
3. The app embeds a HTML5 canvas or WebGL surface for its real UI. Figma's drawing surface, a browser-based game engine, a canvas-backed data grid. Each tool's hit region lives inside one canvas element with no children.

Terminator ships exactly that fallback. The ocr_screenshot_with_bounds method at engine.rs:720 returns a tree of OCR-detected lines and words with bounding rectangles in absolute screen coordinates, which you can feed back into the same selector grammar. The right mental model is: try the tree, and only if the tree is empty fall through to OCR. That order is the difference between a flaky script and a deterministic one.

What Terminator actually does

Terminator is an open source desktop automation framework for Windows and macOS. The selector grammar is intentionally shaped like Playwright, but the targets are the whole OS. It exposes:

• A Rust core at terminator-rs on crates.io with Windows UIA and macOS AX adapters.
• Node.js bindings at @mediar-ai/terminator via NAPI-RS.
• Python bindings at terminator-py via PyO3.
• An MCP server at npx -y terminator-mcp-agent@latest that exposes desktop control to Claude Code, Cursor, VS Code, and Windsurf as MCP tools.

The source is at github.com/mediar-ai/terminator, MIT licensed. The line numbers cited in this guide are from the current main branch as of 2026-05-17. If the file moves between then and the time you read this, the structure of the argument still holds: structured selectors win on latency, stability, and i18n; OCR is the escape hatch when the tree is empty.