Self-driving agents and the legacy desktop ceiling: what actually breaks, and what to do about it

The ceiling, in one sentence

A self-driving agent is only as autonomous as its weakest grounding source on the surface in front of it. On a modern UIA-clean app the planner can be a 7B model and still hit 90%+ task completion because the click target is unambiguous. On a 1998 MFC window the planner can be a frontier model and you still get 30% completion if the runtime cannot read the controls. The autonomy level is not a property of the model. It is a property of the runtime that translates the model's intent into a click.

What “legacy” means here, structurally

For the purposes of a self-driving agent, an app is “legacy” when its accessibility metadata sits on the wrong side of three Microsoft transitions. First, it predates UIA (introduced with Windows Vista, 2006), so the tree is populated by the MSAA-to-UIA bridge rather than by native UIA providers. Second, it was built before the UIA pattern model became normative (InvokePattern, ValuePattern, ExpandCollapsePattern), so its controls expose only the older IAccessible default-action verb. Third, in many cases the developer never implemented IAccessible at all, so what UIA can extract is whatever the default Win32 IAccessible proxy can scrape from the HWND tree. The end result is a UIA tree that is shallow, mostly unnamed, and whose only non-empty fields are in the LegacyIAccessible* namespace.

The eight properties that keep an autonomous agent alive on those windows

Every UIA wrapper that wants to recover something useful from a legacy control reads from the MSAA bridge. In Terminator, the mapping is in crates/terminator/src/platforms/windows/utils.rs at lines 198-205:

// Text properties
"LegacyIAccessibleValue"            => Some(UIProperty::LegacyIAccessibleValue),
"LegacyIAccessibleDescription"      => Some(UIProperty::LegacyIAccessibleDescription),
"LegacyIAccessibleRole"             => Some(UIProperty::LegacyIAccessibleRole),
"LegacyIAccessibleState"            => Some(UIProperty::LegacyIAccessibleState),
"LegacyIAccessibleHelp"             => Some(UIProperty::LegacyIAccessibleHelp),
"LegacyIAccessibleKeyboardShortcut" => Some(UIProperty::LegacyIAccessibleKeyboardShortcut),
"LegacyIAccessibleName"             => Some(UIProperty::LegacyIAccessibleName),
"LegacyIAccessibleDefaultAction"    => Some(UIProperty::LegacyIAccessibleDefaultAction),

Those eight properties are the difference between an agent that can save a record in a 25-year-old MFC app and an agent that emits coordinates derived from a screenshot. The LegacyIAccessibleName field carries the button label that UIA's Name often does not. The LegacyIAccessibleDefaultAction field carries the verb (“Press”, “Open”, “Toggle”) that lets the runtime invoke the control without a synthesized mouse event. The LegacyIAccessibleRole field disambiguates a button from a menu item from an edit when the modern ControlType comes back as Pane by default.

Why one tool signature, not two

The recurring temptation is to expose the grounding sources as separate tools (click_by_selector, click_by_ocr, click_by_vision, click_by_coords) and let the model pick. That fails for a structural reason. The model has no way of knowing which source will work on the surface it is looking at without trying. So it tries one, observes the failure, tries another, observes that failure, and burns its planning budget on bookkeeping. The grounding source is a runtime concern, not a planning concern. It belongs in one tool that takes a selector or an index, and the runtime decides which source to consult by trying them in priority order.

Terminator's click_element accepts three modes (selector, index, raw coordinates) and one vision_type field that tells the runtime which index source the model is referring to. The five legal index sources are defined as one enum at crates/terminator-mcp-agent/src/utils.rs lines 1062-1073:

#[derive(Debug, Clone, Copy, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "lowercase")]
pub enum VisionType {
    Ocr,
    Omniparser,
    UiTree,
    Dom,
    Gemini,
}

From the model's view there is one tool. From the runtime's view there are five sources of grounding plus a coordinate escape hatch, and the LegacyIAccessible bridge sits inside the UiTree path so a selector-based call already gets the legacy-aware fall-through for free.

The ceiling, by tier

Stack the grounding sources from cheapest to most expensive, and the autonomy ceiling on a given UI is whichever tier first returns a usable element. A self-driving agent is not asking which tier is “best” on average; it is asking which one matches the surface in front of it right now.

The three failure modes that survive even fall-through

Even a well-implemented chain has a bottom. Three categories of UI surface stay hard for any autonomous loop:

1. Owner-drawn controls. Custom grids, third-party drawing libraries that paint via raw GDI, in-house chart widgets that never call any IAccessible API. The MSAA bridge returns ROLE_SYSTEM_CLIENT (43) with no name and no default action. OCR or vision is the only path. Latency goes up by an order of magnitude on these steps.
2. Apps with UIA explicitly disabled. A small population of game-adjacent and industrial apps disable UIA for performance. The tree is empty by design. Vision is the only viable source. Plan workflows around a small number of these, never as the steady state.
3. Direct2D / swap-chain text. Modern UIs that render text into a swap chain bypass GDI text rendering and therefore are invisible to UIA in many cases. OCR is the recovery path; even a strong vision model handles them comfortably because the rendered text is sharp.

The point of the chain is not that it eliminates these. It is that the agent does not collapse when it hits one. It degrades to a slower tier, the loop continues, and the cost shows up only on the steps where it is unavoidable.

The minimum change that moves the ceiling

If you are running an autonomous loop on top of Anthropic computer use, OpenAI Operator, or Gemini computer use today, the smallest useful change is to stop letting the model emit raw screen coordinates as its first move. Put a structural-grounding tool in front of the model and let the runtime fall through to vision only when the structural sources fail. Concretely:

• Install an MCP server whose click_element tool reads the UIA tree and the LegacyIAccessible bridge before falling through to OCR and vision. (Terminator is one such server: claude mcp add terminator "npx -y terminator-mcp-agent@latest".)
• Wire the same tool into Cursor, VS Code, or Windsurf via the same MCP config so the agent can drive native windows the same way it drives the editor.
• Reserve the model's coordinate-emitting capability for the canvases and PDFs where structural grounding genuinely cannot help. Treat any coordinate the model emits on a non-canvas surface as a smell, not a feature.