MCP desktop accessibility automation: an honest reader’s guide to the holes

Why a tour of the holes is more useful than another tour of the features

The features are well covered. The accessibility tree is structural; it survives DPI, theme, scroll, and most localisation. UIA exposes Control Patterns (Invoke, Toggle, Value) that fire writes without moving the cursor. The MCP server bundles roughly 35 tools end to end and routes them through one dispatch table. All of that is real and you can read about it on a dozen pages including a few on this site.

What you cannot easily find is the inverse list: the cfg-gated subsets that quietly return success on the platform they do not support, the protocol features the most popular MCP clients have not shipped yet, the parsing edge cases the LLM will produce on the first try, the surfaces the accessibility tree never exposes. That is what this page is. Each section names one hole, points at the source line that pins it down, and notes what to do today.

Hole 1: most MCP clients still cannot pause a tool to ask the user a question

The MCP spec has a primitive called elicitation. The server can pause a tool, send the client a structured question with a JSON schema for the expected answer, and resume the tool when the user fills it in. It is the right primitive for “I see seven Save buttons in the tree, which one did you mean?” and “this step deletes a row, are you sure?”. Terminator defines six elicitation schemas at crates/terminator-mcp-agent/src/elicitation/schemas.rs: WorkflowContext, ElementDisambiguation, ErrorRecoveryChoice, ActionConfirmation, SelectorRefinement, UserResponse.

The hole is that the documentation comment in elicitation/mod.rs lines 14 to 18 says verbatim:

“As of December 2025, Claude Desktop and Claude Code do not yet support elicitation. The implementation includes graceful fallback for unsupported clients.”

The graceful fallback is real: if the calling peer does not support elicitation, the helper at helpers.rs line 30 returns the supplied default and the tool keeps going. There is also a stored-peer pattern at helpers.rs line 90 that lets a separately connected peer (a UI app like mediar-app) field the question even when the calling peer (Claude Code) cannot. That is the workaround you have today: connect two peers to the same MCP server and let the elicitation-capable one carry the questions. If you only have Claude Code, the schemas are inert.

Hole 2: the on-screen inspector overlay is Windows-only, and macOS calls return success while doing nothing

The inspect overlay is the killer feature for grounding the LLM’s tree to the user’s monitor: pass show_overlay:"ui_tree" to get_window_tree and labelled rectangles get drawn over the live desktop in seven possible label modes. The implementation is Win32 layered windows with WS_EX_LAYERED + WS_EX_TRANSPARENT + WS_EX_TOPMOST.

The cfg gate at crates/terminator-mcp-agent/src/server.rs:1717 reads:

#[cfg(target_os = "windows")]
if let Some(ref overlay_type) = args.show_overlay {
    // ...build labelled rectangles, paint them with GDI...
}

The matching tool to clear the overlay is hide_inspect_overlay at server.rs:5769. Read what it does on macOS:

async fn hide_inspect_overlay(&self) -> Result<CallToolResult, McpError> {
    #[cfg(target_os = "windows")]
    {
        terminator::hide_inspect_overlay();
        info!("Signaled inspect overlay to close");
    }

    Ok(CallToolResult::success(vec![Content::json(json!({
        "action": "hide_inspect_overlay",
        "status": "executed_without_error",
        "message": "Inspect overlay hidden"
    }))?]))
}

On macOS the cfg block is skipped, the function falls through, and it returns status: "executed_without_error" with message: "Inspect overlay hidden". The user sees nothing because there was no overlay. Same shape on the show path: passing show_overlay:"ui_tree" returns the JSON tree fine, but the rectangles never appear on your monitor. The non-visual subset (validate_element, wait_for_element, click_element, type_into_element, invoke_element) is full parity. If your workflow assumes the rectangles, your workflow is Windows-only even when the rest of it is not.

Hole 3: parse_duration accepts “1.5s” and “500ms”, but not “1m30s”

Lots of MCP tools take a duration argument: delay, wait_for_element, retry budgets inside execute_sequence. The parser at crates/terminator-mcp-agent/src/duration_parser.rs lines 5 to 29 splits the input on the first alphabetic character and matches the unit against a fixed list (ms, s, sec, m, min, h, hr).

What works: "500" (treated as ms), "1500ms", "2.5s", "0.5m", "2h".

What does not work: "1m30s". The splitter pulls 1 as the number, takes m30s as the unit, fails the match, and bubbles Unknown time unit: m30s up the stack. If your workflow has a retry budget like “give it a minute and a half”, you have to write 90s or 90000ms by hand. Cosmetic, but it bites the first time an LLM writes the natural form and the typecheck pass does not catch it. The fix is a four-line change to the splitter, and the issue is open territory if you want to PR it.

Hole 4: surfaces the accessibility tree never sees

There are three classes of UI that no MCP-over-accessibility server can reach without a fallback. They are not Terminator-specific; they are the floor of the abstraction.

1. Game canvases. Unity, Unreal, web-canvas applets render to a single texture. The OS sees one Canvas role with no children. There is nothing in the tree to find.
2. Custom-rendered controls. Some Electron apps override the renderer in ways that drop UIA bindings. Some in-house Win32 apps never set up a UIA provider for the controls they draw with GDI. Some Qt builds expose a generic Pane and stop. The tree shows a container; the meaningful elements inside are absent.
3. Document interiors. A specific cell in an Excel sheet, a specific run of text in a Word document, a specific shape on a PowerPoint slide. These exist in the tree, but addressing them needs Text and Range patterns, not just role+name. The surface is real but the locator vocabulary is different.

Terminator’s answer is the multi-source fallback inside get_window_tree: include_ocr, include_omniparser, include_gemini_vision, include_browser_dom. They produce a clustered tree the agent can reason about even when the AX tree is silent. But the right framing is still: the answer is grounding fallbacks, not magic. If the surface is invisible to both the tree and a vision model on a screenshot, no MCP server gets you in.

Hole 5: this is a per-host process, not a hosted RPA platform

The MCP agent is a per-host binary. It binds to stdio when launched by Claude Code, or to localhost HTTP/SSE on the same machine. Workflow state checkpoints are local-only files at ~/Library/Application Support/mediar/workflows/<folder>/state.json on macOS or %LOCALAPPDATA%\mediar\workflows\<folder>\state.json on Windows. The survival path resumes on the same machine, against the same desktop.

What you do not get out of the box: a multi-tenant dispatcher, a queue, a centralised run history server, RBAC, audit logs, per-tenant credential vaults. If your purchase gate is hosted bot orchestration, this is the wrong shape. The product config calls this out as a disqualifier: “enterprise RPA buyer whose gate is a hosted platform with bot orchestration, RBAC, and audit logs”. Distributed execution is on you to build on top, the same way you would on top of a Playwright fleet.

Verify the holes yourself in five minutes

The whole verification path is five commands. Clone the repo, run grep, read the comments. Every claim above is a file plus a line you can open.

Where this page does not go

Three things that matter for production but already have their own treatment elsewhere.

• Workflow checkpointing (state.json after every step that mutates env). This is the survival path; without it, a 20-step workflow that fails at step 14 replays from step 1.
• The dev-tools loop exposed by the MCP server (inspect overlay, highlight, validate, wait, clear). The Chrome DevTools shape, but for the desktop accessibility tree.
• Control Patterns vs synthesised input (invoke fires the pattern in-process, click does SendInput). When to prefer which.