Computer use agent state tracking, done with two regexes and a tree diff

It is the part of an agent loop that answers the question 'did my action actually change the UI, and how?' without taking another full screenshot or re-querying the whole accessibility tree by hand. For Terminator that answer is a structured diff returned alongside the click or type response, listing only the lines of the tree that changed after the volatile fields (IDs, bounds) were stripped. For a typical screenshot-only computer use loop it is a second model call against a pair of images.

Because the raw tree is full of fields that change every time you fetch it even when nothing visible changed. AutomationId values regenerate, runtime element_id values rotate, and bounding rectangles shift by a pixel when a parent reflows. If you diff the raw trees you get a wall of noise on every click. The interesting signal (a new dialog appeared, a button became enabled, a text field gained focus) drowns. Terminator's preprocess_tree and remove_ids_and_bounds_from_compact_yaml functions remove id, element_id, and bounds entries before the line diff runs, so the diff is over semantic structure only. A window that moved 118 pixels without re-rendering returns no diff at all. There is a test for exactly that case (test_simple_ui_tree_diff_yaml_bounds_change_no_diff at line 202 of ui_tree_diff.rs).

Two functions in crates/terminator/src/ui_tree_diff.rs do the work. remove_ids_and_bounds_from_compact_yaml uses a regex ` #[\w\-]+` (literal space, hash, then word characters or hyphens) to strip the inline id markers from compact YAML lines like `- [Button] Submit #id123 (bounds: [10,20,100,30], focusable)`. Then it uses `bounds: \[[^\]]+\],?\s*` to strip the bounds payload. preprocess_tree handles the JSON variant: it walks the tree and drops any key named id or element_id. The cleaned strings then go into similar::TextDiff::from_lines, the Rust equivalent of Python's difflib.ndiff, and only the lines tagged Insert or Delete come back. Equal lines are skipped. The whole module is 227 lines including tests.

If you are using the Rust crate directly, you call desktop.execute_with_ui_diff(selector, action, Some(UiDiffOptions { settle_delay_ms: Some(1500), .. })) and get back a tuple of (action_result, element, Option<UiDiffResult>). UiDiffResult has two fields: diff (a string of + and - lines) and has_changes (a boolean). If you are on the Node.js SDK, you pass `uiDiffBeforeAfter: true` to .click() or .typeText() options and the result object includes a uiDiff field with the same shape. If you are using the MCP server from Claude Code, Cursor, VS Code, or Windsurf, you set `ui_diff_before_after: true` on click_element, type_into_element, press_key, or any other action tool and the response JSON carries ui_diff and has_ui_changes. The system prompt the MCP server ships explicitly tells the assistant not to call get_window_tree after an action, because the diff is already attached to the action response.

A list of lines that changed after the strip pass. Lines starting with `-` were in the before tree and not in the after. Lines starting with `+` are new. For a Save dialog opening, you would see something like `+ - [Dialog] Save As (focusable)` followed by `+ - [Edit] File name` and `+ - [Button] Save`. Plus and minus signs are the only markers; equal lines are dropped on purpose to keep the diff short enough that an LLM can read it in one token budget. The diff is plain text, not JSON, because the model already speaks diff syntax fluently.

Because most native UI animations (Windows Fluent transitions, macOS view animations, web view fade-ins inside Electron apps) complete inside one second, and you need a margin. If you capture the AFTER tree too early you see the half-rendered intermediate state, which produces a diff full of partially-built dialog children that disappear on the next frame. 1500ms is the empirical floor that produces a clean diff for the common cases. The number is the unwrap_or default on opts.settle_delay_ms inside execute_with_ui_diff in crates/terminator/src/lib.rs around line 1814. Override it per call: shorter (200-500ms) for purely keyboard actions inside a single text field, longer (2500-5000ms) for actions that trigger a network round-trip the UI is waiting on.

Screenshot-based loops track state by comparing two raster images. The model has to do the comparison, which costs another full inference, and the comparison is pixel-aware (not semantically aware), so font rendering changes, subpixel jitter, and animation easing all register as differences. Terminator's loop tracks state structurally at the accessibility tree level. The diff is computed in Rust before the model ever sees it, so the per-action token cost goes to zero unless something semantically changed. Vision still has a place in the loop (Terminator's open source SDK ships a Gemini Computer Use adapter for cases where the tree is empty), but for the 90% case where the app exposes a real accessibility tree, you want structural state tracking, not visual.

Yes. It walks the tree twice (once before, once after) and computes a line diff between the two compact YAML strings. On a typical foreground window with 200-800 elements this costs in the tens of milliseconds per direction on Windows UIA, plus the settle delay. The tradeoff is that you save a model call per action. If your action tool would otherwise be followed by get_window_tree (the LLM's default move to figure out what happened), you have replaced one tree walk plus one full model turn with one tree walk and a small diff string. Cheaper, faster, and more honest about what changed.

You see `No UI changes detected` in the diff field and has_changes false. This is the failure mode the diff was built to surface. A click that resolved against the right element but did not change anything (because the button was already pressed, or the element is a dead area inside a misnamed container) shows up as `has_changes: false` instead of being silently reported as success. Pair this with the system prompt rule that says 'never hallucinate success' and the agent can decide whether to retry, re-acquire the tree, or escalate. The output of the action and the structural state delta are separated on purpose: one tells you the call returned ok, the other tells you the world moved.

Yes. On the Rust SDK, just call the non-diff variants of the action methods (element.click(), element.type_text()). On the Node SDK, leave uiDiffBeforeAfter false (it defaults to false at the binding layer to keep simple scripts fast). On the MCP server, the server prompt strongly encourages it for agent loops; in the changelog at version 0.24 the team made the parameter mandatory at the MCP layer specifically because agents that omitted it kept losing track of state. If you are running a deterministic recorded workflow where you already know the expected state, you can omit it and gain back the settle delay. If you are running an LLM-driven loop, leave it on.