Microsoft UI Automation has no spatial selectors. Terminator adds five.
IUIAutomation lets you walk a tree and filter by property. It does not let you ask for "the edit control to the right of the Username label". Terminator extends the surface with five geometry-aware operators that resolve an anchor through UIA and then filter visible candidates by bounding-rectangle math. The whole resolver is 82 lines of Rust in engine.rs.
What the SERP keeps missing
Search "microsoft ui automation" and the first page is the Microsoft Learn UI Automation Overview, the Win32 entry-uiauto-win32 reference, Wikipedia, a FlaUI tutorial, and a TestComplete marketing page. They all cover the same ground: UIA succeeded MSAA, it exposes IUIAutomation, IUIAutomationElement, a TreeWalker, property conditions, and control patterns. Every one of them stops at the COM surface.
None of them answers the question you actually hit five minutes into a real project: the accessible name I want is on a static Text element, but the thing I need to click is a different element that happens to sit next to it. UIA has BoundingRectangle on every node. It just does not have any primitive that says "elements to the right of X" or "elements near Y". You can build it from raw coordinates, and people have, but nobody on the first SERP page ships it as a locator.
This page is the resolver, in Rust, with line numbers.
The five operators, orbiting the anchor
Every spatial selector takes an inner selector (the anchor) and returns the candidates that satisfy the relation. RightOf, LeftOf, Above, and Below use overlap on the perpendicular axis. Near uses a Euclidean radius.
The resolver, watched step by step
Scroll into the frame below and the pipeline plays out: anchor resolution, candidate collection, geometric filter. This is exactly what happens inside a single locator() call when you use any of the five operators.
The raw UIA surface
“The spatial resolver is the only 82-line block that sits between a UIA FindFirst and a click. One anchor call, one broad candidate query, one bounding-box filter.”
engine.rs::find_elements, Selector::RightOf match arm
The anchor: engine.rs lines 1754 to 1836
This is the function arm. It is the entire implementation. Nothing hidden in another file, nothing deferred to a plugin. Paste the keyword into GitHub search on mediar-ai/terminator and you land here.
Three things to notice. Anchor resolution uses find_element, so the inner selector must be unambiguous. The candidate pool is a broad Selector::Visible(true) query at depth 100 with a 500 ms cap, which keeps the cost bounded even on a noisy desktop. And the Near branch is the only one that uses distance; everything else is an overlap check.
What the same query looks like against raw IUIAutomation
UIA gives you BoundingRectangle per element, but you have to bring your own filter. Tab through to see what a one-line Terminator selector expands to when you write it by hand in C++.
Same semantics, different amount of code
FindFirst the anchor. FindAll every descendant. Loop over the array, read every BoundingRectangle, compare by hand. You wrote the overlap test, you wrote the loop, you wrote the array release.
- No primitive for 'rightof' in the COM surface
- You own the overlap math
- One full tree walk per candidate collection
- Error handling and release calls are yours
What actually talks to what
Your code calls a locator. The selector parser turns the string into an AST with a RightOf / LeftOf / Above / Below / Near node. The engine resolves the anchor against IUIAutomation, collects candidates, and filters by geometry before returning the element list.
rightof:name:Username resolution
Numbers from the source
The 0 px Near radius is the only magic number in the resolver. Everything else is an overlap test, which is scale-invariant: whether you are on a 1080p laptop or a 5K display, rightof still means rightof.
Each operator, in one cell
Treat this as the reference card. Every bullet here maps to an arm of the match statement above.
rightof:
candidate_left >= anchor_right with vertical overlap. Grab the text box that sits next to the label, even when the tree order does not match the visual order.
leftof:
Mirror of rightof. candidate_right <= anchor_left with vertical overlap. Useful for prefix labels, icons, and row headers that render to the left of data.
above:
candidate_bottom <= anchor_top with horizontal overlap. Finds the heading or form label stacked on top of a control.
below:
candidate_top >= anchor_bottom with horizontal overlap. The click target for validation messages, helper text, or the next input down the form.
near:
Center-to-center Euclidean distance under NEAR_THRESHOLD (50.0 px). Matches elements that visibly belong to the same cluster as the anchor, regardless of axis.
anchor
The inner selector passed to any spatial operator. Resolved via find_element, so it must match a single element. If your anchor is ambiguous, narrow it with && before wrapping.
What the resolver does, in four steps
Inside a single spatial locator call
- 1
Parse selector
selector.rs tokenizes the string and builds a Selector AST. `rightof:name:Username` becomes Selector::RightOf(Box::new(Selector::Name("Username"))).
- 2
Resolve anchor
engine.rs calls find_element on the inner selector. One UIA FindFirst with the anchor's property condition, returning a single IUIAutomationElement.
- 3
Collect candidates
A broad Selector::Visible(true) query runs with depth 100 and a 500 ms budget. This is the only expensive part of the pipeline.
- 4
Filter by geometry
Each candidate's bounding rectangle is compared against the anchor's using the overlap rules for rightof/leftof/above/below, or the Euclidean distance for near.
The raw UIA version, for contrast
If you have written against IUIAutomation in C++ before, this will look familiar. It is what you end up with when you need "rightof" and the only primitive available is BoundingRectangle.
From the SDK, one line each
The TypeScript surface is the shortest. Same selector strings work from terminator-py and the MCP tool calls without translation.
What you see when you run it
Terminator versus direct IUIAutomation
| Feature | Direct IUIAutomation | Terminator spatial selector |
|---|---|---|
| find element by screen geometry relative to an anchor | write manual BoundingRectangle math per element | rightof:name:Username resolves in one locator call |
| compose spatial and property filters | one PropertyCondition at a time | role:Edit && rightof:name:Username && !visible:false |
| overlap-aware left / right / above / below | raw x coordinate comparison | requires vertical overlap for rightof/leftof, horizontal overlap for above/below |
| Euclidean near selector | not exposed by IUIAutomation | NEAR_THRESHOLD = 50.0 px, center-to-center |
| available from TypeScript, Python, Rust, MCP | C++ COM, C# via UIAutomationClient | @mediar-ai/terminator, terminator-py, terminator-rs, terminator-mcp-agent |
| MIT-licensed source you can audit | Win32 COM surface, no implementation visibility | engine.rs lines 1754 to 1836, on GitHub |
A checklist before you ship a spatial selector
Five things to double-check
- Use the same selector everywhere. `rightof:name:Username` works from Rust, Node, Python, and the MCP tool calls.
- Narrow the anchor with &&. A bare name: often matches more than one element; spatial selectors need a unique anchor.
- Prefer rightof/leftof/above/below over near when you actually mean 'in the same row/column'. Overlap beats radius.
- Pair spatial selectors with role: or classname:. `role:Edit && rightof:name:Username` is faster and sharper than rightof alone.
- Remember the 500 ms candidate timeout. If the app is still loading, spatial selectors will return empty; wrap them in locator.first(timeout) to retry.
Why this matters for AI coding agents
An agent driving a UIA desktop cannot always see a clean accessible name on the thing it needs to click. Buttons with icon-only labels, inputs that inherit their semantics from a sibling label, cells in custom grids, all of these are unreachable through role and name alone. Spatial selectors close that gap: the agent anchors on the closest labeled element and reaches over with rightof:, below:, or near:. No vision model, no screenshot, no heuristic. Just UIA plus 82 lines of Rust.
One install: claude mcp add terminator "npx -y terminator-mcp-agent@latest"
The wider Microsoft UI Automation ecosystem
Spatial selectors are one slice. The rest of the UIA ecosystem is the usual cast of inspectors, language wrappers, and RPA suites. Every name below consumes the same IUIAutomation surface; the differences are API shape, license, and how much work they do for you above the COM layer.
Hitting the wall where UIA roles are not enough?
Book 20 minutes. We will walk through anchor selection, the overlap rules, and how to wire a spatial locator into your stack.
Frequently asked questions
What is Microsoft UI Automation?
Microsoft UI Automation (UIA) is the COM-based accessibility framework that ships with Windows. It is the successor to MSAA and exposes the entire desktop as a tree of IUIAutomationElement nodes with typed control patterns (Invoke, Value, Toggle, ExpandCollapse, Window, Selection, Scroll, RangeValue). You navigate the tree with IUIAutomationTreeWalker, filter by property with IUIAutomationPropertyCondition, and read fields like Name, ControlType, BoundingRectangle, and AutomationId from every node. Screen readers like Narrator, inspection tools like Inspect.exe, and automation frameworks like FlaUI, WinAppDriver, and Terminator all drive this surface.
Why does Microsoft UI Automation not have a spatial selector?
UIA is an accessibility API, not a layout API. The assumption is that assistive technology walks a logical tree and reads semantic roles, so the surface exposes parent, child, and sibling relationships plus a flat BoundingRectangle property per element. There is no built-in IUIAutomation method like FindRightOf or FindNear. If you want to say 'the edit control to the right of the Username label' you have to read every candidate's BoundingRectangle and do the math yourself. Terminator does that math for you as a first-class selector operator.
How does Terminator implement rightof and the other spatial selectors?
The implementation lives in crates/terminator/src/platforms/windows/engine.rs between lines 1754 and 1836. Step one: resolve the anchor selector through the normal UIA path so you know the anchor's (x, y, width, height). Step two: collect candidate elements via a broad Selector::Visible(true) query with depth 100 and a 500 ms timeout. Step three: filter candidates by bounding-rectangle math. For RightOf, a candidate qualifies when candidate_left >= anchor_right and the two rectangles share vertical overlap (candidate_top < anchor_bottom && candidate_bottom > anchor_top). LeftOf mirrors that. Above and Below use horizontal overlap instead. Near computes Euclidean distance between rectangle centers and accepts anything under NEAR_THRESHOLD, which is 50.0 pixels.
What is the exact NEAR_THRESHOLD value?
50.0 pixels, defined inline on line 1815 of engine.rs as `const NEAR_THRESHOLD: f64 = 50.0;`. The distance is center-to-center Euclidean: `(dx*dx + dy*dy).sqrt()`. That is a small radius chosen so near: matches only elements that visibly belong to the same cluster (a label and its input, a row header and its first cell). For wider hits use below: or rightof: where the bound is an overlap test, not a distance.
Why filter by vertical or horizontal overlap instead of pure left/right?
If you only tested candidate_left >= anchor_right for RightOf, you would match every element anywhere on the screen that happens to sit to the right of the anchor's x coordinate, including things on different rows. Requiring that the candidate and anchor rectangles overlap vertically scopes RightOf to items that are in the same horizontal band as the anchor, which matches what a human means by 'to the right of'. Above and Below flip the axis and require horizontal overlap for the same reason.
Can I chain spatial selectors with boolean operators?
Yes. The selector grammar in crates/terminator/src/selector.rs (753 lines) tokenizes boolean operators &&, ||, !, parentheses, and the comma shorthand, and the Shunting-Yard parser merges them with atomic prefixes like role:, name:, id:, classname:, visible:, process:, rightof:, leftof:, above:, below:, near:, nth:, and has:. So `role:Edit && rightof:name:Username` compiles to a Chain([And([Role(Edit, None), RightOf(Name(Username))])]) predicate. The inner selector of a spatial operator can itself be any expression, which is why the engine recurses into inner selectors for process scoping, visibility, and anchor resolution.
How is this different from Inspect.exe, FlaUI, or Microsoft's own automation samples?
Inspect.exe is a viewer. It shows you the UIA tree and properties but does not let you script actions. FlaUI is the canonical .NET UIA wrapper; it gives you a FindAll with conditions and a bounding rectangle per element, but no spatial operator. Microsoft's own samples under windows-classic-samples demonstrate TreeWalker and PropertyCondition but never layout-relative queries. Terminator is the only open-source framework I know that bakes spatial selectors into the locator grammar and ships the resolver across Rust, Node.js, Python, and an MCP server.
What happens if the anchor resolves to more than one element?
find_element is used to resolve the anchor, not find_elements, so the API contract is that the anchor must be unique. If your inner selector matches multiple candidates, resolution fails with an ambiguity error and the spatial selector never runs. In practice you narrow the anchor with the same grammar you use everywhere else: `rightof:(role:Text && name:Username)` instead of `rightof:name:Username` if the bare name matches more than one label.
Does this work for WPF, WinUI 3, UWP, and Win32?
Yes, because every surface that UIA exposes has a BoundingRectangle. Terminator's selector engine does not care whether a given HWND is a Win32 button, a WPF Button, a UWP AppBarButton, or a WinUI 3 ContentDialog primary button; it only reads what UIA returns. For Win32 controls that pre-date UIA, the IAccessible bridge fills in the bounds and the same RightOf / Above math applies. The only surface where spatial selectors do not apply cleanly is off-screen virtualized items, because their bounds come back as zeros until they scroll into view.
Where is the source and what is the license?
Terminator is MIT licensed. The spatial selector resolver is in crates/terminator/src/platforms/windows/engine.rs between lines 1754 and 1836. The selector AST (the Rust enum with RightOf, LeftOf, Above, Below, Near variants) is at the top of crates/terminator/src/selector.rs. The documented selector grammar is in docs/SELECTORS_CHEATSHEET.md. Install from npm as @mediar-ai/terminator, from pip as terminator-py, or run the MCP server with `npx -y terminator-mcp-agent@latest`.
Other Terminator pieces on the same surface
Keep reading
Microsoft UI Automation tool: cached subtree builder
The companion piece. How tree_builder.rs pre-fetches seven UIProperty fields with one IUIAutomationCacheRequest and TreeScope::Subtree.
Coded UI automation, reshaped around a selector chain
Microsoft deprecated Coded UI Test. Terminator brings a Playwright-style >> chain that reuses the same AutomationId CUIT addressed.
UI automation testing without a dedicated test framework
Locator timeouts, validate(), and waitFor() compose into a deterministic workflow without dragging in Mocha or Pytest.