Local AI on native apps: a 1B model drives the desktop because the accessibility tree is text, not pixels

The three steps that let a tiny local model read your screen

The pipeline is short on purpose. Capture, filter, run. Each step has one job, and the filter step is the one most descriptions of accessibility-tree agents skip past, even though it is the one that makes the local-model angle actually work.

Step one: capture the focused window's tree

On hotkey press, the example calls two MCP tools. First get_applications_and_windows_list to find the entry whose is_focused is true and grab its PID. Second get_window_tree with that PID, which returns the full accessibility subtree for that process as JSON. On Windows that is the result of walking IUIAutomationElement; on macOS the result of walking AXUIElement; on Linux AT-SPI2. From the model's point of view the shape is the same in all three cases: a tree of objects with role, name, and a selector path you can navigate back to.

The actual MCP call sits inside get_mcp_tool_result in client.rs. The example spawns the terminator-mcp-agent binary as a child process over stdio, calls the tool, and parses the response. The MCP boundary is what keeps the AX traversal code, the selector engine, and the Win32 / AppKit FFI calls out of your local-AI script. Your script only sees JSON.

Step two: drop everything except the tree

The MCP response includes more than the model needs. It carries request metadata, timing fields, and a flat element list alongside the nested tree. Sending all of it to a 1B model burns context on attributes the model cannot use. The filter is seventeen lines in client.rs, and it is the most load-bearing seventeen lines in the example.

// crates/terminator-mcp-agent/examples/terminator-ai-summarizer/src/client.rs
// Lines 47 to 63: the filter that makes a 1B model viable.

if let Some(first_content) = result.content.first() {
    match &first_content.raw {
        rmcp::model::RawContent::Text(raw_text_content) => {
            let parsed_json: serde_json::Value =
                serde_json::from_str(&raw_text_content.text)?;

            let ui_tree = parsed_json
                .get("ui_tree")
                .cloned()
                .ok_or_else(|| anyhow!("missing ui_tree"))?;
            let focused_window = parsed_json
                .get("focused_window")
                .cloned()
                .ok_or_else(|| anyhow!("missing focused_window"))?;

            let filtered_result = json!({
                "ui_tree": ui_tree,
                "focused_window": focused_window
            });

            Ok(filtered_result)
        }
        _ => Err(anyhow!("expected text content in CallToolResult")),
    }
}

Two keys reach the model: ui_tree (the nested element graph) and focused_window (PID, app name, title). On a real Outlook compose window the filtered payload measured around one third of the unfiltered response in token count. That difference is what turns a barely-fits 1B context into a comfortable 1B context with room for a system prompt and the model's answer.

Step three: hand it to a local model

The Ollama integration is twenty one lines. The whole file:

// crates/terminator-mcp-agent/examples/terminator-ai-summarizer/src/ollama.rs
// All 21 lines. localhost:11434 is implicit in Ollama::default().

use anyhow::Result;
use ollama_rs::{generation::completion::request::GenerationRequest, Ollama};
use serde_json::Value;

pub async fn summrize_by_ollama(
    model: &str,
    system_prompt: &str,
    mcp_result: &Value,
) -> Result<String> {
    let ollama = Ollama::default();
    tracing::info!("sending context to ollama model: {}", model);

    let prompt = format!("{system_prompt}\n Screen ui element tree: {mcp_result}");

    let request = GenerationRequest::new(model.to_string(), prompt);
    let response = ollama.generate(request).await?;

    tracing::info!("successfully received response from Ollama");

    Ok(response.response)
}

Ollama::default() points at http://127.0.0.1:11434 implicitly, which is the only network call the entire pipeline makes. The prompt template is one line: {system_prompt}\n Screen ui element tree: {mcp_result}. The default system prompt (in utils.rs line 14) tells the model it is a screen summarizer reading a UI tree in JSON and to use the name and text attributes. That is the entire contract.

// utils.rs lines 8 to 23: the CLI defaults that tell you what's intended out of the box.

#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
pub struct Args {
    #[arg(short, long, default_value = "you're are screen summarizer assitant ...")]
    pub system_prompt: String,
    #[arg(short, long, default_value = "gemma3:1b")]   // <-- 815 MB on disk
    pub model: String,
    #[arg(short, long, default_value = "ctrl+alt+j")]  // <-- the trigger
    pub hotkey: String,
    #[arg(short, long, action = clap::ArgAction::SetTrue)]
    pub ai_mode: bool,
}

What the data flow actually looks like end to end

One hotkey press, one MCP child process, one localhost call to Ollama, one clipboard write. No external network, no model running on a remote GPU.

Why this only works because the input is structured text

The entire point of the page is in this comparison. Vision models work on the desktop. Vision models do not work for tiny local models on the desktop, because the perception cost scales with image area and the regression task scales with model capability. A 1B model has neither. Replace the perception step with a structured-text input that the OS already produces for accessibility tools, and the same 1B model is suddenly useful.

How big does the model actually need to be?

Three rough thresholds, based on what shows up in actual issues and chats from users running tree-driven local agents.

Numbers move with model family. Gemma 3 at 1B holds up better than Llama 3.2 at 1B for this input shape because the gemma3 tokenizer treats role / name / value JSON keys efficiently. Qwen 2.5 at 3B is roughly equivalent to Llama 3.2 at 3B for tool-call output. Once you cross 7B the model is no longer the bottleneck on a single-window task; the AX traversal time on a busy Office app is.

Run it in three commands

What it looks like to install Ollama, pull the default model, install the example, and trigger it on a window.

On macOS the first run will prompt for Accessibility permission for whichever process is hosting the rdev hotkey listener. On Windows the binary works out of the box. On Linux you need AT-SPI2 enabled in your desktop environment (most modern GNOME and KDE sessions already have it on).

When the accessibility tree is not the right input

The tree wins on every native app that exposes one. The honest list of cases where it does not, and where you have to stack a vision model on top, is short and worth memorizing before you build an agent on this stack.

Frequently asked questions