Run vLLM locally with a desktop agent

GEMINI_COMPUTER_USE_BACKEND_URL. It is read in crates/terminator-computer-use/src/lib.rs on line 145 and falls back to https://app.mediar.ai/api/vision/computer-use when unset. Set it to your shim's endpoint before constructing the Desktop object, and every screenshot in that session goes to your shim instead of Mediar's hosted route.

It is a real hand-off. The Rust side does not speak Gemini. It POSTs a fixed JSON payload and expects a fixed response shape. Any process on any URL that honours the contract can be the brain. The 'gemini' prefix in the env var is a historical naming choice, not a binding. You can run Qwen2.5-VL, Llama 3.2 Vision, InternVL, Pixtral, or any other VL model vLLM supports.

{"completed": bool, "function_call": {"name": string, "args": object} | null, "text": string | null, "safety_decision": "require_confirmation" | null}. If completed is true, the loop ends and text is treated as the final answer. If completed is false, function_call.name must be one of click_at, type_text_at, key_combination, scroll_document, scroll_at, drag_and_drop, wait_5_seconds, hover_at, navigate, or search. Anything else logs a warning and ends the loop.

Integers in a 0..999 grid that is normalized to the screenshot you were sent. Terminator divides by 1000 to get a pixel inside the (possibly resized) image, then multiplies back up by 1/resize_scale, 1/dpi_scale, and finally adds the captured window's top-left offset. The shim does none of this math. It just returns the normalized numbers the VL model emitted.

Line 679 of crates/terminator/src/computer_use/mod.rs runs `if previous_actions.len() > 3 { previous_actions.remove(0); }` at the end of every step. Each past action carries a full base64 PNG, and without the cap a 20-step plan would balloon the request body to 20 full screenshots. Three is enough context for action chaining and keeps the POST payload under typical proxy limits. Design your prompts knowing the model sees at most the current screen plus 3 prior screens.

You need a GPU for the VL model vLLM is serving, not for the shim itself. The shim is a FastAPI process that translates JSON. Qwen2.5-VL-7B fits comfortably on a single 24GB card in FP16, and vLLM's paged-attention kernels make it practical for one user driving one desktop session. Smaller models like Qwen2.5-VL-3B run on 12GB. The shim can be a CPU-only container talking to vLLM over localhost.

With GEMINI_COMPUTER_USE_BACKEND_URL pointed at localhost, nothing leaves the machine on the inference path. Terminator writes every screenshot to %LOCALAPPDATA%/terminator/executions/ as <timestamp>_geminiComputerUse_<process>_NNN.png. Your shim receives the same bytes over a loopback socket, forwards them to vLLM on another loopback socket, and returns JSON. The only wire traffic is to whatever target the agent is navigating to (e.g., a browser fetching a page).

Yes. Return safety_decision='require_confirmation' instead of a function_call and Terminator will break the loop with final_status='needs_confirmation', write the pending (action, args, text) into pending_confirmation, and return. Your outer harness can inspect the pending args, prompt the human, and resume. This is how you put 'send', 'delete', 'pay', or 'overwrite' behind an approval step without trusting the model.

Same shape, different backend. The Terminator example folder at crates/terminator-mcp-agent/examples/terminator-ai-summarizer uses the ollama-rs crate for a simpler summarize-the-screen loop. For the full computer-use agentic loop, you still need a shim that honours the JSON contract above because Ollama's /api/generate response shape is not the one Terminator's Rust side expects. If you prefer Ollama, replace the AsyncOpenAI client in the shim with an Ollama client and keep everything else.

The four coordinate transforms are in convert_normalized_to_screen. If the click lands above-and-left of the target, the resize_scale is being undercounted. If it lands in the right spot on your primary monitor but wrong on your secondary monitor, window_x and window_y are wrong (the capture happened on the wrong window). Terminator saves the initial screenshot at executions/<id>_000_initial.png and every post-action screenshot at executions/<id>_NNN_after.png, so you can confirm what the model actually saw versus what it tried to click.