Visual Sonar: Inspectable Automation for Remote Desktop Forms

A remote desktop can expose only a stream of pixels to the local machine. Selenium cannot inspect the application’s DOM through that video stream. Visual Sonar explores a different approach: move focus with TAB, compare screenshots, and use the changed region as a candidate field location.

The focus-difference loop

capture before -> press TAB -> wait for rendering -> capture after
              -> compare changed pixels -> review candidate field

The largest changed region is a heuristic, not proof of the intended field. Cursor movement, animations, compression artifacts, and delayed remote rendering can dominate the difference. A useful mapper needs a controlled window and reviewable results.

A minimal image-difference example

This standalone teaching example compares two screenshots. It does not capture the desktop or send input. It returns no region when there is no detectable change.

import cv2
import numpy as np

def changed_region(before, after):
    a = np.asarray(before)
    b = np.asarray(after)
    if a.shape != b.shape:
        raise ValueError('Screenshots must have the same dimensions')
    gray_a = cv2.cvtColor(a, cv2.COLOR_RGB2GRAY)
    gray_b = cv2.cvtColor(b, cv2.COLOR_RGB2GRAY)
    diff = cv2.absdiff(gray_a, gray_b)
    threshold = min(255, int(np.median(diff)) + 15)
    _, mask = cv2.threshold(diff, threshold, 255, cv2.THRESH_BINARY)
    contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL,
                                   cv2.CHAIN_APPROX_SIMPLE)
    if not contours:
        return None
    return cv2.boundingRect(max(contours, key=cv2.contourArea))

Try the simulator before a remote session

git clone https://github.com/godhiraj-code/wvdautomation.git
cd wvdautomation
python -m pip install -e .
python wvd_simulator.py

In a second terminal, use visual-sonar map to inspect and label the form, then visual-sonar run with synthetic input. The repository also documents CSV/JSON batches and optional OCR extraction.

What a reliable mapping requires

• A visible, unlocked desktop session and the correct foreground window.
• Stable focus order and visible focus indicators.
• Compatible display scaling, resolution, and window geometry.
• Bounded settling waits and a maximum number of TAB steps.
• A reviewed field map and independent verification of the final state.

Suppressing OpenCV preview windows does not make desktop input possible on a machine with no interactive desktop. Likewise, three TAB presses without visible change do not prove that a form has ended. Treat these observations as limits to investigate, not universal rules.

Privacy includes screenshots and the clipboard

Input masking is not a guarantee that secrets never appear elsewhere. Screenshots, OCR, clipboard content, remote-session logs, and saved input files can expose data. Use synthetic fixtures while developing and minimize retained captures.

Choose the approach around the application

Prefer a supported API or native accessibility interface when available. Pixel-based interaction can be useful when those paths are unavailable, but it needs more environmental control and review. Compare maintenance and verification effort before treating it as a replacement for a larger automation platform.

Visual Sonar repository and user guide