vercel-labs/agent-desktop
Gives AI agents computer control via OS accessibility trees instead of guessing from pixels.
Lets an agent read and operate any desktop app through the OS accessibility tree rather than matching screenshots: a snapshot command returns JSON with a stable ref for every clickable element, the agent acts on a ref, then snapshots again to confirm. Refs are scoped to one snapshot — a stale or ambiguous one returns a clear STALE_REF or AMBIGUOUS_TARGET error instead of silently clicking the wrong thing. Written in Rust, it ships as a CLI and C library for macOS, Windows and Linux, installable via npm, with accessibility permissions required on macOS and at-spi2-core on Linux.
What holds up
- +Reads real UI structure via accessibility trees, not pixel matching — refs stay stable
- +Stale or ambiguous refs fail with a specific error code instead of silently misclicking
- +Works across macOS, Windows and Linux from one CLI and C library
- +Headless calls block foreground activation and synthesized input, by design
Mind the limits
- −Official Homebrew and Scoop packages aren't available yet — only npm or raw archives for now
- −Snapshots can contain private desktop content, so logs and uploads need care
Picked here? Add the badge
Put it on your site. It links back to this page. Free for every listed product.
<a href="https://stillworks.dev/products/p/vercel-labs-agent-desktop/"><img src="https://stillworks.dev/badge/vercel-labs-agent-desktop.svg" alt="Picked by StillWorks" width="230" height="54"></a>