Vercel Labs’ agent-desktop reads desktop accessibility trees as JSON on macOS, Windows and Linux, letting agents click and type using element references. It offers a CLI, MCP and a C library without an embedded model. References belong to a snapshot and must be refreshed when the UI changes; accessibility actions avoid moving the pointer or taking focus by default. The separate agent-browser targets web pages. Automation depends on the app’s accessibility information and OS permissions, and successful actions still need follow-up observation.
Back to Tech
Vercel agent-desktop exposes desktop accessibility trees to AI agents
Element references replace screen coordinates, but OS permissions and post-action verification remain necessary.