Have a project in mind? Let's talk.
agent-desktop is Vercel Labs' newly open-sourced answer to a problem every agent builder hits eventually: how does a coding agent operate a real desktop application, not just a terminal or a browser? Released on GitHub under Apache-2.0 with version 1.0.2 on October 30, 2026, it takes a different path from screenshot-based computer use. Instead of sending pixels to a vision model, agent-desktop reads each app through the operating system's accessibility tree and hands the agent structured JSON with a stable reference for every element it can act on. This post walks through what the tool does, how its snapshot-and-ref loop works, what Vercel's self-reported numbers do and do not show, and the privacy and permission details worth knowing before an agent touches your desktop. Source
The README describes agent-desktop as "computer use for AI agents": a CLI and C library for macOS, Windows and Linux that reads and operates desktop apps through the OS accessibility tree. A snapshot command returns JSON with a ref for every element an agent can act on; the agent acts on a ref and snapshots again to check the result, with no pixel matching, and the loop stays in the agent. Two things in that description matter for how you evaluate it. First, agent-desktop is a tool that agents call, not an agent: it embeds no language model and has no GUI of its own, so any shell-capable coding agent can use it. Second, it is local-first: installation is a single npm package with native binaries for x64 and ARM64, it needs no GPU, and version 1.0.2 supports macOS 13 or later, Windows 10 1809 or Server 2019 or later, and Ubuntu 24.04 or later. Source
Every command prints one JSON envelope with an ok flag and either data or an error, then exits. A snapshot assigns qualified refs such as @s8f3k2p9:e8, where the snapshot ID inside the ref makes it self-describing: a bare @e8 is rejected with INVALID_ARGS. When the UI changes between actions, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent takes a fresh snapshot rather than guessing at coordinates. Every error carries a disposition field, and the documented contract is that an agent should retry only when the disposition says safe, so ref resolution fails closed instead of clicking the wrong thing. On macOS, CLI calls reuse a private command host per snapshot namespace that retains native accessibility objects, so a removed row cannot silently resolve to a surviving row with the same identifier; the host exits after 15 idle minutes. The tool ships 60 command names, 56 of them operational, covering observation, ref actions, keyboard and mouse, window management, notifications, clipboard, waiting, sessions and system status. The remaining four, key-down, key-up, mouse-down and mouse-up, refuse to run until a daemon exists that can release a held key or button, which is a deliberate safety gap rather than an oversight. Source
Accessibility trees get large fast on dense apps, so agent-desktop offers a --skeleton snapshot that reads only three levels deep, reports a children_count for cut containers with a drill ref, and lets snapshot --root read one region into the same snapshot. Vercel's reported figures for this mode are a 78 to 96 percent token reduction versus full-tree dumps in Slack, VS Code and Notion, with Apple Maps dropping from 3,811 tokens per look to 273, and a simulated 52-step task falling from 198,000 to 31,000 tokens. Treat these as what they are: self-reported numbers from the builder, not an independent computer-use benchmark, and we did not find an independent benchmark. What is verifiable from the repository is the mechanism, skeleton plus drill-down, and the design choice that makes token cost controllable at all: the agent reads text structure instead of images, so it pays for nodes, not pixels. Source
The platform table in the README is refreshingly honest about asymmetry. The accessibility tree, clicking, typing and clipboard work across all three operating systems: macOS through the AX API, Windows through UI Automation, and Linux through AT-SPI, with Wayland and X11 caveats listed per feature. macOS requires an Accessibility permission grant, which the CLI walks through with agent-desktop permissions --guide. Windows needs no grant for apps at the terminal's own integrity level. Linux needs the user's graphical session with at-spi2-core. The default posture is headless: ref actions go through the accessibility API without moving the pointer or taking focus, and a global --headed flag is required before any physical input is allowed. Headless calls still prohibit foreground activation and synthesized input. For automation that should never surprise the person sitting at the machine, headless-by-default is the right default, and it is the same class of scoping decision we covered in our guide to AI agent tool permissions. Source
The README carries a warning that deserves quoting in spirit: snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content, so keep them out of public logs and uploads. An agent that can read your screen can read whatever is on your screen, including the password manager you left unlocked and the message thread in the background. The macOS command host also requires TMPDIR to be owned by the current user with mode 0700, and an optional helper skill can send screen descriptions to a configured model provider, which is a separate data path from the native CLI and worth reviewing before enabling. If you run agents against a desktop with real accounts, the practical mitigations are the boring ones: run the agent in a dedicated user session, keep snapshots out of shared logs, and treat the optional model-provider helper as a data-sharing decision. Our earlier piece on the Microsoft execution containers sandbox covers the isolation side of this problem. Source
Everything above is what the sources say; this section is our read. agent-desktop arrives three days after DeepSeek's dsh-superagent prototype, and the two share a thesis: the next phase of agent tooling is not smarter models, it is better interfaces between agents and the real systems where work happens. The accessibility-tree approach trades the generality of screenshot-based computer use for determinism: refs either resolve or they fail closed, and that trade makes retry logic, auditing and testing far easier for engineering teams. The honest gaps are just as clear. We did not find an independent benchmark, the token figures are self-reported, four input commands are intentionally disabled, shell surfaces are only partially planned outside Windows, and a tool this capable hands the agent real power over whatever session it runs in. If your team evaluates it, start headless, start in a session you can afford to lose, and measure your own task completion rather than trusting anyone's numbers, including Vercel's. Source
agent-desktop is an open-source CLI and C library from Vercel Labs, released under Apache-2.0, that lets AI agents read and operate desktop applications on macOS, Windows and Linux through the OS accessibility tree. It returns JSON snapshots with a ref for every actionable element, and the agent acts on those refs instead of interpreting screenshots. Source
Yes. It is published on GitHub under the Apache-2.0 license and installs with npm install -g agent-desktop, with native binaries for x64 and ARM64. It runs on plain CPU with no GPU requirement, and the model you pair it with is your own choice and cost. Source
Screenshot-based approaches send images to a vision model, which then guesses coordinates. agent-desktop reads the operating system's accessibility tree and returns structured elements with qualified refs; actions on a stale or ambiguous ref fail with explicit errors instead of clicking the wrong place. The trade-off is determinism and token efficiency on one side, versus the broader app coverage of pixel-based methods on the other. Source
It is headless by default, so ref actions do not move your pointer or take focus, and physical input requires an explicit --headed flag. That said, snapshots and clipboard reads can contain private desktop content, macOS requires an Accessibility permission grant, and four held-key commands are disabled until a daemon can release them safely. Run it in a dedicated session and keep snapshots out of shared logs. Source
Any agent that can run shell commands. The binary carries its own agent-facing docs through agent-desktop skills get desktop and skills get platform, and Vercel publishes a skills.sh installer plus an AGENTS.md snippet that teaches the observe, act, verify loop. It embeds no model of its own, so it pairs with whatever coding agent you already run. Source
Ready to take the first step towards unlocking opportunities, realizing goals, and embracing innovation? We're here and eager to connect.