# Vercel's agent-desktop Gives Agents Computer Use

> Vercel Labs released agent-desktop, open-source computer use for AI agents through the OS accessibility tree. How it works, what it saves, what to watch.

- Author: Nitish Garg, Founder & CEO, CellCog
- Published: 2026-10-31
- Canonical (HTML): https://cellcog.ai/blog/vercel-agent-desktop/
- Section: Guides / Workflows & use cases
- Publisher: CellCog (https://cellcog.ai), the AI employee platform. Blog index for agents: https://cellcog.ai/blog/llms.txt

## Key points

- Vercel Labs released agent-desktop, open-source computer use for AI agents on macOS, Windows and Linux, announced on X on October 30, 2026.
- It reads apps through the operating system's accessibility tree and hands the agent a ref for every element, so the agent clicks named elements instead of pixels.
- Its builder says skeleton drilling cuts Apple Maps to 273 tokens a look from 3,811, and a simulated 52-step task to 31k tokens from 198k.
- It runs headless by default: actions do not move the pointer or take focus, so agents can work while a person uses the machine.
- Snapshots can contain private desktop content; the README says to keep them out of public logs.

## At a glance

- **What is agent-desktop?** An open-source Vercel Labs CLI that gives AI agents computer use on macOS, Windows and Linux by reading apps through the accessibility tree.
- **How do I install it?** Run npm i -g agent-desktop. It is Apache-2.0 licensed and ships native binaries for x64 and ARM64.
- **What does it save?** Its builder reports 273 tokens per look instead of 3,811 in Apple Maps by snapshotting one region, not the whole window.

Vercel Labs has released agent-desktop, an open-source tool that lets any AI agent see and operate desktop apps on macOS, Windows and Linux. It reads each app through the operating system's accessibility tree instead of screenshots, so an agent clicks a named element rather than guessing at pixels. It installs with one command, `npm i -g agent-desktop`, under the Apache-2.0 license.

The launch went out on X late on October 30, 2026 (US time). Lahfir, who lists himself on Vercel Labs and as the builder, [introduced it](https://x.com/mdlahfir/status/2116316228046500348) at 23:48 UTC as "Computer use for your agents". A [second post](https://x.com/ctatedev/status/2116323148706771302) pitched it as "agent-browser, for your whole computer" and passed 200,000 views; Vercel CEO Guillermo Rauch [shared it](https://x.com/rauchg/status/2116345666108854781) with "An agent on every desktop".

## What agent-desktop is

*Table: agent-desktop at launch (GitHub and npm, October 31, 2026)*

| Item | Detail |
|---|---|
| Maker | Vercel Labs |
| What it does | Computer use for AI agents through the OS accessibility tree |
| Platforms | macOS, Windows, Linux (x64 and ARM64) |
| Install | npm i -g agent-desktop |
| Built in | Rust, shipped as a CLI and a C library |
| License | Apache-2.0 |
| Version | 1.0.2 (published October 30, 2026) |
| Repository | vercel-labs/agent-desktop, created February 2026 |

The [repository](https://github.com/vercel-labs/agent-desktop) is older than the launch: it was created in February 2026, and version 1.0.0 reached [npm](https://www.npmjs.com/package/agent-desktop) on October 28. The October 30 posts announced it to a wider audience.

## How it works

![Animated diagram: two copies of an app window; on the left a scan lights up all 12 regions for 3,811 tokens a look; on the right only the outline is read, then one region is drilled, for 273 tokens a look; a band reads Simulated 52-step task: 198k vs 31k tokens](https://cellcog.ai/blog/media/vercel-agent-desktop/drill.svg)
*Reading one region instead of the whole window: 273 tokens a look, not 3,811.*

An agent asks for a snapshot of an app, and agent-desktop returns JSON with a ref for every element the agent can act on. The README describes the loop plainly: "The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent."

Dense apps are where the tool tries to save money. The agent first maps a window with a skeleton snapshot, then drills into one region and works only there. Lahfir's numbers for Apple Maps:

*Table: Skeleton drilling vs reading the full window (Lahfir's post, October 31, 2026)*

| Measure | Full window | Skeleton, then drill |
|---|---|---|
| Tokens per look | 3,811 | 273 |
| Tokens over a simulated 52-step task | 198k | 31k |

In his words: "This saves tokens and the operation is faster!" These are the builder's own figures from one simulated task, not an independent benchmark.

## Refs, not pixels

![Animated diagram: three boxes in a loop, 1 Snapshot with a ref, 2 Act on the ref, 3 Snapshot again with a check; a curved arrow back carries the tag STALE_REF: snapshot again; below, a laptop whose cursor stays put](https://cellcog.ai/blog/media/vercel-agent-desktop/refs.svg)
*Snapshot, act on a ref, snapshot again; the cursor never moves.*

Every ref belongs to the snapshot that produced it. When the interface changes, an action returns `STALE_REF` or `AMBIGUOUS_TARGET`, and the agent takes a new snapshot before trying again. Errors carry a disposition that says whether a retry is safe.

The tool runs headless by default. Per the README, ref actions "go through the accessibility API without moving the pointer or taking focus", which is why the launch posts say agents can work in the background while a person keeps using the machine, and why several agents can work on one app at once.

On macOS it needs Accessibility permission. On Linux it needs a graphical session with `at-spi2-core`. Windows needs no extra grant for apps at the terminal's own integrity level.

## What to watch for

The README is direct about privacy: "Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content." An agent that can read every app can also read whatever those apps show. Keep its output out of public logs, and give an agent a machine or an account scoped to the work.

There is also no shared yardstick yet. One developer replying to the launch asked: "Is there an actually reliable computer use benchmark?" Accessibility-tree tools depend on how well each app exposes its interface, so results will vary by app.

## Where it fits

agent-desktop is a building block: it gives a coding agent hands on a desktop, and the agent supplies the judgment. Vercel ships a matching skill on [skills.sh](https://skills.sh/vercel-labs/agent-desktop/agent-desktop) so harnesses can learn the snapshot, act and verify loop.

CellCog's AI employees already work on a computer of their own, with their own secure VM, file system and browser identity, and with Cowork they can run commands on a person's own machine with approval for anything risky. An AI software engineer can write and run end-to-end tests for a desktop app while your team keeps the review.

## FAQ

**What is Vercel agent-desktop?**

An open-source tool from Vercel Labs that lets AI agents see and operate desktop apps on macOS, Windows and Linux through the operating system's accessibility tree.

**Does agent-desktop use screenshots?**

No. It returns a JSON snapshot with a ref for each element; the agent acts on a ref and snapshots again to check, with no pixel matching.

**Does agent-desktop take over my mouse?**

Not by default. Ref actions go through the accessibility API without moving the pointer or taking focus; a headed flag allows physical input.

**How many tokens does agent-desktop save?**

Its builder reports 273 tokens a look instead of 3,811 in Apple Maps, and 31k instead of 198k over a simulated 52-step task. There is no independent benchmark yet.

**Is agent-desktop free?**

Yes. It is open source under the Apache-2.0 license and installs from npm.

## Related

- [ChatGPT in Chrome: Tabs, Computer Use, What's New](https://cellcog.ai/blog/chatgpt-chrome-computer-use/index.md)
- [Claude Code Extension Backdoored on Open VSX and Cursor](https://cellcog.ai/blog/claude-code-extension-backdoor/index.md)
- [Claude Code Now Declines to Recreate Existing Games](https://cellcog.ai/blog/claude-code-game-recreation-rule/index.md)
- [What Is an AI Employee? The 5-Part Test for a Standing AI Worker](https://cellcog.ai/blog/what-is-an-ai-employee/index.md)

## The AI employee for this read

[AI Software Engineer](https://cellcog.ai/ai-employees/ai-software-engineer): I built this page. For what it covers, hire an engineer: it works in your repo behind an approval gate, so nothing reaches your world unclassified.

---

Markdown alternate of https://cellcog.ai/blog/vercel-agent-desktop/. Try CellCog free, no credit card needed: https://cellcog.ai/signup