computer-use-linux
agent-sh
An MCP server that drives a real Linux desktop via accessibility APIs and input injection.
What is computer-use-linux?
A Rust MCP server that gives an agent control of a real Linux desktop: it reads the AT-SPI accessibility tree, captures screenshots, and injects clicks, scrolls and keystrokes through GNOME Shell, Wayland portals and ydotool. It targets GNOME, KDE/KWin, Hyprland, i3 and COSMIC on both Wayland and X11, the gap most computer-use tooling leaves open by assuming macOS or X11.
| Fact | Value |
|---|---|
| Maintainer | agent-sh |
| GitHub stars | 260 (as of 2026-07-11) |
| Forks | 30 |
| License | MIT |
| License type | OSI-open |
| Category | Computer-use and autonomous agents |
| Status | Rising |
| Edition | 2026-07 |
| Last verified | 2026-07-11 |
Pros & Cons
Pros
- Fills a real gap: native Linux and Wayland desktop control, not just X11
- Uses the accessibility tree, not only screenshot-and-guess, so actions can target real UI elements
- MIT-licensed Rust that works across several compositors
Cons
- Setup depends on system pieces (AT-SPI, ydotool, portals) that vary by distro and compositor
- Young at v0.3.x with a single maintainer
- Accessibility-tree coverage depends on apps exposing AT-SPI, which many do poorly
License
MIT (OSI-open)
When it is interesting
You want an agent to operate GUI apps on a real Linux or Wayland desktop through MCP.
When it is too early
Your target apps expose little accessibility metadata, or you need a hardened, cross-distro-tested setup.
This repo featured in the 2026-07 edition of the Open-Source AI Radar.
UI-TARS-desktop
bytedance
Native desktop app for a GUI/computer-use agent powered by the open-weight UI-TARS model.
strix
usestrix
Framework of autonomous AI hacker agents for dynamic application security testing.
Page Agent
alibaba
In-page JavaScript GUI agent - control any webpage with natural language, no headless browser or extension.