computer-use-linux
agent-sh
An MCP server that drives a real Linux desktop via accessibility APIs and input injection.
What is computer-use-linux?
A Rust MCP server that gives an agent control of a real Linux desktop: it reads the AT-SPI accessibility tree, captures screenshots, and injects clicks, scrolls and keystrokes through GNOME Shell, Wayland portals and ydotool. It targets GNOME, KDE/KWin, Hyprland, i3 and COSMIC on both Wayland and X11, the gap most computer-use tooling leaves open by assuming macOS or X11.
Pros & Cons
Pros
- Fills a real gap: native Linux and Wayland desktop control, not just X11
- Uses the accessibility tree, not only screenshot-and-guess, so actions can target real UI elements
- MIT-licensed Rust that works across several compositors
Cons
- Setup depends on system pieces (AT-SPI, ydotool, portals) that vary by distro and compositor
- Young at v0.3.x with a single maintainer
- Accessibility-tree coverage depends on apps exposing AT-SPI, which many do poorly
License
MIT (OSI-open)
When it is interesting
You want an agent to operate GUI apps on a real Linux or Wayland desktop through MCP.
When it is too early
Your target apps expose little accessibility metadata, or you need a hardened, cross-distro-tested setup.
This repo featured in the 2026-07 edition of the Open-Source AI Radar.
UI-TARS-desktop
bytedance
Native desktop app for a GUI/computer-use agent powered by the open-weight UI-TARS model.
strix
usestrix
Framework of autonomous AI hacker agents for dynamic application security testing.
Page Agent
alibaba
In-page JavaScript GUI agent - control any webpage with natural language, no headless browser or extension.