typesafe-computer-use

Frontier vision models are the default way to do computer use, but they are slow and expensive: a screenshot goes to a model, the model reasons about it, and a plan comes back a few seconds later. Most of those decisions do not need reasoning; they need one choice from a short list of visible actions. typesafe-computer-use is built around that observation. Instead of asking a general-purpose vision model what to click every step, it reads the screen deterministically through OCR and the macOS accessibility tree, then asks a small purpose-built classifier which of the visible options is the next action, reserving a full language model for the rare step that genuinely needs free text or a final written answer.

The pipeline separates three roles. A capture layer merges OCR text and the accessibility tree into one list of clickable items per screen, re-scanning only the tiles that changed between steps to keep latency down. That list is handed to TypeSafe's jev model, a decision model that returns a full probability distribution and a calibrated confidence over up to 255 options in a few hundred milliseconds, at what the project describes as a fraction of a cent per decision. A separate writer model, Anthropic-compatible by default but swappable for any OpenAI-compatible endpoint including local ones such as LM Studio or Ollama, is only invoked to type free text, propose a URL, or answer once the classifier has stopped and the screen needs reading.

The natural user is a developer automating repetitive desktop workflows, browser purchases, form-filling, or app testing on a Mac, who wants an agent that acts fast and cheaply on routine steps and only pays for a frontier model's reasoning when a step genuinely calls for it. Because the writer model is fully swappable, teams already running a local or self-hosted LLM can keep expensive frontier calls out of the loop entirely except for TypeSafe's own classifier, which remains a required external dependency for every step. It suits people comfortable running a real mouse and keyboard against production apps, ideally starting with a dry run.

The project is one week old at the time of writing, tagged v0.2.0 and explicitly marked beta, so expect rapid, possibly breaking changes. The headline efficiency numbers, such as roughly 155 times cheaper and up to 40 times faster per decision than a bare-screenshot call to Claude Opus 5, are the project's own benchmark table, not an independently reproduced result, and they only hold because the classifier step is outsourced to TypeSafe's paid API rather than run locally. Windows support exists but is explicitly experimental and untested in CI. Anyone relying on this for unattended automation should also weigh that computer-use tools drive a real mouse and keyboard, with the attendant risk of a misclick or a prompt-injected page steering the session.

typesafe-computer-use is a genuinely engineered answer to the cost problem in computer-use agents, not a thin wrapper: a real capture pipeline, a documented test suite of dozens of scripted scenarios, and a swappable writer model make it more substantial than its one-week age suggests. It is worth trying if the TypeSafe dependency and its per-decision cost are acceptable to you, and if you can tolerate a fast-moving 0.x beta. It is too early if you need a fully self-contained, no-external-API open-source tool, if you are on Windows and need reliability, or if you need proven stability rather than a project still finding its shape a week after its first commit.