shimmy

Running large language models locally has a recurring tax: the toolchain. Most paths to a local OpenAI-style endpoint run through Python environments, llama.cpp builds, or a CUDA stack that ties you to NVIDIA hardware. shimmy attacks that friction directly. It is a pure-Rust inference engine that exposes an OpenAI-API-compatible endpoint and ships as a single binary, with no Python runtime and no llama.cpp dependency to compile.

The design choices follow from that goal. Because it targets Vulkan, D3D12 and Metal rather than CUDA alone, shimmy runs across a much wider range of GPUs, including AMD and Apple hardware that the CUDA-first ecosystem treats as second-class. It also auto-discovers models already on your machine from HuggingFace, Ollama and LM Studio, so you point it at what you have instead of re-downloading weights into yet another tool's directory layout. For anyone who just wants a drop-in `/v1/chat/completions` endpoint backed by local weights, that is a genuinely short path.

The natural fit is a developer or homelab user who wants to swap a hosted OpenAI call for a local one without rebuilding their stack: mixed-GPU desktops, machines without NVIDIA cards, or setups where a Python environment is unwelcome. Because the API is OpenAI-compatible, existing client code and SDKs keep working by changing only the base URL, which is the feature that makes local inference actually adoptable in an existing project.

The caveats are real and worth stating plainly. The Airframe GPU core cannot be built from source by the public, which is a meaningful asterisk on the word "open" for a project sold on openness. The license is inconsistent between the Apache-2.0 badges and a README that says MIT, something to resolve before you depend on it commercially. It serves one model per instance with no multi-model support, mixture-of-experts is not yet implemented, and the headline performance claims (such as sub-100ms startup versus Ollama) are unverified project figures rather than independently benchmarked results.

The verdict: shimmy is compelling if your priority is a frictionless OpenAI-compatible endpoint on non-CUDA or mixed GPU hardware, and you can live with single-model serving. Treat it as an early-but-promising alternative to Ollama and LM Studio for that specific niche. If you need to audit or compile the GPU core yourself, serve multiple models, or rely on guaranteed performance numbers, it is too early, and the more established local runtimes remain the safer choice for now.