oMLX

Running large language models locally on a Mac is technically possible thanks to Apple's MLX framework, but the day-to-day experience tends to be a collection of terminal commands, manual model swaps, and abandoned context that has to be recomputed on every request. oMLX, an Apache-2.0 project from developer jundot, is a macOS-native inference server built specifically for Apple Silicon that tries to make this practical. It ships a FastAPI backend with OpenAI and Anthropic compatible endpoints, a SwiftUI menubar app, and a browser admin dashboard. The goal stated in the README is mundane but real: pin everyday models in memory, auto-swap heavier ones on demand, set context limits, and keep the whole thing manageable without a Docker stack or a long-running command-line daemon.

Under the hood, oMLX leans on mlx-lm's BatchGenerator for continuous batching, with a first-come-first-served scheduler and a default cap of eight concurrent requests. The more distinctive piece is its tiered KV cache. Hot cache blocks stay in RAM, and when memory fills, blocks spill to SSD as safetensors files that can be restored on a matching prefix, even across server restarts. This is the project's own claim, and it aims to let context survive between requests instead of being thrown away. Multiple models can be served at once with LRU eviction, per-model pinning, idle TTLs, and a process-wide memory ceiling defaulting to system RAM minus eight gigabytes. A three-tier memory guard tries to prevent system-wide out-of-memory crashes.

The natural audience is Apple Silicon owners who want a managed local inference setup without living in a terminal. The dashboard handles model downloads from Hugging Face, real-time memory monitoring, benchmarking, a built-in chat, and one-click integration for tools like Claude Code and Copilot. A concrete use case the project emphasizes is agentic coding: when a tool-assisted session would normally overflow context and force a costly recompute, the SSD-backed cache is meant to keep prior context reusable. Beyond plain text models, oMLX supports vision-language models such as Pixtral and GLM-4V, OCR models including DeepSeek-OCR, plus embedding and reranker models, so it can act as a single local backend for several model types at once.

The limitations are clear-eyed. oMLX is Apple Silicon and macOS 15.0 only, with no Linux or Windows path, which rules it out for server deployments or multi-GPU clusters. The project reports a large open-issue backlog, on the order of several hundred issues, which signals rough edges in a fast-moving codebase that originated from a vllm-mlx fork. Its core differentiation from MLX-LM and llama.cpp is mostly the GUI and serving layer rather than the inference engine itself, so users comfortable on the command line gain less. The SSD tiering trades disk space and I/O for RAM, and context scaling is framed around smaller-context models with Claude Code, which hints that very large contexts on modest hardware remain constrained. Benchmark and performance claims should be verified on your own machine.

Against LM Studio, the obvious commercial reference point, oMLX offers a more server-oriented and scriptable approach with open-source transparency, Anthropic-compatible endpoints, and that SSD-tiered cache, while LM Studio remains the more polished, broadly supported, and battle-tested desktop option across platforms. oMLX is worth adopting if you are on a Mac, want a GUI-managed local server that drops into OpenAI or Anthropic clients, and value the persistent cache for agentic coding work. It is too early if you need Linux or Windows hosting, multi-GPU inference, or production-grade stability, given the issue backlog and the young codebase. For Apple Silicon tinkerers and local-first developers, it is a credible, actively developed option to keep an eye on rather than a finished product to standardize on.