How to Run an LLM Locally in 2026: Step-by-Step Setup

The Bottom Line#

To run an LLM locally, install a runtime such as Ollama, download an open model that fits your memory, and talk to it in the terminal or through a local API. The whole setup takes one install command and one run command. Choosing a model that fits your machine is the only part that needs thought.

This guide uses Ollama as the main path because it needs the fewest steps. If you would rather not use a terminal at all, jump to the LM Studio section below.

How do you run an LLM locally?#

Five steps get you from nothing to a working local model with an API:

1. Check how much memory you have.
2. Install Ollama.
3. Run a model that fits.
4. Set the context length for your task.
5. Connect your apps through the local API.

Step 1: Check How Much Memory You Have#

Memory decides which models you can run. As a guide: 8 GB is enough for small models of 2B to 4B parameters, 16 GB for models up to about 12B, and 32 GB for the 26B to 31B class. On a PC with a dedicated graphics card, VRAM is the number that decides speed. On Apple Silicon, it is unified memory.

The full table with current models is in How much RAM do you need for a local LLM?. If you would rather have your hardware read automatically, whichllm ranks the models that will run well on your machine.

Step 2: Install Ollama#

macOS and Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows, in PowerShell:

irm https://ollama.com/install.ps1 | iex

You can also use the installer from ollama.com/download. Requirements are macOS Sonoma or newer or Windows 10 22H2 or newer. Ollama on Windows needs at least 4 GB of disk space for the install, and models then take anywhere from a few GB to hundreds of GB.

GPU acceleration works on Apple Metal, NVIDIA and AMD cards, and through Vulkan on Windows and Linux for many others. The hardware page lists the supported cards. Intel Macs run on the CPU only.

Step 3: Run a Model That Fits#

ollama run gemma4

The first run downloads the model, then opens a chat in your terminal. The default gemma4 tag is 6.6 to 9.5 GB. Other sizes, by memory:

Sizes as listed in the Ollama library on 9 November 2026. Avoid tags ending in :cloud if the goal is to keep everything on your machine, because those models run on Ollama's servers.

Step 4: Set the Context Length#

Context length is how much text the model can hold in memory at once, and the default is smaller than most people expect. Ollama's default is 4K tokens below 24 GiB of VRAM, 32K from 24 to 48 GiB, and 256K from 48 GiB up.

4K is fine for short chats. It is too small for long documents, and Ollama says agents, web search and coding tools should use at least 64,000 tokens. You can change it with the slider in the Ollama app, or when you start the server:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

A larger context uses more memory, so check what happened afterwards:

ollama ps

The PROCESSOR column should read 100% GPU. If a share has moved to the CPU, the model no longer fits completely and it will slow down. Reduce the context or use a smaller model.

Step 5: Connect Your Apps Through the Local API#

Ollama runs a server on port 11434 with an OpenAI-compatible API, so most apps built for OpenAI only need a new base URL:

curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma4", "messages": [{"role": "user", "content": "Hello"}]}'

For coding agents, ollama launch claude or ollama launch codex handles the connection for you. Best Local LLM for Coding covers which models and context sizes make sense.

Running a Local LLM with LM Studio Instead#

LM Studio is the route without a terminal. Install the desktop app, search for a model in the built-in downloader, which pulls from Hugging Face, and start chatting. It can also serve models on port 1234, and documents you drop into the app stay on your machine.

Check the requirements first: on a Mac it needs Apple Silicon and macOS 14 or newer, and Intel Macs are not supported. On x64 Windows, your CPU needs AVX2. For the full decision, see Ollama vs LM Studio.

When to Use llama.cpp or vLLM#

llama.cpp suits you if you want direct control over GGUF files or have unusual hardware. It supports Metal, CUDA, Vulkan, SYCL and more, and it pulls models straight from Hugging Face:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Install it from llama.app or the project's releases page.

vLLM is built for serving several users from a GPU server. It requires Linux, with Windows only through WSL. It is more than a single user at a laptop needs.

On Apple Silicon, MLX-based servers from our radar such as oMLX and vllm-mlx are further options. shimmy is a single Rust binary if you want no runtime to manage. To see how they compare, read What is the best local LLM runtime?

Common Problems#

The model is painfully slow. It probably does not fit in GPU memory. Check ollama ps, then pick a smaller model or reduce the context.

The model forgets what it just read. Your context is too small, most likely the 4K default. See step 4.

The download fails or fills the disk. Models can be large. Ollama lets you change where models are stored if your home directory is short on space.

Ollama ignores the graphics card. Check the supported card list and your driver version. NVIDIA cards need driver 550 or newer.

FAQ#

Is it free to run an LLM locally?#

Yes, the software is free. Ollama, llama.cpp and vLLM are open source, and LM Studio has a free plan. You pay for the hardware and the electricity, and only pay a provider if you opt into cloud models.

Can I run an LLM locally without a GPU?#

Yes, but only small models at a usable speed. Ollama and llama.cpp both run on a CPU, which suits batch work more than conversation. Apple Silicon needs no separate graphics card because the GPU shares unified memory with the CPU.

Does a local LLM work offline?#

Yes, after the model has been downloaded. LM Studio documents that chatting, chatting with documents and running a local server need no internet connection. Ollama's cloud-hosted models are the exception, since they need a connection by definition.

Is my data private when I run an LLM locally?#

Your prompts stay on your machine as long as you use local models. Ollama states it does not see your prompts or data when you run locally, and LM Studio states nothing you enter leaves your device. Cloud models in either tool are a different matter.

What is the easiest way to run an LLM locally?#

Ollama, if a terminal is fine, because two commands get you to a working model. LM Studio, if you prefer a desktop app with a model browser.

Sources#

• Ollama on GitHub, quickstart, context length, FAQ, hardware support, OpenAI compatibility, Gemma 4 tags
• LM Studio system requirements, offline operation, model download
• llama.cpp on GitHub
• vLLM on GitHub and GPU installation