How to run LLMs locally

Running a language model on your own computer takes minutes once you know three things: how much memory the model needs, which file format to download and which tool will serve it. This guide covers the memory math, the main quantization formats, eight popular tools, a working Ollama setup and the open models that fit common hardware as of October 2026.

Why run an LLM locally

Local inference means the weights sit on your disk and every token is computed on your own CPU or GPU. Four reasons push people to do it.

• Privacy. Prompts, documents and outputs never have to leave the machine, which simplifies work with client files, unreleased code or regulated data. Treat it as one control among many, not a compliance guarantee.
• Cost. There are no per-token fees. Once you own the hardware, the marginal cost is electricity, which favors heavy, steady workloads such as bulk tagging or an all-day coding assistant.
• Offline use. A local model works on a plane, inside an air-gapped network and during a provider outage, with no rate limits.
• Control. You choose the exact model version, sampling settings and system prompt, and nothing gets retired unless you retire it. You can also fine-tune and run your own variants.

The trade-offs are real. The best open-weight models that fit on a desktop still trail the top hosted models on hard reasoning and long agentic tasks, and you become the operator responsible for updates and security.

Hardware basics: memory decides what you can run

VRAM, system RAM and unified memory

A model runs fastest when its weights and working memory fit entirely in GPU memory, or VRAM. Generating each token requires reading every active weight once, so speed is mostly limited by memory bandwidth. A GPU with about 1 TB/s of bandwidth holding a 16 GB model tops out near 60 tokens per second in theory (1,000 / 16), and real results land somewhat lower.

System RAM works too, but desktop memory delivers well under 100 GB/s, so CPU-only inference is several times slower. When a model slightly exceeds VRAM, llama.cpp-based tools split it between GPU and CPU, and speed drops as more layers move to the CPU.

Apple Silicon Macs use unified memory shared by the CPU and GPU, with bandwidth from roughly 100 GB/s on base chips to several hundred GB/s on Max and Ultra chips. A 64 GB or 128 GB Mac can load models no single consumer graphics card can hold. By default macOS lets the GPU use only part of that pool, commonly around two-thirds to three-quarters, so budget accordingly.

This is why mixture-of-experts (MoE) models are popular locally. Qwen3.6-35B-A3B stores 35 billion parameters but activates about 3 billion per token, so it needs the memory of a 35B model while generating at speeds closer to a small one.

A rule of thumb for weight memory

Weight memory is parameters times bytes per parameter. Quantized files carry a little extra for scaling factors, so 4-bit models work out to about 0.55-0.6 GB per billion parameters.

Then add 10-20 percent for runtime buffers, the KV cache for your context length and 1-2 GB of headroom if the same GPU drives your displays. Ollama's default build of Qwen3.6 27B, for example, is a 17 GB download, which makes it a natural fit for a 24 GB card.

The KV cache grows with context

During generation the model stores attention keys and values for every token in the conversation. This KV cache grows linearly with context length and with the number of parallel requests.

KV cache bytes = 2 x layers x KV heads x head size x bytes per value x tokens

Llama 3.1 8B has 32 layers, 8 key-value heads and a head size of 128. At 16-bit precision that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KiB per token. An 8K context needs 1 GiB, 32K needs 4 GiB and the full 128K needs 16 GiB, more than three times the 4-bit weights.

Newer designs that mix linear-attention layers with full attention, such as the Qwen3.5 and Qwen3.6 families, need much less per token. The VRAM calculator runs this arithmetic for specific models.

Quantization formats and quality trade-offs

Quantization stores weights in fewer bits. Going from 16-bit to 4-bit cuts weight memory by roughly 70 percent, and for most chat, writing and coding work the quality loss at 4 bits is small. Losses grow quickly below 4 bits and hit small models harder than large ones.

Practical defaults:

• Start with Q4_K_M in GGUF or 4-bit MLX. Step up to Q5_K_M, Q6_K or Q8_0 when memory allows and the task punishes small errors, such as code, math, structured extraction or less common languages.
• A common rule of thumb holds down to about 4 bits: a larger model quantized harder usually beats a smaller model at higher precision in the same memory. Below 4 bits, test carefully.
• Prefer files from the model's publisher or well-known quantizers. Some labs ship quantized weights directly; OpenAI released gpt-oss in the 4-bit MXFP4 format, which avoids an extra lossy conversion.
• If you create your own GPU quantizations, note that the original AutoAWQ package has not shipped a release since May 2025, while GPTQModel and the vLLM project's llm-compressor are actively maintained.

Local LLM tools compared

Most tools are front ends or servers built on a few engines: llama.cpp for GGUF, MLX for Apple Silicon and vLLM for GPU servers. Status below reflects each project's releases as of October 2026.

A sensible path: start with Ollama or LM Studio, add Open WebUI when several people need a chat interface and switch to vLLM when an application needs real concurrency.

Quick start with Ollama

1. Install. On macOS or Linux, run the script below. On Windows, use the installer from the Ollama website or run irm https://ollama.com/install.ps1 | iex in PowerShell. A Docker image, ollama/ollama, is also available.

curl -fsSL https://ollama.com/install.sh | sh

2. Pull a model sized for your machine. Gemma 4 E2B is about a 7 GB download and suits 8 GB machines; on a 24 GB GPU, try Qwen3.6 27B (17 GB).

ollama pull gemma4:e2b
ollama pull qwen3.6:27b

3. Chat in the terminal. Type /bye to exit. The --verbose flag prints generation speed after each reply.

ollama run gemma4:e2b --verbose

4. Confirm it runs on the GPU. Run ollama ps. The PROCESSOR column should read 100% GPU; a split such as 40%/60% CPU/GPU means part of the model spilled into system memory. The CONTEXT column shows the context length actually allocated.

5. Call the API. Ollama listens on port 11434.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e2b",
  "messages": [{"role": "user", "content": "Explain the KV cache in two sentences."}],
  "stream": false
}'

6. Reuse OpenAI-style code. Point any OpenAI SDK at http://localhost:11434/v1 with a placeholder API key.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma4:e2b", "messages": [{"role": "user", "content": "Say hello"}]}'

7. Raise the context length. Ollama's docs list a 4,096-token default on machines with under 24 GB of VRAM, which silently truncates long documents. Set a server-wide value as below, or pass "options": {"num_ctx": 32768} per request.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Which open models fit your hardware

The table assumes roughly 4-bit quantization and 8K-32K tokens of context. Picks are open-weight releases available as of October 2026, mostly Apache 2.0; confirm the license on each model card and check that your runtime supports the architecture.

Frontier-scale open models, such as the DeepSeek V4 family and Qwen3.8's 2.4-trillion-parameter flagship, need multi-GPU servers. Browse the model directory for specs and licenses.

Performance tips

• Keep the whole model on the GPU. Check ollama ps or your tool's logs. A smaller quant that fits fully usually beats a bigger one that spills.
• Size the context to the job. Allocating 128K tokens when you need 8K wastes memory and can force CPU offload.
• Quantize the KV cache. In Ollama, OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves KV memory with very little quality loss; it requires flash attention, which Ollama enables automatically on supported hardware.
• Prefer MoE models for speed. Few active parameters mean fewer bytes read per token, which matters most on Macs and CPUs.
• Try multi-token prediction drafts. Gemma 4 and Qwen3.6 support speculative decoding variants in llama.cpp, LM Studio and Ollama; gains depend on model and hardware.
• Keep models loaded. Ollama unloads idle models after five minutes; set OLLAMA_KEEP_ALIVE higher to avoid reload delays.
• Use a server built for concurrency. Ollama handles one request per model at a time unless you raise OLLAMA_NUM_PARALLEL, which multiplies KV memory. vLLM's continuous batching and PagedAttention deliver far higher total throughput for many users.

Troubleshooting common problems

Common mistakes

• Buying hardware first. Test the exact model on a rented GPU or hosted endpoint before spending thousands of dollars.
• Judging a model by a 2-bit or 3-bit file. Those squeeze huge models into small machines and lose noticeable quality.
• Forgetting the KV cache. A model that loads at 4K tokens can fail at 64K.
• Assuming local means private. Ollama and LM Studio also offer cloud-hosted models, so confirm a model actually runs on your machine before sending sensitive data.
• Downloading random re-uploads. Stick to the publisher or well-known quantizers, and check the license allows your use.

Next steps

Estimate memory for a specific model, quant and context length with the VRAM calculator. If cost is your main motivation, compare hardware and hosting against per-token prices in the self-host vs API calculator. For the bigger picture on licenses and capability gaps, read open-source vs closed LLMs. If you are planning a private deployment for a team and want hands-on help, you can work with an LLM expert.