GPU and VRAM calculator
Work out how much GPU memory an open-weight model needs: the weights at your chosen precision, the KV cache for your context length and users, and runtime overhead. Then see which GPUs and Macs fit it.
Model and settings
Users or requests served at the same time.
Architecture (for the KV cache)
Filled from the model's published config where available. KV factor: 1.
Estimated memory needed
0 GB
What fits it
| Hardware | Memory | Fits? |
|---|
Multi-GPU counts assume the model is split evenly with about 10% of each card kept free. Macs: assumes about 75% of unified memory is available to the GPU by default.
How the estimate works
Three things take GPU memory when you run a model: the weights, the KV cache and the runtime itself.
Weights
Every parameter is stored at the precision you choose. Memory in gigabytes is the parameter count in billions times the bits per weight, divided by 8. A 117B model at about 4.85 bits per weight needs roughly 71 GB just to load; at 16 bits it needs about 234 GB.
KV cache
While generating, the model keeps a key and a value vector for every token, in every attention layer, for every sequence it's serving. The standard formula is:
KV bytes = 2 × layers × KV heads × head size × context × sequences × bytes per valueMany recent models cut this sharply. Grouped-query attention shares keys and values across heads, which is already reflected in the KV-head count. Sliding-window and linear-attention layers only keep a short window, compressed (latent) attention stores a much smaller vector per token, and state-space layers keep no per-token cache at all. For models that use these designs, the calculator scales the standard estimate down by a factor shown under the architecture fields. Treat that figure as an approximation and leave headroom.
Runtime overhead
The serving framework, CUDA context and temporary activation buffers add memory too. The calculator adds 0.8 GB plus 6% of the weights, a conservative middle ground across common runtimes such as llama.cpp, Ollama and vLLM. vLLM also pre-allocates a share of free memory for its cache by default, so its reported usage will look higher.
Getting a model to fit
- Drop from 16-bit to 8-bit or a good 4-bit quantization; that's the biggest single saving.
- Shorten the context or serve fewer simultaneous sequences to shrink the KV cache.
- Quantize the KV cache to 8 bits where your runtime supports it.
- Split the model across two or more GPUs, or offload some layers to system RAM if speed matters less.
Step-by-step setup with Ollama, LM Studio and llama.cpp is in how to run LLMs locally, and the local model shortlist lists what fits common hardware.
Planning to self-host?
Get a second opinion on hardware, serving stack and break-even before you buy or rent GPUs.
- Free and vendor-neutral: no model provider pays for placement.
- Your details go only to partners you agree to.
Frequently asked questions
How much VRAM do I need to run an LLM?
For the weights alone, multiply the parameter count in billions by the bits per weight and divide by 8 to get gigabytes. A 30B model needs about 60 GB at 16-bit and about 18 GB at 4-bit. Then add the KV cache, which grows with context length and the number of simultaneous users, plus a gigabyte or two of runtime overhead.
What does quantization do to quality?
Quantization stores weights with fewer bits. 8-bit is usually very close to full precision, and good 4-bit formats keep most of the quality for chat and coding tasks, though results vary by model and task. Below 4 bits, quality drops more noticeably. Test on your own prompts.
Do mixture-of-experts models need less memory?
No. All the experts must be loaded, so memory follows the total parameter count. The advantage is speed: only the active parameters are used for each token, so an MoE model generates faster than a dense model of the same total size.
Can I run a model that doesn't fit in my GPU?
Partly. Tools like llama.cpp can offload some layers to system RAM, which works but is much slower. Splitting a model across several GPUs is the usual approach for large models in production.