Will it
fit?
Take two identical laptops — same CPU, same RAM — and put a different graphics card in each. The one with more VRAM can run bigger AI models; the other simply can't. Not slower. Can't. That single number is the whole story of running AI on your own hardware.
You've decided you might want to run a model locally — for privacy, control, or to escape the per-token meter. The very first question is brutally practical: will the model you want even fit on the hardware you have? Everything about local AI hardware comes back to that.
VRAM decides which doors open
A GPU has its own dedicated memory called VRAM (video RAM). To run fast, an entire model has to be loaded into that VRAM. So VRAM isn't about speed — it's about access. Think of it as a hallway of doors: each amount of VRAM opens the doors up to a certain model size, and slams shut on everything bigger.
- 16 GB opens the doors most people ever need — everyday chat, brainstorming, light coding assistants.
- 24 GB opens a few more doors at the top — serious 30-billion-parameter coding models.
- 48 GB and up reaches the frontier-class open models.
You can try to run a model that's too big — but the moment it doesn't fit in VRAM, it "spills over" into slow system memory and the CPU, and speed falls off a cliff (more on that below). For practical purposes, if it doesn't fit, the door is closed.
What each VRAM tier actually runs
Here's the practical map, using common 4-bit models. Sizes assume 4-bit quantization (the usual default) plus a little room for context:
| VRAM | Typical GPUs | What runs (4-bit) | Good for |
|---|---|---|---|
| 8 GB | RTX 4060, laptop cards | Up to ~7–8B | Entry chat, simple assistants |
| 16 GB | RTX 5080, 4060 Ti 16GB | ~12–14B dense · 20B MoE | Everyday chat, brainstorming, light coding |
| 24 GB | RTX 3090 / 4090 / 5090 | ~30–32B | Real coding assistants, serious work |
| 48 GB | RTX 6000 Ada, 2×24GB | ~70B | Frontier-class open models |
| 80–96 GB | H100, multi-GPU rigs | 70B at higher precision · 100B+ MoE | Production, heavy multi-model |
| 128–512 GB unified | Mac Studio (M3 Ultra) | The biggest open models (200B–670B MoE) | Everything — our own rack lives here |
At 4-bit, a model needs roughly 0.6 GB of VRAM per billion parameters, plus headroom for context. So ~8B ≈ 6 GB, ~30B ≈ 18 GB, ~70B ≈ 42 GB. Memorize that one line and you can eyeball almost any "will it fit?" question.
Squeezing a model to fit a smaller door
Models are born in 16-bit precision — accurate, but heavy. Quantization compresses those weights down to fewer bits, usually 4-bit, shrinking the model roughly fourfold with only a small quality loss. It's the difference between a model that needs a data-center GPU and one that fits on a gaming laptop.
A 30-billion-parameter coding model is about 61 GB at full size — impossible on a consumer card. Quantized to 4-bit it's about 18.6 GB, which slides right into a 24 GB GPU. Same model, a much smaller door.
It's not just the model that has to fit
Your context window lives in VRAM too. Every token of the conversation — your history, a pasted codebase, retrieved documents — takes up space alongside the model itself. A 30B model that fits comfortably for a quick chat can push right up against a 24 GB card once you load it with a 130,000-token codebase.
And when the total finally exceeds VRAM, the system does the only thing it can: it spills the overflow into ordinary system RAM and runs part of the model on the CPU. That path is dramatically slower — a model that would fly at 60 tokens/second can crawl to 9, which is basically unusable. This is why "it technically runs" and "it's actually usable" are two very different lines, and both are drawn by VRAM.
Prompt processing vs generation
There are actually two speed numbers in local AI, and confusing them causes a lot of misunderstanding:
- Prompt processing (prefill) — the GPU reading and digesting your input. It's massively parallel, so it's blisteringly fast: often thousands of tokens per second.
- Generation (decode) — the model writing its answer one token at a time (this is inference). Because each token depends on the last, it can't be parallelized the same way, so it's much slower — tens to a couple hundred tokens per second.
Watch the gap on a single GPU:
Prompt processing leans on raw compute — more GPU cores make it faster. Generation leans more on memory bandwidth. That's why a beefier card can crush a long prompt but barely pull ahead on short back-and-forth chat: for a quick "hi," there's almost no prompt to process. For reference, you read about 5 words a second — so even a "slow" 60 tokens/second is roughly 30× faster than you can read.
Why a "bigger" model can be faster
One more twist that surprises people. A dense model uses all of its parameters on every token — a 12B dense model runs all 12 billion, every time. A mixture-of-experts (MoE) model is split into many "experts" and only activates a few per token. So a 20B MoE model might only use 3–4 billion parameters at a time.
The result feels backwards: a 20-billion-parameter MoE model can run faster than a 12-billion-parameter dense one, because it's doing less work per token even though it's "bigger" on disk. When you're comparing local models, "how many parameters are active" matters as much as the headline size.
Unified memory — the widest doors
Everything above assumes a discrete Nvidia GPU with dedicated VRAM (running CUDA, the software stack that makes Nvidia the default for local AI). But there's a second road: Apple's unified memory, where the CPU and GPU share one big pool. A Mac Studio with 512 GB of unified memory can hold models that no single consumer GPU could dream of.
That's exactly how our own stack is built — a rack of Mac Studios with 512 GB each running the largest open models entirely on-premise, alongside Nvidia DGX Spark boxes for image and video generation. It's the widest set of doors you can open without a data center, and it's the setup we build for clients who want serious local AI without becoming an infrastructure company.
Want to see what a specific model needs before you buy anything? Try the LLM hardware tool. Want it designed, bought, and standing up in your office? That's a custom AI build. Let's talk →
Common questions
How much VRAM do I need to run an AI model?
Rule of thumb: a 4-bit model needs about 0.6 GB of VRAM per billion parameters, plus headroom for context. So ~8B ≈ 6 GB, ~30B ≈ 18 GB, ~70B ≈ 42 GB. The model and its context window both have to fit, or speed drops sharply.
What GPU do I need for local AI?
VRAM matters more than raw speed. 16 GB (e.g. RTX 5080) comfortably runs everyday models up to ~14B dense or 20B MoE. 24 GB (RTX 3090/4090/5090) opens ~30B coding-class models. 48 GB+ reaches 70B, and large unified-memory machines like Mac Studios run the biggest open models.
What's the difference between prompt processing and generation?
Prompt processing (prefill) is the GPU reading your input — highly parallel and very fast, often thousands of tokens/second. Generation (decode) is the model writing its answer one token at a time, much slower — tens to a couple hundred tokens/second. Two different jobs on the same GPU.
What is quantization?
Quantization compresses a model's weights to fewer bits — e.g. 16-bit down to 4-bit — shrinking it roughly fourfold with only a small quality loss. It's how a model that would need lots of memory is squeezed to fit a smaller GPU.
Dense vs mixture-of-experts (MoE) — what's the difference?
A dense model uses all its parameters on every token (a 12B dense model runs all 12B). An MoE model activates only a few experts per token, so a 20B MoE might use just 3–4B at a time — making it much faster despite a larger total size.