Guide: VRAM & quantization

How much of a model fits in how much memory — GGUF quantizations, the KV cache tax, and the tricks for squeezing a model into a card that's slightly too small.

GGUF / llama.cpp the VRAM math

What a model is, in gigabytes

llama.cpp loads GGUF models. Quantization is how you squeeze big models into consumer VRAM. The weights plus the KV cache (which grows with context length and batch size) must all fit in VRAM when everything is GPU-offloaded.

Model size	Q8_0 ≈	Q6_K ≈	Q4_K_M ≈	Q3_K_M ≈
8B	~8.5 GB	~6.3 GB	~4.9 GB	~3.9 GB
13–14B	~14 GB	~10.6 GB	~8.5 GB	~6.7 GB
27–32B (Qwen class)	~33 GB	~25 GB	~19 GB	~14.5 GB
70B	~70 GB	~53 GB	~41 GB	~32 GB

Rule of thumb: add ~0.5–1 GB for the KV cache of an 8K–32K context, plus a little for CUDA overhead. A model "fits" when weights + KV + overhead < VRAM — not just when weights < VRAM.

Choosing a quantization

Q4_K_M is the community sweet spot — near-full quality at ~half the size of FP16. If you only remember one quant, this is it.
Q6_K / Q8_0 — diminishing returns in quality, large VRAM cost. Worth it when you have headroom (13B Q8 on 16 GB, 27B Q8 on 32 GB) and the model is your daily driver.
Q3_K_M and below — noticeable quality loss; use when the model simply can't fit otherwise. IQ-quants (imatrix, e.g. IQ4_XS, IQ6_XS) often beat the same-size non-imatrix quants and are the go-to "one size up or down" escape hatch — e.g. IQ6_XS fits 27B into 24 GB where Q6_K misses by 1 GB.
MoE models: total params set VRAM, active params set speed. A 30B-A3B MoE needs ~18 GB (Q4) but generates at a 3B-ish speed if the card's bandwidth allows.

The KV cache, and why context eats VRAM

The KV cache stores per-token attention state and grows linearly with context length. Its size depends on model architecture, not the quant — so a 70B at Q4 with a 128K context can burn more VRAM on KV than a 13B Q8 does in total.

-c 8192 → ~0.5–1 GB on 13B-class; a lot more on 70B-class.
Compress it: --cache-type-k q8_0 --cache-type-v q8_0 halves the KV cache with negligible quality loss — the standard way to stretch context on a tight card.
Never offload the KV cache to CPU — per-token latency collapses.

When a model doesn't quite fit

In order of preference:

Drop one quant step (Q5 → Q4_K_M) — biggest single win.
Use an imatrix quant (IQ4_XS / IQ6_XS) for more quality per GB.
Shrink context (-c) and/or compress KV (q8_0 cache).
Park the output embedding on CPU: -ot output=CPU reclaims ~0.5–1 GB on 27B/70B-class models at nearly zero speed cost.
Partial offload (-ngl < total layers): offloaded layers run at CPU speed, so this is a last resort — keep it under ~10% of layers or the t/s collapses. Details in the multi-GPU guide.
MoE escape hatch: --n-cpu-moe N keeps MoE experts in system RAM — cheap because only a few experts are read per token.