GDDR7 lands in the budget tier. 448 GB/s on a 128-bit bus — double the 3050's bandwidth at barely more power — still capped at 8 GB.
448 GB/s is double the 3050 and 4050 on the same bus width — the first time GDDR7 matters at budget pricing. Token generation scales roughly with bandwidth: 8B Q4 runs at ~55–72 t/s, where a 3050 manages ~30–42. It matches the 3070's bus speed (448 GB/s) at roughly half the power.
Same story as every 8 GB card: 8B models at full quality with a good context; 13B Q4 loads tight at short context; 27B is out of reach single or doubled. The KV cache is the silent consumer — 16K on an 8B model costs ~1 GB.
3840 CUDA cores is the second-smallest count in this series. Prompt processing lands around 450–600 t/s on small models — fine for chat. The 5060 Ti buys you a wider bus of the same speed, not more cores — the two cards share the GB206 die's bandwidth class.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+40–70% (estimated) on token generation; no per-card community A/B exists for this card yet; the estimate follows the low-bandwidth end of the record, which gains the most at n-max 2. It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B (Llama 3.1 8B, Mistral) | ~4.9 GB | ~55–72 t/s | ~77–122 t/s | ~450–600 t/s | ✅ Q8_0, 16K+ ctx |
| 13B Q4_K_M | ~8.5 GB | ~30–42 t/s | ~43–71 t/s | ~500 t/s | ⚠️ tight, short ctx only |
| 13B Q3_K_M | ~6.7 GB | ~34–46 t/s | ~48–78 t/s | ~550 t/s | ✅ with quality loss |
| 27B | ~19 GB | — | — | — | ❌ no |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates for full GPU offload (-ngl 99), ~8K context, batch 2048. Token generation scales roughly with memory bandwidth; individual runs vary by model architecture (GQA vs MHA, MoE) and llama.cpp build.
| Quant | GGUF size | 1× 5060 (8 GB) | 2× 5060 (16 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ❌ ~11 GB over | ❌ ~3 GB over |
| Q5_K_M | ~22 GB | ❌ | ❌ |
| Q6_K | ~25 GB | ❌ | ❌ |
| Q8_0 | ~33 GB | ❌ | ❌ |
Every 27B quant is out of reach until 4× (32 GB), where Q4_K_M lands around ~15–19 t/s at 4–8K context. Below that, partial CPU offload (~3–5 t/s) is a demo, not a daily driver.
-c.No NVLink: tensor split over PCIe 5.0 with a ~20–30% tax, 8 GB per tile.
| Build | GPU power | VRAM | 13B Q4 | 27B Q4_K_M |
|---|---|---|---|---|
| 2× 5060 | 290 W | 16 GB | ~36–48 t/s | ❌ ~3 GB over |
| 4× 5060 | 580 W | 32 GB | ~55–70 t/s | ~15–19 t/s, 4–8K ctx |
4× (32 GB) is the first build where 27B Q4 runs, at a modest pace. It needs a workstation-class board with four working slots; the 2-slot bodies fit a full tower easily.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. Practical patterns: drop just the last 2–4 layers to the CPU to reclaim ~0.5–1 GB
of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe; park the output
embedding with -ot output=CPU to save another ~0.5–1 GB on big models. Budget 32 GB
of system RAM for 27B-class offload, 64 GB for 70B. Never offload the KV cache — the conversation
gets unusably slow.
llama-server -m 27b-q4_k_m.gguf -ngl 99 -sm layer -c 8192 --tensor-split 1,1,1,1
llama-server -m llama-3.1-8b-q6_k.gguf -ngl 99 -c 8192 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
The best 8 GB card for LLMs ever made — double the 3050's generation speed at comparable power. But 8 GB is 8 GB: the card that actually changes what you can run is one slot over. If your ceiling is 8B and low power, buy it; if 13B+ or MoE models are on the roadmap, put the extra money into the 5060 Ti 16G instead.