The value card of this series. 16 GB of GDDR7 at mainstream money — double the VRAM of the 4060 Ti 16G for similar cost, on a 128-bit bus at 448 GB/s.
This is the spec that matters. 16 GB is the line where the lineup reorganizes:
No 30-series card at this price offered 16 GB. The jump from the 8 GB cards isn't 2× options — it's the MoE and 27B tiers at all.
The 128-bit bus caps token generation at 3070-class speed: ~55–72 t/s on 8B, ~30–42 t/s on 13B Q4. For the models this VRAM unlocks (27B Q3, 30B-A3B MoE), that's plenty for chat — MoE generation in particular runs at ~70–90 t/s because only the active experts stream through the bus.
4608 cores at 180 W — prompt processing around 500–700 t/s on 13B. The card is quiet, cool and cheap to run; the value case is VRAM-per-dollar, not speed.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+40–70% (estimated) on token generation; no per-card community A/B exists for this card yet; the estimate follows the low-bandwidth end of the record, which gains the most at n-max 2. It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B Q8_0 | ~9.7 GB | ~55–72 t/s | ~77–122 t/s | ~500–650 t/s | ✅ 32K ctx |
| 13B Q4_K_M | ~8.5 GB | ~30–42 t/s | ~43–71 t/s | ~550–700 t/s | ✅ 32K+ ctx |
| 13B Q6_K | ~13.8 GB | ~25–33 t/s | ~36–56 t/s | ~500–650 t/s | ✅ 8K ctx |
| 27B Q3_K_M | ~15.8 GB | ~13–18 t/s | ~19–31 t/s | ~350–450 t/s | ⚠️ 4–8K ctx |
| MoE 30B-A3B (Q3) | ~14.5 GB | ~70–90 t/s | ~100–150 t/s | ~500–700 t/s | ⚠️ 4–8K ctx, 3B active |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates for full GPU offload (-ngl 99), ~8K context, batch 2048. Token generation scales roughly with memory bandwidth; individual runs vary by model architecture (GQA vs MHA, MoE) and llama.cpp build. MoE token gen depends on active params, not total — treat as a bonus speed tier.
| Quant | GGUF size | 1× 5060 Ti (16 GB) | 2× 5060 Ti (32 GB) |
|---|---|---|---|
| Q3_K_M | ~15.8 GB | ⚠️ ~13–18 t/s, 4–8K ctx | ✅ (overkill) |
| Q4_K_M | ~19 GB | ❌ ~3 GB over | ✅ ~24–30 t/s, 8–16K ctx |
| Q5_K_M | ~22 GB | ❌ | ✅ ~20–25 t/s, 8K ctx |
| Q6_K | ~25 GB | ❌ | ✅ ~18–22 t/s, 8K ctx |
| Q8_0 | ~33 GB | ❌ | ❌ ~1 GB over |
Single-card, 27B means Q3 — usable for chat, a real quality step down from Q4. The 2× build (360 W, 32 GB) is the cheapest proper 27B rig on this site: Q4_K_M at ~24–30 t/s with 16K context, Q5 at 8K, and MoE 30B models at Q4 (~18 GB) running at ~95–125 t/s.
-c 8192 and quantized KV (--cache-type-k q8_0 --cache-type-v q8_0) to make it fit.No NVLink: tensor split over PCIe 5.0 with a ~20–30% tax. 16 GB per tile means a pair reaches 32 GB — the 27B Q4/Q5 line.
| Build | GPU power | VRAM | 27B Q4_K_M | 27B Q5_K_M | MoE 30B Q4 |
|---|---|---|---|---|---|
| 2× 5060 Ti | 360 W | 32 GB | ~24–30 t/s, 8–16K ctx | ~20–25 t/s, 8K ctx | ~95–125 t/s |
The 2-card build is the whole point of this card in a multi-GPU context: 32 GB at 360 W, and MoE models at full Q4 quantization. A normal ATX board with two x16 (or x8/x8) slots is enough; no workstation hardware required.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. Practical patterns: drop just the last 2–4 layers to the CPU to reclaim ~0.5–1 GB
of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe; park the output
embedding with -ot output=CPU to save another ~0.5–1 GB on big models. Budget 32 GB
of system RAM for 27B-class offload, 64 GB for 70B. Never offload the KV cache — the conversation
gets unusably slow.
llama-server -m 27b-q4_k_m.gguf -ngl 99 -sm layer -c 16384 --tensor-split 1,1
llama-server -m qwen3-30b-a3b-q3_k_l.gguf -ngl 99 -c 8192 -b 2048 -ub 2048 --cache-type-k q8_0 --cache-type-v q8_0 --host 127.0.0.1 --port 8080
The value card of the RTX 50 Series: 16 GB GDDR7 at mainstream money puts the 13B quality tier, MoE models, and (at Q3) 27B on a single quiet 180 W card. The 128-bit bus is the ceiling — if you want 3090-class generation speed alongside the VRAM, the 5070 Ti is the step up. For the price, few cards in any generation offered more model-per-dollar.