Ada's flagship and the 40 series' LLM card: 24 GB and 1008 GB/s — 27B Q4 single-card, 13B at Q8, and the 70B pair that never got its NVLink.
384-bit at 1008 GB/s is the widest consumer memory path Ada ever shipped — ~7% past the 3090 Ti and ~44% past the 4080. Generation speed follows it: ~110–140 t/s on 8B, ~60–78 t/s on 13B Q4, ~19–25 t/s on 27B Q4, with prompt processing at 1400–1800 t/s on 13B. On a single card, the 4090 is the fastest consumer LLM machine of its generation, full stop.
The VRAM pool is what makes the flagship a flagship for LLM work:
The 3090/3090 Ti shipped NVLink at 112.5 GB/s; the 4090 removed it entirely. A 2× 4090 pair therefore splits tensors over PCIe 4.0 with a ~25–35% tax — and that's where the trap is: a 2× 3090 pair, with NVLink and 48 GB, runs the same 70B Q4 workload faster and cheaper at street prices. The 4090 pair wins on 27B Q6/Q8 (which 3090s can't hold per card) and on per-card speed; it loses the flagship 70B story to the previous generation.
450 W, 850 W PSU, 3-slot Founders or AIB cooling. In a 24/7 rig the difference between 450 W and the 3090's 350 W is the price of admission for ~15–20% more speed per card. MTP on this card is measured, not estimated: community A/B runs at n-max 2 put it at ~+60%.
The MTP question: if your model ships a built-in multi-token-prediction head
(Qwen 3.5/3.6/3.8, Gemma 4), the one-flag speculative pass
(--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1) is worth
+60% (measured) on token generation here; community A/B on this card measured +60% at n-max 2 — the largest single-card gain in the record, consistent with its top-of-class bus. The price is ~0.6–2 GB of VRAM, the
output is bit-identical to the unaccelerated run, and models without MTP heads (Llama class) see
nothing. Full tuning rules in the MTP guide.
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B Q8_0 | ~9.7 GB | ~110–140 t/s | ~175–220 t/s | ~1400–1800 t/s | ✅ 64K ctx |
| 13B Q4_K_M | ~8.5 GB | ~60–78 t/s | ~95–125 t/s | ~1400–1800 t/s | ✅ 64K ctx |
| 13B Q8_0 | ~17.4 GB | ~42–54 t/s | ~67–86 t/s | ~1100–1400 t/s | ✅ 16K ctx |
| 27B Q4_K_M | ~19 GB | ~19–25 t/s | ~30–40 t/s | ~900–1200 t/s | ✅ 16K ctx |
| 27B Q5_K_M | ~22 GB | ~16–21 t/s | ~25–33 t/s | ~750–1000 t/s | ⚠️ 8K ctx |
| MoE 30B-A3B (Q4) | ~18 GB | ~80–105 t/s | ~128–168 t/s | ~1200–1500 t/s | ✅ 16K ctx, 3B active |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Figures assume full offload (-ngl 99), ~8K context and batch 2048. Token speed tracks the memory bus almost linearly, so treat ranges as class estimates — your llama.cpp build and the model's attention layout shift them. The 4090 is the only 40-series card with measured MTP data (community A/B, +60%, n-max 2).
| Quant | GGUF size | 1× 4090 (24 GB) | 2× 4090 (48 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ✅ ~19–25 t/s, 16K ctx | ~33–42 t/s, 32K+ ctx |
| Q5_K_M | ~22 GB | ✅ ~16–21 t/s, 8K ctx | ~28–35 t/s, 16K ctx |
| Q6_K | ~25 GB | ❌ ~1 GB over (IQ6_XS fits) | ~25–31 t/s, 16K ctx |
| Q8_0 | ~33 GB | ❌ | ~19–24 t/s, 8–16K ctx |
The single 4090 is the 27B Q4/Q5 tier's home — the only 40-series card that runs it alone. The pair (900 W, 48 GB) reaches Q6 and Q8 with real context; it pays the PCIe 4.0 split tax, which is exactly why the 2× 3090 NVLink rig remains the value 70B/27B-Q8 answer. Four cards (1.8 kW, 96 GB) is where 70B Q6–Q8 and 123B-class demos live.
--spec-type draft-mtp --spec-draft-n-max 2 on any Qwen/Gemma with MTP heads.No NVLink on the 4090 — splits ride PCIe 4.0 at a ~25–35% tax. 24 GB tiles are the generation's largest, though: 2× = 48 GB, 4× = 96 GB.
| Build | GPU power | VRAM | 27B Q8_0 | 70B Q4_K_M | 70B Q5_K_M |
|---|---|---|---|---|---|
| 2× 4090 | 900 W | 48 GB | ~19–24 t/s, 8–16K ctx | ~12–15 t/s, 8K ctx | ~10–13 t/s, 4–8K ctx |
| 4× 4090 | 1800 W | 96 GB | ~30–38 t/s, 32K+ ctx | ~20–26 t/s, 8–16K ctx | ~17–22 t/s, 8K ctx |
The 2-way build is a 48 GB box for 27B Q6/Q8 and 70B Q4 — fast per card, taxed by the split. The 4-way build (1.8 kW, 96 GB, 850 W×2 PSU territory) is the 70B Q5–Q8 rig: it needs a proper workstation, and it drinks like a small office.
When a model won't fit, -ngl N parks the tail layers on the CPU. The mental model
that saves pain: offloaded layers don't run at a fraction of GPU speed — they run at CPU
speed. Handy moves: peel the last 2–4 layers off to reclaim ~0.5–1 GB of VRAM for almost
nothing; keep MoE experts on CPU with --n-cpu-moe; park the output embedding with
-ot output=CPU on big models. Budget 32 GB system RAM for 27B-class offload, 64 GB for
70B. And never offload the KV cache — the session becomes unusable.
llama-server -m 70b-q4_k_m.gguf -ngl 99 -sm layer -c 8192 --tensor-split 1,1
llama-server -m qwen-27b-q4_k_m.gguf -ngl 99 -c 16384 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
The RTX 4090 is the 40 series distilled: the fastest single consumer card of its generation, the only 40-series card that runs 27B alone, the measured +60% MTP card — and the one whose multi-card path got cut off at NVLink's removal. As a single card it's the series' clear flagship. As a pair it's fast, expensive, and outgunned on 70B by a 3090 rig half its price. Buy one, celebrate; buy two, check the 3090s first.