The card this whole site points at. 24 GB, a 384-bit bus at 936 GB/s, and NVLink. Every serious local-LLM conversation from 2023 to 2025 started here.
This is the spec that matters most, and it's why the 3090 dominated local LLMs. 24 GB is the line where the lineup's capability tiers reorganize:
No other 30-series card comes anywhere close to this headroom. The jump from 12 GB to 24 GB is not 2× options — it's a different tier of models entirely.
Token generation: ~40 t/s on 13B Q4, ~70–80 t/s on 8B, ~14–17 t/s on 30B Q4. The 30B number is the interesting one: a model class that doesn't fit at all on the smaller cards runs at a fully usable chat speed. Prompt processing (~1300–1600 t/s on 13B) is GA102-scale and makes long-context sessions feel responsive.
The 3090 is the only card in the lineup with a 112.5 GB/s NVLink bridge — 4× a PCIe 4.0 x16 link per direction. In a 2-way tensor split, activations between layers on the two cards travel across that bridge, so the split tax drops from ~30% (PCIe) to ~10–15%. Result: 2× 3090 ≈ 45–50% of the sum of single-card speeds on token generation, versus ~35–40% for non-NVLink pairs at the same model. And 48 GB of VRAM opens 70B Q4_K_M. This is the single most important fact on the multi-GPU side of this site.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+33–55% (measured) on token generation; community A/Bs on this card ran 31.0 → 41.3 t/s (+33%, day-one build) and 41.3 → 63.5 t/s (+54%, newer build), both at n-max 2. It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B (Llama 3.1 8B) | ~4.9 GB | ~95–124 t/s | ~126–192 t/s | ~1500 t/s | ✅ Q8_0, 32K ctx |
| 13B Q4_K_M | ~8.5 GB | ~51–67 t/s | ~68–104 t/s | ~1000–1200 t/s | ✅ 32K+ ctx |
| 30B Q4_K_M | ~18 GB | ~18–25 t/s | ~24–39 t/s | ~450–550 t/s | ✅ 8K–16K ctx |
| MoE 30B-A3B (Qwen3-30B) | ~18 GB | ~75–100 t/s | ~100–155 t/s | ~800 t/s | ✅ fast (3B active) |
| 27B Q5_K_M | ~22 GB | ~16–22 t/s | ~21–34 t/s | ~350–400 t/s | ⚠️ 8K ctx max |
| 70B Q4_K_M | ~41 GB | — | — | — | ❌ (2× 3090 territory) |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates, full GPU offload, ~8K context. MoE token gen depends on active params, not total — treat as a bonus speed tier.
| Quant | GGUF size | 1× 3090 (24 GB) | 2× 3090 (48 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ✅ ~13–16 t/s, 8–16K ctx | ✅ (overkill, ~30 t/s) |
| Q5_K_M | ~22 GB | ⚠️ ~12–14 t/s, 8K ctx max | ✅ ~24–27 t/s, 16K ctx |
| Q6_K | ~25 GB | ❌ ~1 GB over | ✅ ~22–25 t/s, 16K ctx |
| Q8_0 | ~33 GB | ❌ | ✅ ~18–21 t/s, 8–16K ctx |
The 3090 is the only single 30-series card where 27B runs at all. Q4_K_M is the sweet spot: ~19 GB of weights leaves a comfortable 4–5 GB for KV cache. Q5_K_M loads (22 GB) but you're rationing context at ~8K. Q6_K misses by one GB — the most annoying near-miss in this lineup. The 2× build removes every restriction: Q5, Q6 and even Q8 all fit in 48 GB at proper speeds, and NVLink keeps the split tax low. If 27B-at-Q8 is the goal, the dual 3090 is the cheapest Ampere rig that does it comfortably.
-c 16384 fits; use --cache-type-k q8_0 --cache-type-v q8_0 to stretch toward 32K if you're tight.-c at 8192 and you'll be fine; anything bigger starts failing at load time.The single card already runs 27B Q4/Q5 (table above); tiling is where the 3090 becomes something else. NVLink (112.5 GB/s) cuts the split tax from ~30% to ~10–15% — the reason every serious build on this page is a 3090 build.
| Build | GPU power | VRAM | 27B Q6_K | 27B Q8_0 | 70B Q4_K_M | 70B Q5–Q6 |
|---|---|---|---|---|---|---|
| 2× 3090 (NVLink) | 700 W | 48 GB | ~22–25 t/s | ~18–21 t/s | ~15–18 t/s | ~13–15 t/s |
| 3× 3090 (NVLink+PCIe) | 1050 W | 72 GB | ~27 t/s | ~23–26 t/s | ~19–22 t/s | ~15–17 t/s, 32K+ ctx |
| 4× 3090 (2× NVLink) | 1400 W | 96 GB | ~33 t/s | ~30–33 t/s | ~24–28 t/s | ~25–28 t/s, 32K+ ctx |
One honest caveat for 3-way and 4-way: the 3090's NVLink is 2-way only. In a 3× build the third card joins over PCIe; in 4× you get two NVLink pairs. llama.cpp's layer split mostly doesn't care (it balances layers across all devices), so real-world scaling lands between the pure-PCIe and pure-NVLink rates — the table assumes that. 72 GB (3×) is the long-context flagship point: 70B Q5 at 32K+ is a thing on this rig. 96 GB (4×) runs 70B at Q5/Q6 with big context, which no 48 GB box can touch.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. A 27B model that's half offloaded is not 50% of GPU speed — it's ~3–5 t/s
total on a modern 8-core. Practical patterns: drop just the last 2–4 layers to the CPU to
reclaim ~0.5–1 GB of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe;
park the output embedding with -ot output=CPU to save another ~0.5 GB on big
models. Budget 32 GB of system RAM for 27B-class offload, 64 GB for 70B. Never offload the
KV cache — the conversation gets unusably slow. Offloading is rarely the answer here — 24 GB per tile is the point. Its main job in 3090 builds is the -ot output=CPU trick: parking the 70B output embedding (~0.5–1 GB) on CPU to claw back room for KV cache.
llama-server -m 70b-q4_k_m.gguf -ngl 99 -sm layer -c 32768 --tensor-split 1,1,1llama-server -m 70b-q5_k_m.gguf -ngl 99 -sm layer -c 32768 --tensor-split 1,1,1,1 -ot output=CPU
llama-server -m qwen3-30b-a3b-q4_k_m.gguf -ngl 99 -c 32768 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
The reference LLM card of the RTX 30 Series and arguably of the whole pre-4090 era. Single-card, it runs the quality-sweet-spot 30B class at usable speed. Paired, it's the cheapest route to a real 70B experience, and NVLink keeps that pair scaling better than any other consumer combo. If money is the only constraint: this is the card. The 3090 Ti trades 100 GB/s of extra bandwidth (and 100 W) for ~7% more speed — a luxury, not a need.