Local LLM experiments on one RTX 5090
A coding assistant running on one RTX 5090. 23 model setups that fit in its 32 GB, tested one setting at a time. Some tests are done; others are still running.
Gaps are paired by item vs a control run, with 95% CIs. ★ = CI excludes zero. Hover a dotted term for its definition.
Overview
Everything runs on one RTX 5090. Below is what's on the bench, then what each setting turned out to cost. Every row links to the tab with the working.
| Setup — what's on the bench | ||
|---|---|---|
| Models | 23 model configurations and the settings tested on each | No single winner: nine builds tie on quality. Picks by speed and memory are below, with the command to run them. |
| Engines | llama.cpp vs vLLM: does the stack change the answers? | llama.cpp serves everything today. vLLM is used for comparisons on matching weights. |
| Hardware | The one box everything runs on | RTX 5090 32 GB · Ryzen 9900X · 64 GB DDR5 · all Micro Center. |
| Evals | The eight tasks every cell is scored on | Frozen at run start. No LLM judges. Item count sets how small a gap each task can resolve. |
| Configurations | The factor grid the experiments walk | 276 cells; one knob varied at a time, everything else pinned. |
| Results — what was learned | ||
| Weight precision | Can I recover 5–10 GB without losing accuracy? | For Qwen3.6, yes. Qwen3.8 is more sensitive: Q4_XL and NVFP4 lose 5–6 points of code reading against its 8-bit version. |
| KV cache | Can I shrink the cache for free? | Nearly: q8_0 averages −0.3 points, with ~2-point code-reading losses on 2 of 21 models. Still my choice for the memory. Long-context retrieval still unchecked. |
| MTP | How much speed does MTP drafting buy? | Quality preserved. About 2.1× faster decode in the server logs from the quality runs; the dedicated speed ladder hasn't reported yet. |
| Dense vs MoE | Which gives me the best balance of speed and accuracy? | Currently the dense 27B: quality costs me more than latency. New MoEs are queued that could change this. |
| Load | Does quality survive 8 concurrent requests? | Untested. It's the last check before I switch my everyday server over, queued next. |
| Watermark | Can output carry a detectable watermark without costing quality? | Detection pipeline validated on CPU; the quality-cost test runs on the GPU after the switch-over decision. |
Picks, and how I serve them
Every build below sits in the quality tie, so choose on speed and memory. Decode speeds are single-request; MTP speeds come from server logs during the quality runs.
| Fastest | qwen36-35b-a3b-q4 (MoE, UD-Q4_K_M) | 281 tok/s · 22.1 GB |
| Dense, with MTP | qwen36-27b-mtp-q4xl (UD-Q4_K_XL) | about 160 tok/s · 20.1 GB |
| Least memory | qwen38-27b-mtp-nvfp4 (NVFP4) | about 183 tok/s · 17.9 GB · instruction-following deficits under check (see Precision) |
How I serve them: llama-server in router mode, started by a systemd user service with --models-preset llama-models-router.ini --models-max 1. One model is resident at a time; a request's model field picks it, and the router swaps models on demand. Each model is a section in the preset. The day-to-day default is Qwen3-Coder; this is the dense 27B with MTP, as it runs now:
[qwen3.6-27b-mtp-q5]
model = Qwen3.6-27B-MTP-Q5_K_M.gguf
gpu-layers = all
ctx-size = 65536
parallel = 1
flash-attn = true
jinja = true
reasoning = off
chat-template-kwargs = {"enable_thinking":false}
spec-type = draft-mtp
spec-draft-n-max = 2
no-cache-prompt = true
cache-ram = 0
ctx-checkpoints = 0MTP profiles turn prompt caching off. Serving uses a llama.cpp build from the MTP branch; the quality runs used llama.cpp 0.2.0, both built with CUDA for the 5090 (sm120). The 35B-A3B MoE isn't in my preset yet.
Each tab holds a toy model you can play with (simulated), the run's actual measurements, and my call.
Models
Checkpoints benchmarked on this machine: decode speed (log scale) vs quality, the mean pass rate on the four large tasks every ranked model completed validly (GSM8K, BigCodeBench, CRUXEval, IFEval). The Qwen 27B dense builds and the 35B-A3B MoE share the top; the other MoEs are faster and lower. Models with a failed or invalid run on one of those tasks aren't plotted; the table says which. 18 of the 23 baseline runs (Aug 25–31) used a server flag that garbled the chat template, so their scores may be understated until re-run (see Evals).
The full registry, and which settings each checkpoint supports: an MTP draft head is a property of the checkpoint. A vLLM twin exists only where the same weights are published as an HF checkpoint. The KV columns show which cache quantizations have measured cells so far. Click a column header to sort.
No single pick. Nine builds tie on these tasks: every Qwen3.6 and Qwen3.8 27B quant, with or without MTP. Within a tie I choose on speed, memory and context, so the picks on the Overview are ordered that way, not by quality. This holds for these temperature-0 coding and instruction-following tests on this card, not as a general best pick for 32 GB. The field moves monthly, so four candidates are already queued for after this run's ledger freezes: Nemotron 3.5 Lightning, Ornith 1.5 35B-A3B, GLM-5.3-Flash and Nex-N2.5-mini 35B. Anything new that fits in 32 GB gets the same paired treatment against the frozen baseline rather than a fresh start.
Evals
Eight tasks, chosen to cover what a coding-assistant server actually does, and frozen at run start because a suite that shifts mid-campaign can't support paired comparisons. Item counts matter: they set how small a gap each task can resolve.
| task | items | what it measures | resolution |
|---|---|---|---|
| gsm8k_cot_zeroshot | 1,319 | grade-school math word problems, step-by-step reasoning | fine (~±1 pt) |
| bigcodebench_full | 1,140 | realistic library-using coding tasks, executed and unit-tested | fine |
| cruxeval_output | 800 | predicting what a piece of code prints (code reading) | fine |
| ifeval | 541 | following precise, checkable instructions ("exactly 3 bullets…") | medium |
| ifbench | 300 | harder instruction-following | medium (~±3–5 pts) |
| humaneval_instruct | 164 | classic small Python functions, executed | coarse |
| bigcodebench_hard | 148 | the hardest bigcodebench slice | coarse |
| internal_workload | 5 | private smoke check of the server's real prompts | go/no-go only |
Scoring is automatic end-to-end: code tasks run the generated code against tests; instruction tasks check constraints programmatically; math extracts the final answer. No LLM judges anywhere: a judge model would itself be a confound when comparing engines and quantizations.
A broken week. From Aug 25 to 31 the server ran with a flag that forced a plain chat template, so answers began with stray template text. HumanEval scored zero for all 18 models run in that window, so it's left out of every ranking; only 5 models have valid HumanEval scores. Their other tasks ran under the same template and may read low until re-run. Thinking was off for every quality run.
Coverage is deliberately skewed toward code and instruction-following, because that's the server's job. What it cannot see: long-context retrieval (queued with the 131k pass), conversational quality, and anything a 5-item smoke check merely gestures at. A capped general-knowledge pass (HLE) is queued as a breadth check, not a decision input.
It also doesn't test the settings I use day to day. Each item is one temperature-0 answer, which makes for a clean comparison between configurations, but says nothing about reasoning-effort settings, multi-step coding tasks, retries, tool-call failures, or total time to a finished task. Those need a separate test on real work.
Grid
Every experiment tab is a slice of one crossed design: each cell is a (model × engine × settings) combination run against the full quality suite under identical conditions. These are the factors and the levels being crossed:
| factor | levels | experiment tab |
|---|---|---|
| model checkpoint | 23 registered (see Models) | Models · Dense vs MoE |
| weight precision | Q4_XL · Q5 · Q6 · Q8 · F16 · NVFP4 | Weight precision |
| KV cache type | f16 · q8_0 · q4_0 | KV cache |
| draft depth (MTP) | 0 · 2 · 4 | MTP |
| engine | llama.cpp · vLLM (where mapped) | Engines |
| concurrency (np) | 1 · 8 (quality); 1·2·4·8 (speed) | Load |
| watermark bias δ | 0 · 2 · 4 (γ=0.25, h=4) | Watermark |
| held constant | temp 0 · seed 1234 · 32k context · same prompts | every cell |
Each factor's range: the grey track is what the software allows, solid dots are levels with measured cells, hollow dots are levels queued but not yet run. Hover a dot.
Not every combination runs; the full cross would be thousands of cells. The design measures each factor where it can matter and drops cells whose outcome is provably implied by others (e.g. depth × quantized-KV interactions, since verification makes depth quality-inert). Repeat-run cells re-run identical configurations to measure the noise floor that every comparison is judged against.
The question is never "is model A good." It's "does this knob change the answer while everything else is pinned?" Paired cells that differ in exactly one factor are what let a 0.4-point gap be attributed to the cache instead of to luck.
Weight precision
It depends on the model. Qwen3.6 compresses well: against Q6, Q5 is within noise on every task and Q4_XL differs only on GSM8K (−1.1 points), so for it I use the smallest XL quant and spend the memory on context. Qwen3.8 does not: against its own 8-bit version, Q4_XL loses 4.9 points of code reading (CRUXEval), Q5 loses 2.0, and NVFP4 loses 5.8, all well above the 0.6-point run-to-run noise. This only became visible after the Oct 4 scoring fix doubled CRUXEval's usable range. Separately, NVFP4's 4-point instruction-following gap against Qwen3.6 is not a compression cost: Qwen3.8 at 8 bits shows the same gap. So NVFP4 carries two different costs: the newer model's weaker instruction-following, and its own code-reading loss.
The question: can I recover 5–10 GB for context without losing accuracy? Quantization stores weights in 4–8 bits instead of 16. There are reasons it can work: rounding errors tend to average out in large sums. But where it fails is not predictable from theory, so every cell below is measured rather than assumed.
Same family, only bit-width changes: the line stays flat while VRAM moves 5–10 GB. Flat for Qwen3.6 and the f16 controls; Qwen3.8 dips at Q4_XL, almost all of it code reading (CRUXEval). The y-axis here averages the five tasks named in the legend, so it sits below the Models registry's quality interval, which averages the full suite; the two are not meant to match.
The exception: NVFP4 shows four starred deficits: two instruction-following (IFEval −4.1, IFBench −5.3 points), one code reading (CRUXEval −3.4) and one math reasoning (GSM8K −1.4). Against Qwen3.8 at 8 bits the instruction-following gaps go away (IFEval +0.4, IFBench −2.3, GSM8K −0.1, none significant) but code reading does not (CRUXEval −5.8 [−7.9, −3.6] ★, BigCodeBench −1.75 [−3.3, −0.2] ★): the instruction-following gap is Qwen3.8's own, the code-reading loss is NVFP4's. Both sides of the IFBench comparison hit the output-length cap on more than 5% of items, so that gap is the least certain. Repeat runs of identical settings differ by at most 0.6 points, so every starred gap here is well above noise.
KV cache
I quantize the KV cache to 8 bits. On average it costs almost nothing: on code reading, the most sensitive task once its scoring was fixed, q8_0 averages −0.3 points across 21 models. Two of those 21 showed small significant losses (Qwen3 Coder −1.9, the NVFP4 candidate −1.75); with 21 comparisons about one would show up by chance, so I read that as a possible model-specific cost of about 2 points, not a general one. No other q8_0 comparison showed a significant loss. q4_0 costs more: five significant losses across three models (GLM-4.7 Flash on three tasks, Nemotron Nano and Qwen3 14B on code reading). That still leaves losses these tests can't detect; the main untested case is retrieval at long context, which gets checked at 131k.
The question: can I shrink the cache for free? The KV cache holds previous tokens' attention state, but how much depends on the architecture. A conventional dense model like qwen3-32b keeps it for every layer, roughly 8 GB at 32k context in f16. Qwen3.8 27B keeps it for only 16 of its 64 layers (the rest use linear attention), so its cache at 32k is closer to 2 GB. It should tolerate quantization better than weights do, because attention renormalizes scores. But "should" is the thing under test, and long-range retrieval is where I'd expect it to fail first.
Every clean gap estimate as one dot: the distribution behind the verdict. Point at or tap a dot for its model, task, CI and n.
The payoff: q8_0 saves from under 0.1 GB (hybrid models with few full-attention layers) to 3.8 GB (qwen3-32b), q4_0 up to ~5.9 GB of measured peak VRAM, all at a configured 32k context. That's context capacity, not tested long-context quality; retrieval at long context is still queued.
MTP drafting
Quality results are reassuring, and the first speed evidence is strong. In server logs from the quality runs (depth 2, 32k context, one request at a time), about 85% of drafted tokens were accepted and median decode went from 71 to 151 tok/s on Q5_K_M and from 77 to 160 on Q4_K_XL, about 2.1×. The NVFP4 build reached 183. These are a by-product of the quality runs; the dedicated depth ladder will confirm them.
Why MTP and not another method: the checkpoints I care most about ship an MTP head, so there's no separate draft model to fit in 32 GB. llama.cpp also supports draft models, EAGLE-3 and n-gram speculation; none of those are tested here. Memory: the MTP checkpoints peak about 0.9 GB higher than the same weights without the head (32k context, f16 cache).
The question: how much speed does MTP drafting buy on my workload? A small head guesses the next few tokens; the big model verifies them in one pass and keeps only tokens it would have sampled anyway, so the output distribution is preserved. Speed depends on acceptance rate and drafting overhead, which is why it's measured rather than assumed.
MTP-on vs MTP-off at identical weights: every plotted delta is inside ±0.01, consistent with distribution-preserving verification. Theory does not bound finite-sample deltas, so this is evidence rather than proof. One pair is not plotted: HumanEval on Q5_K_M shows +0.98, far too large to be an MTP effect, because the MTP-off run fell in the Aug 25–31 template bug (see Evals). It stays out until that run is repeated.
The tok/s column is median decode from server logs during the quality runs. The dedicated speed run at depth {0,2,4} hasn't reported yet.
Dense vs MoE
I currently prefer the dense 27B: on my workload the quality gap costs more than the latency gap. Since the CRUXEval scoring fix the Qwen3.6 35B-A3B MoE no longer ties the dense 27Bs: it sits about 3 points below the leader, just outside its interval. At 281 tok/s it's roughly 4× a dense 27B without MTP and about 1.8× one with MTP, so it is the strongest MoE alternative in this set, and an MoE that closed the coding-benchmark gap would change my answer. What this suite doesn't measure could still change it: multi-step coding, tool calls and long context. Nemotron and Ornith are queued.
The question: which model gives me the best balance of speed and accuracy? Dense models run every token through all parameters; these MoEs activate under 10% (3B of 35B), so they read far less memory per token, and memory bandwidth is the wall on one GPU. That mechanism predicts the speed difference well. It does not predict the quality difference; that's what the measurements are for.
Every model with valid scores on the four large tasks every ranked model completed validly (GSM8K, BigCodeBench, CRUXEval, IFEval): decode speed (log) vs mean pass rate. Among these tested checkpoints, the dense 27Bs and the 35B-A3B MoE share the top band; the other MoEs are faster and lower. This supports picking a model from this set; it does not isolate architecture as the cause (training data, scale and tuning all differ).
Load
I won't switch my everyday server over on single-request quality alone. The production server runs loaded, and the new config stacks three changes at once. If quality at np=8 holds within the run-to-run noise the repeat runs measure, the tied builds stay interchangeable on quality and the choice comes down to speed and memory.
Every quality number so far was taken one request at a time. Under batching, kernel shapes and cache sharing change the arithmetic, so the same prompt can legitimately produce different tokens. Throughput at np∈{1,2,4,8} is already measured (the levels are listed under Grid; that chart is not on this page yet); quality under 8-way load never has been. That measurement runs right after the repeat runs.
Watermark
Pipeline validated for zero GPU time. Next: quality cost on the production model (study W1: dose-response), whether quantized KV blunts detection (W2), and whether the mark survives paraphrase and splicing (W5), after the switch-over decision.
The question: can the server's output carry a detectable watermark without costing quality? Hash the last h tokens → split the vocab into a γ "green" fraction → add bias δ to green logits. Text drifts green in a way only the key-holder can test. Detection is a statistical test, not proof of authorship: short or heavily edited text may not carry a detectable signal. Detection strength grows ~√length; δ trades detectability against a possible quality tax.
Detection increased with watermark strength and with text length, and every generation is visible: the control cloud sits at z≈0 (0/30 detections), δ=2 straddles the threshold (16/30 at 300 tokens), δ=4 clears it for 29/30. The spread within each dose is as informative as the means. (One pilot, one small model: staging evidence, not the study.)
Engines
The engine is a knob like any other: the same weights can serve faster or slower, and in principle differently, depending on who runs them. The trap in comparing engines is comparing files instead: llama.cpp serves GGUF, vLLM serves HF checkpoints, and two files of "the same model" usually aren't. This run's design: for a handful of matched-weight comparisons, the GGUF was converted losslessly from the exact HF checkpoint vLLM serves, so an f16 gap is purely the stack; the 27B Q8-vs-FP8 row adds one controlled variable (8-bit format) on top.
| llama.cpp v0.2.0 | vLLM 0.27.1 | |
|---|---|---|
| model format | GGUF (all 23 rows) | HF checkpoints (4 mapped twins) |
| quantized weights | K-quants, NVFP4 | FP8, NVFP4 (blocked on this GPU) |
| quantized KV cache | q8_0 / q4_0, any model | FP8, where the checkpoint pins it |
| speculative decoding | MTP head (3 checkpoints) | not exercised in this run |
| concurrency model | continuous batching (on by default), slot-split cache; np≤8 tested | continuous batching, paged attention; not load-tested yet |
| tool-calling API | supported (--jinja); not tested | supported; not tested yet (queued) |
| Blackwell (sm120) status | clean | partial as of 0.27.1 (see below) |
Three findings, all discovered by trying rather than reading release notes. They apply to vLLM 0.27.1 and haven't been retested on newer releases. One: vLLM 0.27.1 routes hybrid-architecture prefill through a JIT compiler (FlashInfer) that cannot build for this GPU's compute capability with the CUDA toolkit the machine has. That makes the NVFP4 twin unservable on vLLM here. Two: a checkpoint that pins FP8 KV in its own config silently selects an incompatible attention backend unless the launcher detects it and overrides, a class of failure where the config is honored and the server still dies. Three: at 32 GB, the f16 matched-weight comparisons are the only place both stacks can hold identical weights; everywhere else the honest comparison is "stack + format", not "stack".
The quality gaps from those comparisons (f16 rows, Q8-vs-FP8 row) live in the run ledger and land in the decision brief with the fixup pass. They aren't summarized here until they've been checked.
llama.cpp is the serving stack today: it runs every checkpoint, quantizes any cache, and carries the MTP heads the dense 27Bs use for speed. Both engines do continuous batching and tool calling, so neither is a reason to switch on its own. vLLM stays in the campaign because the matched-weight comparisons check llama.cpp against a second implementation, and because throughput, latency, memory and tool-call reliability under my real load haven't been compared yet. A newer vLLM or CUDA toolkit may fix the Blackwell failures above, but that needs a retest, not a release note.
Hardware
I run an AI coding assistant on a desktop computer in my office. Here's the hardware, what each part does, and the limits I've run into.
A Gigabyte RTX 5090 runs the AI model. Its memory decides which models fit entirely on the graphics card.
Separate from graphics memory. Two 32 GB DDR5 sticks that load the models and run the software and tests around them.
Holds the downloaded model files, about 40 of them at 10–30 GB each, and every saved test result.
This is an air-cooled build: a three-fan card and a large air cooler on the processor, in a mesh case. Under sustained testing the graphics card reaches 89–91 °C and slows itself down to stay there. The speed numbers on this page are what this machine does under those conditions, which is why they're measured here rather than quoted from reviews. The power limit is stock (600 W), and the card drew about 380–400 W during test runs. A lower limit is the usual fix for heat; it isn't tested here.
The main constraint is graphics memory: 32 GB is the budget, and each setting on this page changes how much of it the model needs. System memory turned out to matter more than I expected. One piece of model-serving software peaked at 56.6 GB of it mid-run and the process was stopped after running out. That's one measurement on one configuration, not a minimum spec.
Full hardware specifications
| Part | Component | What it does here |
|---|---|---|
| Graphics card | Gigabyte RTX 5090 Gaming OC, 32 GB GDDR7SKU 805390 | Runs the model. Its 32 GB is the limit everything else is tuned against. |
| Processor | AMD Ryzen 9 9900X, 12 coresSKU 721209 | Ran the whole watermark pilot on idle cores while the graphics card was busy: 7 CPU-hours, no GPU time. |
| System memory | 64 GB DDR5-6400 CL34 (2×32) | Loads models and runs the test harness. Peak observed: 56.6 GB. |
| Storage | SanDisk WD_BLACK SN850X 4 TB NVMeSKU 753095 | Saved model files and run results. At 10–30 GB per model, 2 TB would already be tight. |
| Motherboard | MSI MPG X870E Carbon WiFiSKU 755892 | Connects the graphics card and the drive at full speed. |
| Power supply | 1200 W, 80+ Platinum, ATX 3.x | Headroom for a card that pulls hard in bursts. |
| CPU cooler | Noctua NH-D15 chromax.blackSKU 958470 | Air, not liquid. Keeps the processor quiet during CPU-only runs. |
| Case | Lian Li LANCOOL 217 InfinitySKU 919928 | Mesh front. The airflow the card is working against. |
Everything from Micro Center, bought together in June 2026. Where the specific model matters, the link goes to Micro Center's search for that SKU; memory and power supply are listed by spec, since any part meeting it would do.
Technical notes. The 5090 is a Blackwell card, which is why NVFP4 is available as a weight format at all (see Precision). It's also a factory-overclocked model, so its speed numbers run a little above a reference card's, which is one more reason every speed on this page is measured on this box. The board gives the card PCIe 5.0 x16 and the drive a direct NVMe link. The CPU-only watermark pilot is under Watermark.
How I test
Paired by item. Two configurations are only ever compared on the exact same questions; the difference is computed per question and summarized, which cancels question difficulty.
Intervals, not points. A gap is reported with a 95% bootstrap confidence interval over items. A ★ means the interval excludes zero. The registry's per-model quality intervals are a looser approximation (binomial per task, over the four shared tasks) used only to refuse false rankings.
Ties are ties. Models whose intervals overlap the leader's are marked ≈ tie rather than ordered.
No LLM judges. Code is executed against tests, instructions are checked programmatically, math answers are extracted. Nothing is graded by another model.
One writer. Every number on this page comes out of one generated data file; nothing is hand-edited. Dates on each Record strip say when and under what environment each number was measured.
Known limits. One machine, one run, greedy decoding at temperature 0, single answers rather than full coding tasks, a suite skewed toward code and instruction-following, thinking off throughout, and 18 baseline runs affected by the Aug 25–31 template bug. The run-to-run noise floor is being measured now (repeat runs) and will retroactively grade every borderline ★.