Every checkpoint benchmarked on this machine: decode speed (log scale) vs mean quality (unweighted mean over shared tasks). The dense 27Bs lead quality, the 30B-class MoEs lead speed.

The full registry, and which permutations each checkpoint supports: an MTP draft head is a property of the checkpoint; a vLLM twin exists only where the same weights are published as an HF checkpoint; KV columns show which cache quantizations have measured cells so far. Click a column header to sort.

Eight tasks, chosen to cover what a coding-assistant server actually does — and frozen at run start, because a suite that shifts mid-campaign can't support paired comparisons. Item counts matter: they set how small a gap each task can resolve.

Scoring is automatic end-to-end: code tasks run the generated code against tests; instruction tasks check constraints programmatically; math extracts the final answer. No LLM judges anywhere — a deliberate choice, since a judge model would itself be a confound when comparing engines and quantizations.

Same family, only bit-width changes: the line stays flat while VRAM moves 5–10 GB. no detectable in-family difference between Q4_XL and Q8 on these tests. The f16 controls agree.

The exception: NVFP4 shows three starred deficits — two instruction-following (IFEval −4.1, IFBench −5.3 points) and one math reasoning (GSM8K −1.4). Large enough to investigate before the cutover; the replicate tier grades them against the run-to-run noise floor.

Every clean gap estimate as one dot — the distribution behind the verdict. Hover a dot for its model, task, CI and n.

The payoff: q8_0 saves ~1–3.8 GB, q4_0 up to ~5.9 GB of measured peak VRAM.

MTP-on vs MTP-off at identical weights: every delta inside ±0.01 — consistent with distribution-preserving verification (theory does not bound finite-sample deltas, so this is evidence, not proof).

Speed at depth {0,2,4} is measuring right now (tier 4b, in progress).

All 23 models: decode speed (log) vs mean quality — the unweighted mean of each model's benchmark scores over its shared task set (internal workload check excluded). Two clusters among these tested checkpoints — the dense models land slow-and-higher, the MoEs fast-and-slightly-lower. This supports picking a model from this set; it does not isolate architecture as the cause (training data, scale and tuning all differ).

Every quality number so far was taken one request at a time. Under batching, kernel shapes and cache sharing change the arithmetic, so the same prompt can legitimately produce different tokens. Throughput at np∈{1,2,4,8} is already measured (see the explorer); quality under 8-way load never has been — that measurement runs right after the replicates.

Detection increased with watermark strength and with text length, and every generation is visible: the control cloud sits at z≈0 (0/30 detections), δ=2 straddles the threshold (16/30 at 300 tokens), δ=4 clears it for 29/30 — and the spread within each dose is as informative as the means. (One pilot, one small model — staging evidence, not the study.)

The engine is a knob like any other — the same weights can serve faster or slower, and in principle differently, depending on who runs them. The trap in comparing engines is comparing files instead: llama.cpp serves GGUF, vLLM serves HF checkpoints, and two files of "the same model" usually aren't. This run's design: for a handful of parity rows, the GGUF was converted losslessly from the exact HF checkpoint vLLM serves, so an f16 gap is purely the stack; the 27B Q8-vs-FP8 row adds one controlled variable (8-bit format) on top.

Three findings, all discovered by trying rather than reading release notes. One: vLLM 0.27.1 routes hybrid-architecture prefill through a JIT compiler (FlashInfer) that cannot build for this GPU's compute capability with the CUDA toolkit the machine has — which makes the NVFP4 twin unservable on vLLM here, full stop. Two: a checkpoint that pins FP8 KV in its own config silently selects an incompatible attention backend unless the launcher detects it and overrides — a class of failure where the config is honored and the server still dies. Three: at 32 GB, the f16 parity rows are the only place both stacks can hold identical weights; everywhere else the honest comparison is "stack + format", not "stack".

The parity quality gaps themselves (f16 rows, Q8-vs-FP8 row) live in the run ledger and land in the decision brief with the fixup pass — not summarized here until they're checked.