Running Qwen3.8-27B on a Single RTX 5090

The hardware and memory budget

Qwen3.8-27B has 64 text layers: 48 Gated DeltaNet (GDN) recurrent layers and 16 full-attention layers, arranged in a repeating three-to-one pattern. Attention retains a growing KV cache; GDN carries a fixed-size recurrent state. Both must remain consistent during prefix reuse and speculative decoding, while sharing memory with the vision tower and MTP module.

I measured the complete serving process: which kernel launched, whether output checks passed, and how much memory remained after runtime allocations. The configured request limits above are scheduler settings, not evidence that eight requests can each use the maximum context simultaneously.

1. Calibrating and quantizing the full model

Compressing the text decoder alone left too little room for the BF16 MTP module, vision tower, and runtime state. The final procedure quantizes 401 eligible matrices in the text decoder and language-model head, plus 118 across MTP and vision.

Calibration data and evaluation isolation

Activation statistics determine which weight errors a data-aware quantizer penalizes. For this deployment, calibration needed to cover reasoning, tool use, prose, long documents, OCR, charts, and general images, using the same tokenizer and state transitions as inference.

The deterministic text collector draws from pinned sources: OpenThoughts3 and OpenMathInstruct for mathematical reasoning; OpenCodeReasoning and AgentTrove for code and tool use; FineMath and FineWeb-Edu for technical prose; UltraChat, CNN/DailyMail, and WikiText for general language; and PG-19 for long books. Conversations are rendered with the deployed chat template and tokenizer before assignment to length buckets, so the buckets reflect actual model inputs.

The candidate pool contains 97.32 million tokens. A weighted interleave across sources and lengths produces a ranked, 512-trajectory core with 12.22 million tokens. Pilot runs use prefixes of this ordering. The manifest distinguishes natural trajectories, deterministic windows, and packed samples, and records source revisions, row locations, text and token hashes, and packing recipes.

Collection is resumable. A fixed seed and source identifiers determine scan order; accepted samples are stored as raw little-endian token IDs to avoid later retokenization. A partial manifest records completed jobs so an interrupted run resumes without changing the pool.

Calibration uses training splits only. Normalized prompt and document hashes, fixed text chunks, and content-defined chunks are checked against the AIME and AA-LCR evaluation exclusion lists. Vision samples are also deduplicated by decoded RGB pixels, which catches identical images stored with different encodings.

Preserving state across long calibration sequences

Replaying all 97.32 million candidate tokens for each quantization iteration would be costly. The final pass selects twelve trajectories by reproducible hash ranking: four 32,768-token prefixes from code, math, and educational web data; four 64,000-token prefixes from reasoning, agents, WikiText, and a long AgentTrove slice; and four 128,000-token prefixes from reasoning, agents, and PG-19. The total is 899,072 input tokens.

Each trajectory starts at token zero and advances in 2,048-token chunks. Positions, attention KV, and all 48 GDN states persist across chunks; activation observations are sampled every 256 tokens. Calibration proceeds in eight-layer blocks, followed by the language-model head. Once a block is quantized, subsequent blocks receive its outputs, including the upstream quantization error they will encounter in deployment.

The vision pool contains 4,096 candidates from 36 training-set slices and a 2,048-image core. The core includes 1,600 document-heavy images—forms, receipts, charts, tables, diagrams, scene text, handwriting, and multilingual pages—and 448 general images for VQA, spatial reasoning, counting, captioning, and multi-image comparison. Sources include DocVQA, ChartQA, TextVQA, FUNSD, CORD, multilingual Wikipedia renders, VQAv2, CLEVR, and NLVR2. The approximately 78/22 split emphasizes OCR while retaining general visual coverage.

The collector reads Parquet sources by row group and indexed tar datasets by the selected member’s byte range. Images are decoded to RGB, checked for resolution and aspect ratio, and hashed as both source payloads and decoded pixels. Weighted interleaving produces nested cores of 256, 512, 1,024, and 2,048 images.

Calibration should reproduce the positions, recurrent states, modalities, and upstream quantization errors expected during inference.

The quantization objective and runtime constraints

For each text matrix, I compare round-to-nearest (RTN) with a BlockLDLQ update followed by ten coordinate-descent passes. Candidates minimize a local quadratic reconstruction objective under the runtime format constraints:

minimize_Q  tr((W - Q) H (W - Q)ᵀ)

subject to Q being exactly representable by the target
NVFP4 or FP8 runtime format and its block-scale layout.

Here H is an activation second-moment matrix. It gives greater weight to reconstruction errors along directions that occur more often or at larger magnitude in the calibration data.

BlockLDLQ compensates for rounding error by updating later columns using the activation statistics. Coordinate descent then refines the candidate. Scoring uses the exact packed representation consumed by the Blackwell kernels, including native NVFP4 or FP8 values, block scales, and layout.

The procedure is a hardware-constrained, second-order post-training quantizer operating in the model’s original coordinates. It shares the BlockLDLQ error-feedback idea used in QuIP. QuIP# additionally uses randomized Hadamard transforms and an E8-family lattice codebook. Supporting that approach in this SGLang/FlashInfer path would require matching transforms in the runtime or correctly fused rotations around the operators. The current implementation does not make that comparison.

Why lower reconstruction loss did not prevent repetition

On August 22, a quantized tail passed short smoke tests and improved local reconstruction loss, but repeated on a held-out trajectory of roughly 79K tokens. There was no out-of-memory event, GPU error, restart, or NaN. With MTP enabled, repetition began after 92 generated tokens on a cold run and 53 on a cache hit; with MTP disabled, it began after 46 and 136 tokens. A previously qualified tail ended naturally on the same input. The failure therefore persisted across both decoding and cache modes.

Layer-by-layer overlays localized the sensitivity to the final decoder blocks, immediately upstream of the model norm and language-model head. Recalibrating layer 63 with continuous long-context state improved all three local objectives below, but generation still failed:

The held-out rollout repeated an identical token 15 times from a cold start and 29 times after a cache hit. Local reconstruction loss measures agreement on sampled activations. It cannot by itself establish that an autoregressive rollout will remain stable after many successive predictions.

The first calibration pass had used 512 independent windows of 1,024 tokens, or 524,288 tokens in total. Every window reset positions, attention KV, and all 48 GDN states. Although the failing document was in the corpus, the observer never saw the model state reached after processing its first 79K tokens. This motivated the continuous calibration procedure above.

A separate, fixed 67K incident branch exposed an algebraic error before NVFP4 rounding. Qwen’s zero-centered RMSNorm stores w, but uses the gain 1 + w. Folding an AWQ channel scale s into the norm must preserve that effective gain:

runtime gain:  γ = 1 + w

wrong fold:    w′ = w / s
correct fold:  w′ = (1 + w) / s - 1

The old builder used the incorrect fold. After undoing the coordinate scale, the effective gain became s + w. With a representative scale near 0.48 and a stored weight near zero, both gate and up activations fell to roughly 48% of their intended amplitude. SwiGLU multiplies these branches, coupling their errors.

The packed gate/up weights were bit-for-bit identical in this comparison. Correcting only the norm fold changed a capped, periodic output into a sequence that ended naturally. The test isolated a function-changing reparameterization error, so verifying the algebra became a prerequisite for evaluating quantization loss.

For the rebuilt checkpoint, I removed the AWQ fold, returned to the official weight coordinates, and replayed the twelve continuous trajectories through eight successive layer blocks and the language-model head. Quantized upstream blocks remained active while later activation statistics were collected.

Checkpoint selection now combines local and end-to-end checks. RTN, BlockLDLQ, and coordinate descent propose packed candidates; joint-SwiGLU and downstream checks can reject them. A final-logit check measures KL divergence, normalized error, and top-1 changes. Held-out rollouts cover cold and cache-hit requests with MTP on and off, recording EOS, periodic suffixes, dominant-token fraction, and repeated-token runs. All 64 gate/up pairs retained RTN + RTN because the optimized alternatives did not improve on that baseline in the deployed path.

On the fixed incident branch, the rebuilt checkpoint ended after 361 generated tokens with 152 unique tokens and a maximum identical-token run of two. On a second held-out branch of 66,911 input tokens, it ended after 344 tokens with 183 unique tokens and the same maximum run. These cases became regression tests; passing them establishes recovery on those traces, not comprehensive model-quality parity.

Long-context calibration requires processing the preceding tokens while preserving positions, attention KV, and recurrent state.

Memory recovered from MTP and vision

MTP fully connected and attention projections use FP8 weights and activations; its MLP uses W4A16 NVFP4. Vision QKV and output projections use FP8, while vision MLP and merger matrices use W4A16 NVFP4. Convolution, positional embeddings, normalization parameters, and biases remain BF16. The exported tensors are checked against the runtime’s packed representation bit for bit.

The full-context speculative configuration had exceeded the budget by roughly 0.93 GiB. Recovering 1.056 GiB from MTP and vision closed that gap. Checkpoint checks then covered paired development prompts, OCR and multimodal inputs, AIME reasoning, long-context rollouts, and the full serving profile. These serving checks are separate from the controlled FP8/NVFP4 task-quality comparison discussed below.

2. Reading packed NVFP4 cache data during prefill

The first NVFP4 cache path stored pages compactly but expanded the full history before attention. A 128K-token prefill took 36.134 seconds, including the cost of reading, dequantizing, and materializing that history in a larger temporary representation.

The replacement SM120 attention kernel consumes page-64 NVFP4 data directly. It pads queries within a bounded workspace and handles the final partial chunk without adding cache tokens. A 2,048-token chunk balanced launch overhead, workspace size, and register pressure in this configuration.

At 128K, the native path was 2.103× faster than the materializing NVFP4 path; an irregular 109,658-token prompt completed in 13.698 seconds. Verifying kernel selection was necessary: an early eligibility check accepted only flattened K/V tensors, while the server supplied a three-dimensional layout and silently fell back. I therefore paired timing measurements with launch audits and output hashes.

3. Speculative decoding with recurrent-state replay

Qwen3.8 includes MTP weights for speculative decoding. An initial depth-three NativeMTP run with a 128K-token prompt and 1,024 generated tokens reached 143.7 tokens/s, with an average acceptance of 3.98 out of four tokens. This was 2.24× the target-only baseline of 64.2 tokens/s. However, the 256K configuration ran out of memory, and repeated requests did not consistently produce identical outputs.

The target and draft caches consumed 34,816 bytes per token in that configuration. Speculative verification also retained roughly half a gigabyte of intermediate GDN states for rollback. I replaced those snapshots with ReplaySSM, a 0.015 GiB ring of recurrence inputs. After verification selects the accepted tokens, replaying their inputs commits the corresponding recurrent state without retaining every intermediate snapshot.

With CUDA graphs and ReplaySSM, the main MTP serving profile reached 166.6 tokens/s for a 128K-token prompt followed by 512 generated tokens. A separate 240K-prompt, 20K-output run sustained 125.4 decode tokens/s after a 44.6-second prefill. Output lengths and acceptance rates differ across these workloads, so the rates should be read with their individual conditions. The paired table above isolates the eager-versus-graph comparison on matching outputs.

4. Serving vision requests within the same memory budget

CPU offload for the 27-block vision encoder reduced useful GPU memory less than expected once transfers and workspaces were included, and increased multimodal latency. Quantizing the tower recovered 0.552 GiB in the payload comparison above while keeping its computation on the GPU.

The serving path uses four changes:

1. Keep image decoding and resizing on CPU, but run the quantized tower on GPU.
2. Serialize items in a multi-image request rather than multiplying peak workspaces.
3. Temporarily unmap the 128 MiB FlashInfer workspace through CUDA virtual memory while preserving its graph address.
4. Use a dedicated SM120 vision-attention kernel instead of a generic attention fallback.

In a separate stress configuration containing eight 128K text sequences, the optimized 16-image workload fell from 2.47 to 1.69 seconds, returned all image outputs, and left no persistent visible allocation. This tests vision-workspace coexistence under that configuration; it is not a measurement of aggregate text-generation throughput.

5. Cache reuse and the cost of reduced precision

Attention KV and GDN checkpoints have different storage requirements and error behavior. Across the 16 attention layers, an FP8 KV cache requires:

16 layers × 2 (K,V) × 4 KV heads × 256 dims × 1 byte
= 32,768 bytes/token = 32 KiB/token

NVFP4 reduces the stored attention cache to roughly 18 KiB per token, or about 20 KiB including the shared FP8 workspace. The cache uses unit global scales for K and V, with E4M3 scales for blocks of 16 values. This saves substantial capacity at the native context length, but its effect on outputs requires a separate comparison.

GDN state has a nearly fixed footprint, yet summarizes the prefix processed by 48 layers. Reusing a prefix requires restoring both the attention pages and the matching recurrent state. Restoring only one produces an inconsistent model history.

Caching both reduced an exact 128K repeat from 17.07 to 0.396 seconds; a request with a 95%-shared prefix took 3.179 seconds. I then tested how the precision of the stored GDN checkpoint affected the resumed computation.

With the NVFP4 attention path held fixed, I replayed the same AgentTrove token stream while changing only the stored recurrent checkpoint. Host-side BF16 reproduced the cold run exactly at 4K and 32K. Block-8 INT8 was close at 4K but diverged substantially at 32K, even though the active recurrence used BF16 after restoration.

Controlled FP8 versus NVFP4 comparison

Four early probes gave an incomplete picture of NVFP4 cache fidelity: three matched the FP8 top token and one differed. I expanded this to a controlled comparison of the two target-only serving paths. The larger test failed the predefined agreement thresholds.

Both profiles used the same checkpoint, tokenizer, input token IDs, BF16 GDN state, context and memory-pool sizes, page size, chunking, concurrency, scheduler, and graph settings. One used FP8 target KV; the other used NVFP4 with its required native prefill selector. MTP was disabled. The experiment therefore compares two deployable cache paths, including their kernel differences, rather than isolating storage precision alone.

The test contained 64 deterministic next-token states: four base histories, four branches, and four context lengths—2K, 32K, 128K, and 240K. Sixteen cases began cold; 48 were planned as shared-prefix branches, of which telemetry confirmed 32 actual cache hits. Before the comparison, eight states were replayed after restarting each profile. All 16 same-format replays returned identical log probabilities, supporting repeatability on those checks.

FP8 and NVFP4 selected the same top token in 45 of 64 states. The FP8 winner remained in the NVFP4 top five in 63 cases, and mean top-10 overlap was 0.773. Nevertheless, 19 top-token changes can alter autoregressive continuations. Seven of the 36 states with an FP8 winning margin above 0.25 nats also changed their top token.

The 70.3% match rate has a Wilson 95% interval of 58.2–80.1%. Mean common-top-100 log-probability error was 0.192, with a bootstrap 95% interval of 0.174–0.212 and p95 of 0.322. Agreement varied across lengths: 62.5% at 2K, 75% at 32K, 62.5% at 128K, and 81.25% at 240K. This small set of related histories does not establish a monotonic relation between context length and error or a general task-accuracy estimate.

The predefined thresholds included at least 63/64 top-1 matches, all 64 FP8 winners retained in the NVFP4 top five, preservation of every high-margin decision, and mean top-10 overlap of at least 0.80. The observed results failed these criteria. These are decision-agreement diagnostics, not downstream accuracy scores; they do not support a claim of no accuracy loss.

Latency was similar through 128K, but the NVFP4 path took 67.766 seconds at 240K versus 40.414 seconds for FP8. An observed 240K prefix hit took 0.714 versus 0.181 seconds. Thus, in this implementation, lower cache memory did not translate into lower latency at the longest tested context. This target-only profile differs from the MTP serving measurements earlier in the note.

The evaluation plan required the deterministic checks to pass before paired AIME, AA-LCR, and vision runs. Because they failed, those downstream comparisons were not completed; the finished FP8 tasks provide only a baseline. A separate MTP3 attempt stopped when the FP8 XQA verification path encountered an uninitialized target mask on its first reasoning case. Speculative KV parity therefore remains unresolved.

The deployment demonstrates a capacity benefit from NVFP4, while the controlled comparison identifies output changes and a long-context latency cost. Attention pages and recurrent checkpoints need separate fidelity checks, and neither capacity nor speed alone establishes model quality.

Practical lessons

• Calibrate continuous state. Preserve positions, attention KV, and recurrence across long sequences, then check held-out generation as well as local reconstruction loss.
• Verify the executed path. Audit kernel launches and pair performance measurements with correctness checks.
• Account for runtime allocations. Include transfers, workspaces, CUDA graphs, and multimodal peaks when evaluating memory savings.
• Measure speculative state overhead. Replaying accepted recurrence inputs reduced the snapshot cost while preserving outputs in the paired checks.
• Report capacity, speed, and quality separately. The NVFP4 cache saved memory but failed the FP8 decision-agreement criteria and was slower at 240K in the target-only comparison.

An optimization is useful only with its measurement conditions: the workload, runtime path, memory budget, and correctness checks that accompany the reported speed.

Reproducing the measurements

The companion repository, poseidonchan/5090-Qwen3.8-27B, contains the pinned upstream base, SGLang and FlashInfer overlay, launch profiles, calibration and benchmark scripts, and a mapping from reported measurements to commands and expected outputs.

Begin by verifying model and checkpoint identity, then run a short correctness case and confirm the launched kernel. Next measure cold prefill, compare output hashes, and test prefix reuse before increasing context length and concurrency. Keep the MTP performance profile and target-only cache A/B profile separate when reproducing their results.

Open questions and next experiments

• Measure tail latency under networked, multi-user request traces.
• Repair the XQA verification path and investigate cache-fidelity failures before attempting paired MTP3, AIME, AA-LCR, and vision comparisons.
• Build transform-aware kernels for a proper randomized-Hadamard comparison against the original-coordinate fixed-grid PTQ used here.
• Expand multilingual, vision, and tool-use evaluation beyond the current regression cases.
• Separate the SM120 changes into maintainable patches suitable for upstream integration.

References

1. Qwen3.8 official repository and Qwen3.8-27B model card.
2. NVIDIA GeForce RTX 5090 specifications.
3. Chee et al., QuIP: 2-Bit Quantization of Large Language Models With Guarantees, 2023.
4. Tseng et al., QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, 2024.
5. FlashInfer attention-kernel documentation.
6. SGLang, Apache-2.0 serving framework.