Qwen3.8-Flash-Next on one DGX Spark

sparkrun recipe: Qwen3.8-Flash-Next (NVFP4 / MXFP8, MTP ×4) on 1x DGX Spark.
vLLM with b12x GB10 kernels, PLE table paged from NVMe, OpenAI-compatible endpoint, quality-gated releases.

60 / 75 tok/s

TG c1
_{defaults (ISL/OSL 2048/512), v2.2.0
/ T=0, thinking off (coding), v2.2.0}

137 / 192 tok/s

TG aggregate c8
_{defaults (ISL/OSL 2048/512), v2.2.0
/ T=0, thinking off (coding), v2.2.0}

92/100

Hardmode
_{pass ≥ 88, T=0, thinking on}

7.1 s

TTFT, ISL 16K
_{c1, cold prefix}

_{262K context, max_num_seqs 8, MTP ×4, OpenAI-compatible API, quality-gated releases}

_{GPU power, GPU only (nvidia-smi), not wall power: idle 9.7 W, c1 24.2 W, c8 28.9 W; TG per GPU watt 2.49 (c1) and 4.73 (c8) tok/s/W at server defaults.}

_{TG cells: server defaults (ISL/OSL 2048/512, T=1.0, thinking on) first, T=0 coding (36 prompts, thinking off) second, both aggregate output tok/s. Means over 6 boots where measured; sd between boots in the details. Release v2.2.0.}

_{Measurement details}

| Metric | Mean | sd between boots | Boots | Workload | Sampling | Release (date) | Source | |---|--:|--:|--:|---|---|---|---| | TG c1, defaults | 60.4 tok/s | 3.22 | 6 | llama-benchy task mode, ISL/OSL 2048/512 | T=1.0, top-p 0.95, top-k 20, thinking on | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | TG c1, coding | 74.9 tok/s | 0.31 | 6 | 36 prompts, OSL ≤768, aggregate output tok/s per pass | T=0, thinking off | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | TG c1, coding, per-prompt median | 77.0 tok/s | – | 1 | 36 prompts (Python, C++, Rust, Go), OSL ≤768, median over prompts | T=0, thinking off | v2.1.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/) | | TG c1, coding, defaults, per-prompt median | 60.9 tok/s | – | 1 | 36 prompts, OSL ≤768; 198/216 requests hit the 768-token cap inside thinking | server defaults, thinking on | v2.1.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/) | | TG aggregate c8, defaults | 136.7 tok/s | 3.30 | 6 | llama-benchy task mode, ISL/OSL 2048/512 | T=1.0, top-p 0.95, top-k 20, thinking on | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | TG aggregate c8, coding | 192.0 tok/s | 1.02 | 6 | 36 prompts, OSL ≤768, aggregate output tok/s per pass | T=0, thinking off | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | TTFT, ISL 16K | 7.10 s | 0.02 | 4 | c1, cold prefix | – | v2.4.0 (2026-10-16) | [results](results/k103-1x-prefill-combo-ab-20261016-1206/) | | Hardmode | 92/100 | – | 1 | 88 multi-step tool-use scenarios; pass ≥ 88 | T=0, thinking on | v2.4.0 (2026-10-16) | [results](results/k103g-1x-combo-gate-20261016-1641/) | | GPU power, idle | 9.7 W | – | 6 | server up, no requests, 120 s | – | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | GPU power, TG c1 defaults | 24.2 W | – | 6 | power.draw, GPU only (nvidia-smi), not wall power, mean over the cell | T=1.0, top-p 0.95, top-k 20, thinking on | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | TG per GPU watt, c1 defaults | 2.49 tok/s/W | – | 6 | TG aggregate / GPU W | T=1.0, top-p 0.95, top-k 20, thinking on | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | GPU power, TG c8 defaults | 28.9 W | – | 6 | power.draw, GPU only (nvidia-smi), not wall power, mean over the cell | T=1.0, top-p 0.95, top-k 20, thinking on | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | TG per GPU watt, c8 defaults | 4.73 tok/s/W | – | 6 | TG aggregate / GPU W | T=1.0, top-p 0.95, top-k 20, thinking on | v2.2.0 (2026-10-10) | [results](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | Method and full tables: [docs/BENCHMARKS.md](/qwen3.8-flash-next-1x-dgx-spark/docs/BENCHMARKS.html).

Quick start

Requirements:

1x DGX Spark, sparkrun ≥ 0.3.6. <spark> = hostname or IP; --solo = single node.
~135 GB free on the internal NVMe: 97.7 GiB checkpoint + 2.6 GiB MXFP8 shard + 28.4 GB image.

sparkrun registry add https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark
sparkrun run qwen3.8-flash-next-1x-dgx-spark --hosts <spark> --solo

First run downloads ~135 GB (a fresh checkpoint download took 3 h 15 min at ~8.6 MB/s on my link; on a slow link, pre-fetch it, see below). Boot with model and image on disk: 256 s on the v2.4.0 image check, with the plan file and the compile cache coming from the image (171 s on v2.1.0 with a warm page cache); a cold-page-cache boot was not timed. sparkrun run returns before the engine is ready: poll http://<spark>:8000/health, then use http://<spark>:8000/v1, model qwen3.8-flash-next:

curl http://<spark>:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.8-flash-next", "messages": [{"role": "user", "content": "Hello"}]}'

No API key: keep it on a trusted network or put an authenticating proxy in front.

Multi-user setups. I set this recipe up for a single user. --enable-prompt-tokens-details (since v2.3.0) makes the usage field report prompt_tokens_details.cached_tokens exactly, so on a server shared by several users or tenants one user can tell whether someone else already sent the same prompt prefix. Prefix caching already leaks this through time to first token; the flag makes it exact. With several users or tenants, give each one its own random, secret cache_salt in the request body (the pinned vLLM accepts it on /v1/chat/completions, /v1/completions and /v1/responses; a gateway can set it per API key), so their prefix-cache entries never match each other. Otherwise, remove --enable-prompt-tokens-details from the recipe.

Download the model beforehand, check that it works, stop, upgrade

The recipe serves its own checkpoint, about 98 GB. A fresh download took about 3 h 15 min on my link (~8.6 MB/s). Download it on the Spark before the first boot, together with the two files of the base checkpoint that the recipe uses for PP (2.6 GiB): ```sh hf download ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE --revision 03f4a0570496bbe741189f4a2a7d611679e85406 hf download local-inference-lab/Qwen3.8-Flash-Next-NVFP4 --revision 7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd \ model-00035-of-00036.safetensors model.safetensors.index.json ``` `sparkrun run` returns before the engine is ready. Wait for health, then check that the reply has both reasoning and an answer: ```sh H=http://:8000 until [ "$(curl -s -o /dev/null -w '%{http_code}' $H/health)" = 200 ]; do sleep 10; done curl -s $H/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model":"qwen3.8-flash-next","max_tokens":1024, "messages":[{"role":"user","content":"Write a Python function that reverses a string."}]}' \ | python3 -c "import json,sys; m=json.load(sys.stdin)['choices'][0]['message'] print('reasoning:', len(m.get('reasoning') or m.get('reasoning_content') or ''), 'chars') print('content :', (m.get('content') or '')[:300])" ``` Empty `content` with long reasoning means `max_tokens` ran out inside thinking; garbled text means a checkpoint mismatch (see Troubleshooting under Details below). Optional long-context check (stdlib Python), expect `exact 20` at both depths: ```sh git clone https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark && cd qwen3.8-flash-next-1x-dgx-spark python3 scripts/fidelity_probe.py --base $H --model qwen3.8-flash-next --depths 8000,32000 ``` The server binds 0.0.0.0 with no API key: keep it on a trusted network or put a proxy with auth in front. Stop with `sparkrun stop --all`. To upgrade, run `sparkrun registry update qwen38-flashnext-1x` first: sparkrun caches registries and does not refresh them on `run`. > Moved here on 2026-10-08 from [qwen3.8-flash-next-dgx-spark-tp-2](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2), > which keeps a copy of this recipe so existing `sparkrun run` setups keep working. If both registries are added, > pick this one with `sparkrun run @qwen38-flashnext-1x/qwen3.8-flash-next-1x-dgx-spark --hosts --solo`. </details> ## At a glance | | | |---|---| | **Model** | [`ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE`](https://huggingface.co/ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE) @ `03f4a057`: NVFP4 experts, MXFP8 dense and attention, GDN projections NVFP4 for TG and MXFP8 for PP | | **MTP** | MTP ×4, probabilistic drafts, rejection sampling; drafter D3 refit on the served model's outputs (v2.1.0: D1) | | **Hardware** | 1x DGX Spark (GB10, 128 GB unified), checkpoint on local NVMe | | **Parallelism / memory** | TP=1; 26.8 GiB PLE table paged from NVMe (`VLLM_PLE_MMAP=1`), 14 GiB KV pool | | **Engine** | vLLM + b12x (NVFP4 MoE, MXFP8 linears, GDN, QSA), plan and compile caches baked into the image | | **Context** | 262,144 tokens (max_model_len) | | **Concurrent requests** | up to 8 (max_num_seqs) | | **KV cache** | 993,754 tokens, 3.79 concurrent 262K requests (v2.1.0 boot log) | | **API** | OpenAI-compatible, tool calling, reasoning on (`reasoning_effort` medium); `usage.prompt_tokens_details.cached_tokens` reports prefix-cache hits | | **Disk / start time** | ~135 GB / ~3 min warm (256 s on the v2.4.0 image check) | | **License** | Recipe Apache-2.0; weights Qwen Community License 1.0 | | Task | Section | |---|---| | TG, PP, TTFT, context depth | [Performance](#performance) | | Sizing: 1x vs TP=2 vs DP=2 | [1x vs 2x TP=2 vs 2x DP=2](#1x-vs-2x-tp2-vs-2x-dp2) | | Quality gate results | [Quality gate](#quality-gate) | | Raw data and provenance per number | [Every number and where it comes from](#every-number-and-where-it-comes-from) | | Pin or roll back a release | [Recipes](#details), [VERSIONS.md](/qwen3.8-flash-next-1x-dgx-spark/VERSIONS.html) | ## Performance Aggregate TG scales to c8 while per-request TG drops. TTFT grows with ISL on a cold prefix; multi-turn requests hit the prefix cache and run PP only on the new tokens.

TG aggregate tok/s vs concurrency c1-c16 per setup, ISL/OSL 2048/512, server defaults. End values labelled. TTFT in seconds vs ISL, c1, cold prefix, per setup. End values labelled.

_{Left: 1x Spark v2.1.0, 2x Spark TP=2 v2.0.0, 2x Spark DP=2 v2.1.0; llama-benchy task mode. Right: 1x Spark v2.4.0, 2x Spark TP=2 v2.0.0/v1.4.0; c1, cold prefix.}

_{Runs, method and raw data}

_{Left: llama-benchy task mode (agent coding turn), ISL/OSL 2048/512, T=1.0, top-p 0.95, top-k 20, thinking on, prefix caching. Aggregate = all output tokens / wall time, PP included; per request = TG rate of one stream after its first token.
1x Spark: release v2.1.0, 2026-10-08, llama-benchy task mode, mean ± sd over runs, one boot
2x Spark TP=2: release v2.0.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271, mean ± sd of 3 runs, one boot
2x Spark DP=2: release v2.1.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271, mean ± sd of 3 runs, one boot
Versions are numbered per setup. Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.

Right: c1, cold prefix (no cache hit). Each point: mean of 1 to 4 samples of one run (n per point in the CSV).
1x Spark: release v2.4.0, 2026-10-16, cold.py (k101)
2x Spark TP=2: release v2.0.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271; v1.4.0, 2026-10-07, fidelity_probe.py
1x Spark: not measured above 128K yet.
Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.}

PP tok/s vs ISL, c1, cold prefix, per setup. End values labelled. TG aggregate tok/s at context depth 0, 16K and 64K, c1/c4/c8. End values labelled.

_{Left: 1x Spark v2.4.0, 2x Spark TP=2 v2.0.0; c1, cold prefix. Right: 1x Spark v2.1.0; llama-benchy, ISL/OSL 2048/512 at depth.}

_{Runs, method and raw data}

_{Left: c1, cold prefix (no cache hit). Each point: mean of 1 to 4 samples of one run (n per point in the CSV).
1x Spark: release v2.4.0, 2026-10-16, cold.py (k101)
2x Spark TP=2: release v2.0.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271
Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.

Right: 1x Spark: release v2.1.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271, mean ± sd of 2 Sparks x 3 runs, one boot each (older release; shipped v2.4.0 not measured on this test yet). Server default sampling. Each point: ISL/OSL 2048/512 on top of the cached context, same harness as the concurrency chart. Concurrency levels measured at one depth only are left out.
Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.}

Per-request TG and the numbers behind the charts

TG per-request tok/s vs concurrency c1-c16 per setup, ISL/OSL 2048/512, server defaults. End values labelled.

_{1x Spark v2.1.0, 2x Spark TP=2 v2.0.0, 2x Spark DP=2 v2.1.0; llama-benchy task mode.}

_{Runs, method and raw data}

_{llama-benchy task mode (agent coding turn), ISL/OSL 2048/512, T=1.0, top-p 0.95, top-k 20, thinking on, prefix caching. Aggregate = all output tokens / wall time, PP included; per request = TG rate of one stream after its first token.
1x Spark: release v2.1.0, 2026-10-08, llama-benchy task mode, mean ± sd over runs, one boot
2x Spark TP=2: release v2.0.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271, mean ± sd of 3 runs, one boot
2x Spark DP=2: release v2.1.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271, mean ± sd of 3 runs, one boot
Versions are numbered per setup. Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.}

| Concurrency | TG tok/s, aggregate | TG tok/s, per request | Release, run | |--:|--:|--:|---| | 1 | 60.4 ± 3.2 | 56.9 ± 4.1 | [v2.2.0, 2026-10-10](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | | 8 | 136.7 ± 3.3 | 22.0 ± 0.5 | [v2.2.0, 2026-10-10](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/) | Release v2.2.0. llama-benchy task mode, ISL/OSL 2048/512, T=1.0, thinking on; mean ± sd, where sd is between runs or between boots as the CSV `stat` column says for each run.

## 1x vs 2x TP=2 vs 2x DP=2

1x vs 2x TP=2 vs 2x DP=2: TG coding c1, TG c1, TG aggregate c8, TTFT at ISL 16K, concurrent 262K requests in KV. Values labelled on the bars.

_{Shipped release of each setup where measured; the bracket names an older release.}

_{Runs, method and raw data}

_{TG, coding c1 (median, n=36, T=0): 1x v2.1.0, 2026-10-10, coding_probe.py, up to 768 tokens out; TP=2 v2.0.0, 2026-10-10, coding_probe.py, up to 768 tokens out; not measured: 2x Spark DP=2.
TG c1, defaults: 1x v2.2.0, 2026-10-10, llama-benchy task mode; TP=2 v2.0.0, 2026-10-10, llama-benchy task mode; DP=2 v2.1.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271.
TG, aggregate c8, defaults: 1x v2.2.0, 2026-10-10, llama-benchy task mode; TP=2 v2.0.0, 2026-10-10, llama-benchy task mode; DP=2 v2.1.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271.
TTFT, ISL 16K, cold prefix: 1x v2.4.0, 2026-10-16, cold.py (k101); TP=2 v2.0.0, 2026-10-10, llama-benchy 0.4.1.dev4+g0d4de4271; not measured: 2x Spark DP=2.
KV cache: concurrent 262K-token requests: 1x v2.1.0, 2026-10-08, serve log; TP=2 v2.0.0, 2026-10-09, serve log; DP=2 v2.1.0, 2026-10-08, serve log.
TG c1/c8: llama-benchy task mode, ISL/OSL 2048/512, server defaults. KV row: vLLM's 'Maximum concurrency for 262,144 tokens per request' at boot (DP=2: two replicas, one pool each).
Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.}

- **1x (this recipe):** max_num_seqs 8. - **2x TP=2:** [2x recipe](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2), higher per-request TG and a larger KV cache; fits low concurrency and long contexts. - **2x DP=2:** two 1x replicas behind the [router](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/main/tools/dp2/README.md). Lower wall time than TP=2 on all six agent-replay workloads:

Agent replay wall time in seconds, TP=2 vs DP=2, six workloads. Values labelled on the bars.

_{Agent replay, one boot per layout: two-Spark v1.5.0 (old name b1.6) (TP=2); DP=2 on one-Spark v2.1.0 (old name v3e) (DP=2).}

_{Runs, method and raw data}

_{All sessions start at once; each turn resends the full conversation (prefix cache on), tools on, T=0.6, thinking off.
2x Spark TP=2: two-Spark v1.5.0 (old name b1.6), 2026-10-08, agent replay (drive.py), one boot
2x Spark DP=2: DP=2 on one-Spark v2.1.0 (old name v3e), 2026-10-08, agent replay (drive.py), one boot
Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.}

## Quality gate | Check | Result | Criterion | |---|---|---| | Hardmode | **92/100** | score /100, pass ≥ 88 (88 scenarios), T=0, thinking on | | TC-45 | **100/100** | regression test (1 scenario, 2 pts, 5 trials) | | Fidelity | **20/20** | 20 needles retrieved via tool calls, ISL 8K to ~245K | | Stragglers | **none** | no stalled request at c7-c16 (max_num_seqs 8, rest queued) | Gate run: release v2.4.0, 2026-10-16. Every release passes this gate before promotion. Full gate tables: [docs/BENCHMARKS.md](/qwen3.8-flash-next-1x-dgx-spark/docs/BENCHMARKS.html). ## Release history

Release history: TG aggregate c8 and TG c1 tok/s, and hardmode score, per shipped release. Values labelled.

_{Each release's own promotion run and gate.}

_{Runs, method and raw data}

_{TG: llama-benchy task mode, ISL/OSL 2048/512, T=1.0, thinking on, from each release's own first run (usually its promotion A/B; the runs line below names each), unpaired across releases, so day-to-day drift is included; paired A/B per release in [VERSIONS.md](/qwen3.8-flash-next-1x-dgx-spark/VERSIONS.html).
TG runs: v1.0.0 2026-10-02, v1.1.0 2026-10-04, v1.2.0 2026-10-04, v1.3.0 2026-10-05, v2.0.0 2026-10-05 (llama-benchy task mode, mean ± sd over 3 runs, one boot); v2.1.0 2026-10-08 (llama-benchy task mode, mean ± sd over runs, one boot); v2.2.0 2026-10-10 (llama-benchy task mode, mean of 6 boots, sd between boots).
Hardmode: promotion gate of each release (b0: gate on the pinned checkpoint), v1.1.0 2026-10-04, v1.2.0 2026-10-04, v1.3.0 2026-10-05, v2.0.0 2026-10-05, v2.1.0 2026-10-07, v2.2.0 2026-10-09, v2.3.0 2026-10-13, v2.4.0 2026-10-16; v1.0.0: score not in the data.
Data: [docs/data/capability.csv](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), with the source file of every point.}

## Stack - **Weights:** NVFP4 experts, MXFP8 dense and attention. GDN projections in weight-only NVFP4 for TG, MXFP8 copy for calls of 41+ rows. - **PLE table:** 26.8 GiB n-gram table read through the page cache (`VLLM_PLE_MMAP=1`), WILLNEED before each TG gather, 50 ms NVMe keepalive; ~72 GiB of other weights stay resident. - **KV:** one GDN PP staging buffer shared by all 36 GDN layers (~8.8 GiB freed) and compact MTP draft records; 14 GiB KV pool. - **MTP:** 4 probabilistic drafts per step over a 131k-id draft vocabulary, rejection sampling (output distribution unchanged); drafter D3 refit on the served model's outputs (D1 in v2.1.0). - **Kernels:** b12x for GB10 (NVFP4 MoE, MXFP8 linears, 36 GDN layers, 12 QSA sparse-attention layers), autotuned plan cache and compile cache baked into the image. - **Serving defaults:** reasoning on at `reasoning_effort` medium, tool calling, max_model_len 262,144. ## Details

Every number and where it comes from: Capability table for one Spark, TP=2 and DP=2, with the run behind each number

### Every number and where it comes from The shipped releases were measured as a full grid in k76 on 2026-10-10 ([tp-2 #128](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/issues/128)); where a cell has no k76 value, the table shows the newest release that has one. TG rows are aggregate unless marked "per request"; sampling is the server default (temperature 1.0, thinking on) unless the run says otherwise. The charts and tables are written by `uv run scripts/make_charts.py` from [`docs/data/capability.csv`](/qwen3.8-flash-next-1x-dgx-spark/docs/data/capability.csv), where every point lists its raw file (`1x:` paths are in this repo, `2x:` paths in the [two-Spark repo](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2)). Full grids for every build: [docs/BENCHMARKS.md](/qwen3.8-flash-next-1x-dgx-spark/docs/BENCHMARKS.html). | | 1x Spark | 2x Spark TP=2 | 2x Spark DP=2 | |---|---|---|---| | TG c1, tok/s | 60.4 ^a | 88.7 ^b | 57.5 ^c | | TG c4, tok/s: per request / aggregate | 32.4 ^d / 111.4 ^d | 48.6 ^b / 156.6 ^b | 43.4 ^c / 137.5 ^c | | TG c8, tok/s: per request / aggregate | 22.0 ^d / 136.7 ^a | 33.2 ^b / 172.2 ^b | 32.5 ^c / 203.6 ^c | | TG c16, tok/s: per request / aggregate | > max_num_seqs 8 | 22.1 ^b / 231.3 ^b | 21.7 ^c / 240.9 ^c | | TG, coding c1 (36 prompts), tok/s median: T=0 / defaults | 77 ^e / 61 ^e | 119 ^f / 92 ^f | not measured | | Copy-heavy (MTP acceptance near 1), aggregate tok/s c1 / c4 / c8, mean | 80.8 ^g / 183.5 ^g / 274.1 ^g | 120.5 ^h / 303.5 ^h / 453.4 ^h | not measured | | PP tok/s, ISL 2K / 16K / 64K / 128K, c1 | 1,916 ⁱ / 2,306 ^j / 2,133 ^j / 1,872 ^j | 2,809 ^b / 2,913 ^b / 2,651 ^b / 2,362 ^b | not measured | | TTFT s, ISL 2K / 16K / 64K / 128K, cold prefix | 1.1 ⁱ / 7.1 ^j / 30.7 ^j / 70.0 ^j | 0.7 ^b / 5.6 ^b / 24.8 ^b / 55.5 ^b | not measured | | ITL p50, ms, c1 / c8 (MTP emits several tokens per step) | 19 ^k / 47 ^k | 13 ^l / 31 ^l | 19 ^m / 48 ^m | | Stream chunk gap p50, ms, c1 / c8 | 56 ⁿ / 143 ⁿ | 41 ^o / 90 ^o | not measured | | TG aggregate tok/s, depth 0 → 64K, c1 / c4 | 56.9 ^d → 55.6 ^d / 111.4 ^d → 98.4 ^d | 88.7 ^b → 82.8 ^b / 156.6 ^b → 104.3 ^b | 57.5 ^c → – / 137.5 ^c → – | | max_model_len | 262,144 (recipe) | 262,144 (recipe) | 262,144 (recipe) | | KV cache, tokens | 993,754 ^p | 3,525,633 ^q | 1,987,508 ^r | | Concurrent 262K requests in KV (vLLM count) | 3.79 ^s | 13.46 ^t | 2 × 3.79 ^u | | Requests that fit KV at 16K / 64K / 128K | 39.7 ^v / 14.1 ^v / 7.5 ^v | not measured | not measured | | Gate: hardmode / TC-45 / fidelity to ~245K / stragglers | 92 ^w / 100 ^w / 20/20 ^w / none, c7-c16 ^w | 90 ^x / 100 ^x / 20/20 ^x / none, c8-c16 ^x | 93 ^y / 100 ^y / 20/20 ^y / none, c5-c16 ^y | Releases and runs behind the numbers: - ^a one-Spark v2.2.0, 2026-10-10, k86-boot-noise-power, llama-benchy task mode ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k86-boot-noise-power-20261010-1704/)) - ^b two-Spark v2.0.0, 2026-10-10, k76-capability-matrix, llama-benchy 0.4.1.dev4+g0d4de4271 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/2x/)) - ^c DP=2 on one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, llama-benchy 0.4.1.dev4+g0d4de4271 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/dp2/)) - ^d one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, llama-benchy 0.4.1.dev4+g0d4de4271 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/)) - ^e one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, coding_probe.py, up to 768 tokens out ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/)) - ^f two-Spark v2.0.0, 2026-10-10, k76-capability-matrix, coding_probe.py, up to 768 tokens out ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/2x/)) - ^g one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, copy-heavy benchmark, 1,500 tokens out ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/)) - ^h two-Spark v2.0.0, 2026-10-10, k76-capability-matrix, copy-heavy benchmark, 1,500 tokens out ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/2x/)) - ⁱ one-Spark v2.4.0, 2026-10-16, promotion A/B, llama-benchy task mode ([files](results/k103-1x-prefill-combo-ab-20261016-1206/)) - ^j one-Spark v2.4.0, 2026-10-16, promotion A/B, cold.py (k101) ([files](results/k103-1x-prefill-combo-ab-20261016-1206/)) - ^k one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, llm-inference-bench 0.7.6 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/)) - ^l two-Spark v2.0.0, 2026-10-10, k76-capability-matrix, llm-inference-bench 0.7.6 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/2x/)) - ^m DP=2 on one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, llm-inference-bench 0.7.6 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/dp2/)) - ⁿ one-Spark v2.0.0 (old name v3d), 2026-10-05, depth and PP sweep, llm-inference-bench 0.7.6 ([files](results/tp1-v3d-20261005/bench/)) - ^o two-Spark v1.4.0 (old name b1.4), 2026-10-05, depth and PP sweep, llm-inference-bench 0.7.6 ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/lib-bench-20261005/tp2-b1.4/)) - ^p one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, vLLM serve log ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/)) - ^q two-Spark v2.0.0, 2026-10-10, k76-capability-matrix, vLLM serve log ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/2x/)) - ^r DP=2 on one-Spark v2.1.0 (old name v3e), 2026-10-10, k76-capability-matrix, vLLM serve log ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k76-capability-matrix-20261010-1059/)) - ^s one-Spark v2.1.0 (old name v3e), 2026-10-08, serve log ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/dp2-gate-k72-20261008-1135/)) - ^t two-Spark v2.0.0, 2026-10-09, serve log ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/k77-2x-ship-check-20261009-2149/check/)) - ^u DP=2 on one-Spark v2.1.0 (old name v3e), 2026-10-08, serve log ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/dp2-gate-k72-20261008-1135/)) - ^v one-Spark v1.3.0 (old name v3c), 2026-10-05, kv-capacity page accounting, same 14 GiB pool in 1× v2.0.0 and v2.1.0 ([files](results/tp1-v3c-20261005/)) - ^w one-Spark v2.4.0, 2026-10-16, promotion gate ([files](results/k103g-1x-combo-gate-20261016-1641/)) - ^x two-Spark v2.0.0, 2026-10-09, promotion gate ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/main/docs/BENCHMARKS.md)) - ^y DP=2 on one-Spark v2.1.0 (old name v3e), 2026-10-08, quality gate through the router ([files](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/tree/main/results/dp2-gate-k72-20261008-1135/))

Requirements: Hardware, disk, kernel, host settings, boot time, checkpoint

| | | |---|---| | **Hardware** | One DGX Spark (GB10, 128 GB unified); checkpoint on local NVMe (the PLE table is read through the page cache) | | **Launcher** | sparkrun ≥ 0.3.6 | | **Disk** | ~135 GB (97.7 GiB checkpoint + 2.6 GiB MXFP8 shard + 28.4 GB image) | | **Kernel** | No NCCL at TP=1, so the `7.0.0-1019-nvidia` NCCL regression of the two-Spark recipe does not apply | | **Boot** | ~3 min (171 s) with the compile cache from the image and a warm page cache, once the checkpoint is downloaded | | **Concurrency** | `max_num_seqs` 8, KV pool 14 GiB (993,754 tokens, 3.79x at 262,144) | Checkpoint: [`ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE`](https://huggingface.co/ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE) @ `03f4a057` (v2.2.0, drafter D3; v2.1.0: `16c9bd54`, drafter D1). It is the NVFP4 checkpoint [`local-inference-lab/Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/local-inference-lab/Qwen3.8-Flash-Next-NVFP4) @ `7c4f1bc1` with the GDN projections in weight-only NVFP4 and, since `16c9bd54`, a retrained MTP drafter. Coming from `16c9bd54`, only `model-00034-of-00036.safetensors` (4.5 GB) is new. Its model card covers what changed, how it was built and the license (Qwen Community License 1.0). The recipe also fetches shard 35 and the index of `7c4f1bc1` (2.6 GiB) for the MXFP8 PP copy. Image and video input are not tested.

Known limits: What is not measured or does not work yet

- The 14 GiB KV pool holds ~7.5 requests at 128K, so 8 concurrent 128K requests can wait for KV space. 8 × 64K fits. - On a first boot without a usable plan seed, it autotunes and compiles every kernel: ~30 min, with host MemAvailable down to 3.8 GiB for about a minute (earlyoom triggers at ~2.4 GiB). The images ship the plan seed and compile cache, so a normal first boot skips this. Close other memory-heavy work during the first boot. - In steady state there is ~14 GiB MemAvailable (lowest 13.3 GiB in the benchmark run), most of it PLE page cache. - The b12x plan seed and the compile cache are keyed by the model's snapshot path, so they match only the pinned checkpoint revision. Serving another revision or a local copy of the files autotunes and compiles on its first boot. - The recipe and its rollback serve the same checkpoint revision but keep one vLLM compile cache root each (`VLLM_CACHE_ROOT`): their torch AOT keys are the same and the artifacts are not. The first boot of each copies the AOT seed from its own image; the b12x plan cache is shared. - The image's Triton kernel cache was built on one Spark. On another host the first boot recompiles a few Triton kernels (~2 min once: 277 s instead of 147 s) and logs W-level TritonBundler tracebacks ("Cubin file saved by TritonBundler not found"); serving is not affected ([#39](https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/issues/39)). - Hardmode still fails a few multi-step scenarios (e.g. TC-30, TC-68, TC-74, TC-88) on every release. - Above 8 requests: a `max_num_seqs` 32 run (quality gate not run at that cap, v2.0.0) reached 189.3 tok/s aggregate TG at c32 (llama-benchy task mode, mean of 3 runs); a boot at `max_num_seqs` 16 was stopped by the 4 GiB memory guard ([high concurrency](/qwen3.8-flash-next-1x-dgx-spark/docs/BENCHMARKS.html#high-concurrency-max_num_seqs-32-2026-10-05)).

Recipes: The recipes in this repo and how to roll back

| Recipe | Release | Image | Use | |---|---|---|---| | [`qwen3.8-flash-next-1x-dgx-spark`](/qwen3.8-flash-next-1x-dgx-spark/recipes/qwen3.8-flash-next/qwen3.8-flash-next-1x-dgx-spark.yaml) | v2.4.0 | `k106s-20261017-a1ad1b69-5dad364d-10d0acc8-warm` (also `1x-v2.4.0`) | Default (checkpoint @ `03f4a057`, drafter D3, b12x a1ad1b69 with the HC down projection BF16 path, GDN prefill views) | | [`qwen3.8-flash-next-1x-dgx-spark-previous`](/qwen3.8-flash-next-1x-dgx-spark/recipes/qwen3.8-flash-next/qwen3.8-flash-next-1x-dgx-spark-previous.yaml) | v2.3.0 | `k95-20261013-b45fe867-5dad364d-warm` (also `1x-v2.3.0`) | Rollback (checkpoint @ `03f4a057`, drafter D3, b12x tile fix b45fe867) | Each recipe's header lists every change since the first release with its measured delta. Earlier recipes: [archive/recipes/](/qwen3.8-flash-next-1x-dgx-spark/archive/recipes/README.html). Renames: [recipes/RENAMES.md](/qwen3.8-flash-next-1x-dgx-spark/recipes/RENAMES.html).

Releases: Version names and what each release changed

Releases use semantic versions per repo; [VERSIONS.md](/qwen3.8-flash-next-1x-dgx-spark/VERSIONS.html) lists every shipped release with its old build name, date, what changed, its measured delta and its manifest (image digest, vLLM and b12x commits, checkpoint, drafter, plan seed). Each release's own A/B against the one before it is in the [release history](#release-history) and in [docs/BENCHMARKS.md](/qwen3.8-flash-next-1x-dgx-spark/docs/BENCHMARKS.html). Arms that were screened and not promoted: [results/README.md](results/README.md).

How I measure: How builds are screened, compared and promoted

- Candidate builds are screened with [Thunderdome](/qwen3.8-flash-next-1x-dgx-spark/docs/THUNDERDOME.html): one arm per Spark, control and candidate booted alternately, 2 passes, T=0 TG probes at c1/c4/c8 and at 16K context, plus llama-benchy at temperature 1.0. Noise band = the control's boot-to-boot spread, at least 1%. - To be promoted, a build must pass the gate, be faster beyond noise in at least one coding or counting cell, and be slower beyond noise in no cell at 1–4 requests ([`scripts/arm_verdict.py`](/qwen3.8-flash-next-1x-dgx-spark/scripts/arm_verdict.py)). - MTP acceptance per draft position is compared cell by cell as a numerics canary; logprob agreement against the previous build must sit within self-noise. - Copy-heavy and counting workloads accept nearly every draft and show the TG ceiling of MTP; the coding grid (llama-benchy task mode, thinking on) is the rate an agent sees. Every table states its workload. Index of every run and verdict: [results/README.md](results/README.md). The issue history up to 2026-10-08 is in the [tp-2 repo](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/issues?q=label%3Asingle-spark); open single-Spark issues are [here](https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/issues).

How it works: Parallelism, kernels, MTP, where the time goes

- The 26.8 GiB PLE n-gram table is read through the page cache from the checkpoint files (`VLLM_PLE_MMAP=1`), with a WILLNEED pass before each TG gather and a 50 ms NVMe keepalive, so ~72 GiB of other weights fit on one GB10. - One GDN PP staging buffer shared by all 36 GDN layers frees ~8.8 GiB, which goes to a 14 GiB KV pool, and compact MTP draft records cut each request's GDN state blocks from 185 to 37. - The GDN projections run TG from weight-only NVFP4 and switch to an MXFP8 copy for calls of 41+ rows. - b12x kernels cover NVFP4 MoE, MXFP8 linears, GDN (36 layers) and QSA sparse attention (12 layers), with an autotuned plan cache and the torch compile cache baked into each image ([`docker/b0-warm/`](docker/b0-warm/)). - MTP ×4 uses probabilistic drafts over a 131k-id draft vocabulary; rejection sampling keeps the output distribution unchanged. The drafter was retrained on the served model's own outputs: D1 in v2.1.0, D3 (more data, a longer run) in v2.2.0 ([`tools/mtp_refit/`](tools/mtp_refit/)). Every flag and environment variable, with the reason for it, is in the recipe header. Engine-wide configuration, image provenance and rejected experiments shared with the two-Spark recipe: [REFERENCE.md](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/main/docs/REFERENCE.md) and [ENGINEERING.md](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/main/docs/ENGINEERING.md) in the tp-2 repo.

Thinking effort: The default reasoning effort and how to change it per request

The recipe sets the server default to `reasoning_effort: medium`. On my DevOps task set (two Sparks), b1.2 at the template default `xhigh` passed 42.5% of checks with 23/42 runaway-thinking runs and a 246 s median per task; b1.4 at `medium` passed 95.9% with 0/42 runaway and 33 s. A request can still ask for `xhigh` or `low`; `"reasoning_effort": "none"` turns thinking off. For hard reasoning problems `xhigh` per request helps: on the hotel-lights check, v3c (old name) went from 21/32 at `medium` to 31/31 completed runs; allow a long client timeout, since most of those answers took over 30 minutes at 8 concurrent requests. Override table: [REFERENCE.md](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/main/docs/REFERENCE.md#thinking-effort).

Troubleshooting: Common errors and fixes

| Symptom | Fix | |---|---| | `/health` silent for minutes | Expected on a cold boot. `sparkrun logs qwen3.8-flash-next-1x-dgx-spark -f` | | OOM / earlyoom at first boot | Plan seed not used, so it autotunes; see Known limits above | | `Recipe ... matches multiple registries` | Both this registry and the tp-2 one are added; use `@qwen38-flashnext-1x/qwen3.8-flash-next-1x-dgx-spark` | | Garbled output | Checkpoint mismatch: the serve log must show the pinned `snapshots/03f4a057...` (or `16c9bd54...` for `-previous`) | | Empty `content`, long reasoning | `max_tokens` ran out during thinking; raise it or send `"reasoning_effort": "low"` | | Old release boots after an upgrade | `sparkrun registry update qwen38-flashnext-1x` | | Anything else | Try the `-previous` recipe, then open an issue with `sparkrun logs -a` | </details>

Docs: Where the rest of the documentation is

| | | |---|---| | [VERSIONS.md](/qwen3.8-flash-next-1x-dgx-spark/VERSIONS.html) | Release names, old build names, manifests | | [docs/BENCHMARKS.md](/qwen3.8-flash-next-1x-dgx-spark/docs/BENCHMARKS.html) | Full benchmark and quality tables for every build, method, history | | [docs/THUNDERDOME.md](/qwen3.8-flash-next-1x-dgx-spark/docs/THUNDERDOME.html) | The screen every candidate build runs | | [results/](results/README.md) | Every raw measurement and verdict | | [archive/](/qwen3.8-flash-next-1x-dgx-spark/archive/recipes/README.html) | Superseded recipes (not listed by sparkrun) | | [tools/mtp_refit/](tools/mtp_refit/) | MTP drafter refit pipeline behind v2.1.0 (D1) and v2.2.0 (D3) |

## Credits This build stands on these projects: - [local-inference-lab](https://github.com/local-inference-lab): the vLLM fork my branches start from, the b12x kernels (NVFP4 MoE, GDN, QSA) and the NVFP4 checkpoint this checkpoint is derived from. - [Qwen](https://huggingface.co/Qwen/Qwen3.8-Flash-Next): the base model, under the Qwen Community License 1.0. - [eugr](https://github.com/eugr): spark-vllm-docker (the image base), sparkrun (the launcher), and llama-benchy (the base of my benchmark fork). - [tonyd2wild](https://github.com/tonyd2wild): the bench_sweep counting harness behind the counting numbers. - [SeraphimSerapis](https://github.com/SeraphimSerapis): tool-eval-bench, which runs the hardmode and TC-45 quality gate. ## License Apache-2.0, see [`LICENSE`](/qwen3.8-flash-next-1x-dgx-spark/LICENSE). The vLLM overlays under `archive/mods/` keep their upstream Apache-2.0 headers. Model weights are not part of this repo and keep their own license (Qwen Community License 1.0; see the checkpoint's model card).

This site is open source. Improve this page.