On 2026-10-05 NVIDIA published nvidia/GLM-5.3-Flash-FP4-Local — a second, very different NVFP4-family quant of the same GLM-5.3-Flash base, aimed explicitly at “local AI hardware” (2× DGX Spark and 2× RTX PRO 6000 Blackwell are the validated targets). It is not a refresh of the existing nvidia/GLM-5.3-Flash-NVFP4 checkpoint: it changes what is quantized, how the weights were produced, and — the part that matters operationally — which serving stack can load it. This note decodes the checkpoint from its configs and evaluates it against our 4-node DGX Spark switchless NVFP4 lane. For background on the wider GLM-5.3-Flash quant ecosystem, see the earlier five-quants comparison.

What actually changed

Two headline differences from the stock NVFP4 pack:

1. The experts were retrained, not just rounded. The stock NVFP4 is a post-training quantization (PTQ): experts quantized once from the source weights, everything else left BF16. FP4-Local uses quantization-aware distillation (QAD) — a BF16 teacher supplies probability and hidden-state targets while the student’s routed experts are quantized to NVFP4 and reconstructed each forward pass, with routers, shared experts, dense MLPs, and norms “adapted” to compensate. Training corpus: 500M tokens from the Nemotron post-training/pretraining collections, prompts only, repeated 3 epochs, modelopt v0.47.0.
2. The compressed-scales format. The card opens with the load-bearing warning: “the NVFP4 scale factors in this checkpoint are in a new lossless compression format, but upstream vLLM support for this format has not yet landed. You must use one of the containers provided below.” Verified from hf_quant_config.json: the 42 routed-expert entries carry weight_scale_encoding: csf_32x2_v1 — a compressed scale encoding nothing in upstream vLLM decodes today. Two auxiliary files ship with it: model-inputscales.safetensors (4.7 MB of activation scales) and tensors/native-QB.safetensors (753 MB, the native quantized-block payload for the custom loader).

The precision map, verified from the config

The stock NVFP4 checkpoint quantizes exactly one thing — routed experts (36,297 linears including the first-3 dense MLPs) — and keeps attention, shared experts, MTP, lm_head, embeddings, and the vision tower in BF16, via a 132-entry ignore list. FP4-Local inverts the philosophy: an empty exclude_modules list and a 494-entry quantized_layers map:

This is a whole-model quantization pass with QAD rescuing accuracy — a fundamentally different bet than the stock pack’s “only touch the experts, keep everything the lossless part BF16.”

Size: where the 26 GiB went

The card claims ~166 GiB of VRAM. The 26 GiB saving comes from moving attention, shared experts, dense MLPs, and lm_head off BF16 — and on our 4-node lane it would translate into KV headroom: ~6.4 GiB/rank freed is roughly one-third more KV than the current 20 GiB/rank pin (~3.25M fp8-KV tokens), enough to make the proven-but-gated 24 and 28 GiB/rank rungs comfortable. On the card’s own 2-node target the same headroom is what buys its “2M tokens of FP8 KV cache” claim.

Accuracy: real evidence, but not comparable to the stock NVFP4 table

The card publishes a same-harness BF16 A/B (the same evidence culture as the stock NVFP4’s table):

Two readings. First, the good: a −0.8 pp GPQA cost after quantizing the entire attention and dense path is a strong QAD result — the stock PTQ achieved noise-level deltas only by leaving those components in BF16. Second, the caveat: these numbers are not comparable to the stock NVFP4 card’s table (GPQA 0.9211, SciCode, MMMU Pro, AA-LCR, IFBench, Terminal Bench 2.1). Different benchmark set, different GPQA population (the FP4-Local card describes GPQA Diamond as 198 questions — the actual Diamond set size — while the NVFP4 card says 448), different tool-use suite. NVIDIA published no benchmark that scores both checkpoints. So “is FP4-Local more or less accurate than stock NVFP4?” is genuinely unmeasured; what is measured is that each is within ~1 pp of BF16 on its own suite.

One structural quality note for our lane: the stock pack’s corruption-free reputation rests partly on attention staying BF16 (the 0/0/0 U+FFFD receipts, tonyd2wild’s issue #23 analysis). FP4-Local’s attention is FP8 — the boot-gate suite (corruption probe, prefix scan, tool-call round-trip) would have to be re-earned from zero on any lane serving it.

The catch: it is a stack, not a checkpoint

This is the part the “new quantization targeting our environment” framing hides. FP4-Local cannot be served by a stock vLLM build at all. The card’s launch commands require:

• --quantization modelopt_mixed — the mixed-algorithm loader (NVFP4 + FP8-PB + MXFP8 + W4A16 in one checkpoint)
• --load-format b12x (Spark) / instanttensor (RTX) — custom loaders that understand the compressed scales and native-QB payload
• --attention-backend B12X --moe-backend b12x --linear-backend b12x — the B12X kernel family from the local-inference-lab vLLM fork, since folded into a FlashInfer fork
• Community containers: eugr/spark-vllm-b12x:latest (Spark) or ghcr.io/local-inference-lab/vllm (RTX)

The eugr/spark-vllm-docker changelog (2026-10-16) confirms the intent: the glm-5.3-flash recipe “now uses a new FP4-Local checkpoint from NVIDIA, replacing an older version hosted in local-inference-lab repository. As with Qwen, this checkpoint features compressed scales and is QAD quantized.” That recipe runs TP2 on 2 Sparks — max-model-len 500000, max-num-seqs 4, 8 GiB KV pin, and native MTP speculative decoding (num_speculative_tokens: 5, humming MoE backend) instead of a drafter model.

Two more implications fall out of that:

• Native MTP on Spark is a stack feature, not an FP4-Local feature. On our current stack (knapcio-family), the stock pack’s BF16 MTP head has no serving path — which is why the lane runs the DFlash2 drafter (itself licensed CC BY-NC-ND, no derivative re-encodes allowed). The B12X stack changes that for both checkpoints: a community 4-node recipe (detailed below) runs the stock NVFP4 checkpoint with native adaptive MTP (K1–K5, window 32), and FP4-Local additionally ships its MTP-layer experts quantized in-checkpoint. Shedding the DFlash2 license constraint is therefore a decision about adopting the B12X stack, not about which checkpoint is loaded.
• The transport story is different. The card’s Spark launch uses VLLM_ENABLE_ROCE_ALLREDUCE=1 plus vanilla NCCL env — the b12x stack’s own collective paths — not the four-PF patched-NCCL switchless contract our ring is built around.

Can it replace the weight in the NVFP4 switchless recipe? No — and it isn’t close

The short version: there is no weight-swap path. Our lane’s premise is the stock checkpoint on a stock-capable stack: knapcio’s vendored runner (glm53-roce:v11-b58f34ea), the stock ModelOpt NVFP4 loader, the four-PF switchless NCCL build, DFlash2, the 1M-context/20-GiB-KV invariant set, and the stock-nvfp4.env strip list — all of which assume the checkpoint’s exact quantization shape. FP4-Local fails the lane’s own adoption gate on its first criterion: it does not apply to the pinned image/revision (no upstream vLLM can decode csf_32x2_v1, and the B12X backends exist only in the fork). Swapping it in means replacing the engine, the loaders, the kernels, the speculative path, and the transport assumptions — i.e., building a third lane.

The author’s own verdict on 4× Spark clusters

The day after publication, the checkpoint’s author settled the 4-node question directly on the main GLM-5.3-Flash forum thread. Post #850 (eugr_nv, 2026-10-16): the checkpoint announcement, with the spark-vllm-docker recipe updated to match (#853). Then davedgd — who was already running GLM-5.3-Flash at TP4 across 4× GB10 Sparks on the B12X stack (glm53-b12x:candidate-2026-10-10, 1M context, 7 concurrent 1M-context sessions, up to 7 concurrent at 1M depth) with the stock NVFP4 checkpoint — asked whether to adopt the new quant at 4× (#856). The author’s reply (#858):

“For 4x Sparks there is not much benefit using this quant, so I’d just continue using the one you have.”

The reasoning, reconstructed from the thread plus the checkpoint math above:

1. FP4-Local solves a problem 4× Spark doesn’t have. Davedgd restates the design goal himself (#859): “a goal with this new quant was increasing the KV cache pool for 2x Spark by shrinking the weights while retaining as much quality as possible.” The whole precision shuffle (attention → FP8/MXFP8, lm_head → MXFP8) exists to wedge a 320B MoE plus a usable KV pool into two 128 GB boxes. Four nodes already have that memory by topology: stock NVFP4 at ~47.6 GiB/rank across 4×128 GB leaves ample KV headroom — our lane runs 1M context with a 20 GiB/rank pin and still has proven-but-unused 24/28 GiB rungs.
2. Quality is the currency, and at 4× it buys nothing. The stock NVFP4’s near-lossless accuracy profile comes from what it doesn’t quantize (BF16 attention, BF16 MTP, BF16 lm_head). FP4-Local spends exactly that to fit 2 nodes. At 4×, adopting it means paying a small accuracy cost (−0.8 pp GPQA vs BF16, attention in FP8) in exchange for memory the cluster doesn’t need — the trade is strictly worse than on 2 nodes, where it’s the price of fitting at all.
3. Throughput doesn’t rescue the case. The smaller active-path footprint (~35–40% fewer weight bytes per decode step at c1, computed from the configs) sounds like a speed win, but on a ring whose c1 decode is latency-bound the realized gain is the modest ~10–20% kind (the measured analogue: knapcio’s lossless8 conversion, same kind of change, bought ~10–15%), prefill is compute-bound and flat, and the B12X stack itself gives some of it back (VLLM_B12X_MOE_FP4_FORCE_A16=1 forces 16-bit compute on FP4 MoE paths; the RTX config even disables the MXFP8 lm_head). Meanwhile the checkpoint’s real throughput gift — KV headroom → higher concurrency — is the thing 4 nodes already have.
4. Davedgd’s 4-node run is the existence proof. His recipe (#856) achieves exactly what FP4-Local was built to deliver on 2 nodes — ~7 concurrent 1M-context sessions — using the stock checkpoint on the same B12X stack, with native adaptive MTP (K1–K5) instead of DFlash2. Nothing in FP4-Local’s feature list adds value on top of that config at 4 nodes.

FP4-Local is a genuinely interesting artifact — NVIDIA validating QAD + compressed scales as the new default for local Blackwell serving, with the accuracy table to back it — and if we were standing up a 2-node Spark pair from scratch today, its recipe would be the obvious pick over the older local-inference-lab checkpoint it replaces. But for the 4-node switchless lane it is not an upgrade path in its current state — and the checkpoint’s own author says the same thing. The memory win (~26 GiB, real as it is) buys KV capacity we already stretch via the rung ladder, the accuracy win is unproven against the checkpoint we serve, and the cost is abandoning an engine/transport combination that took a documented Phase-E effort to validate. The watch conditions that change the verdict: upstream vLLM landing csf_32x2_v1 + modelopt_mixed support (making the checkpoint loadable on stock-capable stacks), or a validated 4×-Spark GLM FP4-Local recipe appearing — now materially cheaper to attempt, since davedgd’s recipe proves the B12X stack at 4× TP4. Until one of those lands, this stays a record-and-monitor item, and any future adoption is a new lane — never an in-place weight swap.

Sources

• nvidia/GLM-5.3-Flash-FP4-Local — model card, config.json, hf_quant_config.json (sha cd3243fc, 2026-10-16 state; created 2026-10-05). Precision map and scale encoding verified from the config files, not just the card.
• nvidia/GLM-5.3-Flash-NVFP4 — model card (PTQ recipe nvfp4_experts_dense_mlp-kv_fp8_cast, BF16-vs-NVFP4 eval table) and hf_quant_config.json (sha da920bb0).
• NVIDIA blog: How Quantization-Aware Training Enables Low-Precision Accuracy Recovery — the QAD method.
• eugr/spark-vllm-docker — README/changelog (2026-10-05 switchless 4× ring support; 2026-10-07 B12X-into-FlashInfer; 2026-10-16 GLM FP4-Local recipe switch) and the glm-5.3-flash.yaml recipe (TP2/500K/4-seqs, native MTP n=5).
• local-inference-lab/vllm (B12X fork) and local-inference-lab/flashinfer — the B12X kernel stack.
• NVIDIA forum thread 381350 — posts #850/#853 (eugr_nv: FP4-Local announcement + recipe push), #856 (davedgd: his 4× GB10 TP4 B12X recipe on the stock NVFP4 checkpoint — 1M context, adaptive native MTP K1–K5, ~7 concurrent 1M sessions), #858 (eugr_nv: “For 4x Sparks there is not much benefit using this quant, so I’d just continue using the one you have”), #859 (davedgd: the quant’s design goal is 2-node KV). All read 2026-10-16 via the Discourse JSON API.
• Shard/file sizes from the Hugging Face blobs API (2026-10-16).