local-llm
Original source

We Unlocked NVFP4 on DGX Spark — and It’s 20% Faster Than AWQ

Summary

Avarok team unlocked NVFP4 on DGX Spark (SM 12.1/GB10) which was broken or no faster than AWQ on all shipping software stacks.

The Problem

  • Vanilla vLLM on Spark: 1.1 tok/s (broken — missing PTX instruction caused catastrophic fallback)
  • NVIDIA’s official vLLM image: ~36 tok/s (same as AWQ, no advantage)
  • TensorRT-LLM: 29.6 tok/s

Their Fixes (Docker: avarok/dgx-vllm-nvfp4-kernel:v22)

  1. Software E2M1 Conversion — patched CUTLASS and FlashInfer to use software bit-manipulation instead of missing cvt.rn.satfinite.e2m1x2.f32 PTX instruction
  2. Enabled Marlin MoE Backend (W4A16 dequantization better for SM 12.1)
  3. SM 12.1 capability routing patches
  4. MTP Speculative Decoding — patched NVIDIA’s exclusion of NVFP4 from MTP code path

Results

  • NVFP4 now 15-19% faster than AWQ (same image)
  • With MTP: 1.7-2.9x faster than AWQ
  • Peak: 111.9 tok/s from 80B model on single desktop GPU

Upstream PRs

  • vllm-project/vllm PR #37059 — “fix(moe): detect and handle unquantized MoE weights in NVFP4 checkpoints”
  • Custom repo: Avarok-Cybersecurity/dgx-vllm

Has Latest vLLM Shipped These Fixes?

Partially yes, but not all:

Already merged into vLLM main:

  • PR #38423 (Mar 2026): Critical bugfix for NVFP4 on Spark/RTX50 — fixed cudaErrorIllegalInstruction
  • PR #40082 (May 2026): FlashInfer b12x MoE and FP4 GEMM kernels for SM120/SM121
  • PR #43332 (Jun 2026): W4A16 NVFP4 in FlashInfer B12x experts
  • PR #35568 (May 2026): Fix SM121 exclusion from Marlin/CUTLASS FP8 paths

Still OPEN / not merged:

  • PR #31740: The comprehensive “Add SM121/GB10 DGX Spark support” PR (device detection, full platform support)
  • PR #38484: Add SM121 to published build targets
  • PR #41738: Add SM120 for NVFP4 tensor core support (Docker build)

Limitation: NVFP4 KV cache only works on SM100/SM103 (GB200/GB202), NOT SM120/SM121.