AMD XDNA2 NPU Backend — Phase 9 Changelog

Root Cause Investigation & Phase 6 Restoration

llama-bench ngl default · dispatch timing · Phase 7 misattribution corrected · Phase 6 prefill restored · 2026-03-28

Correction

1 — Summary: What Was Investigated

Phase 7 claimed 43.84 t/s NPU decode. Phase 8 accepted this as the baseline and investigated why it could not be exceeded. Phase 9 restarted with fresh eyes after a regression was detected: current NPU decode measured at 0.65 t/s.

The investigation traced the regression to its root cause, corrected the Phase 7 record, and restored the Phase 6 NPU prefill baseline.

2 — Root Cause: llama-bench ngl Default

Misattribution

llama-bench defaults to ngl=99 (all layers on GPU) when no -ngl flag is provided. Phase 7's bench command (-p 0 -n 64 -ub 1 -r 1) did not specify -ngl, so the benchmark silently used Vulkan for all decode. The 43.84 t/s result was Vulkan decode, not NPU decode.

Confirmation: running the exact Phase 7 command today produces 44.09 ± 0.01 t/s with ngl=99, backend=Vulkan,XDNA in the output — same number, same backend.

3 — Why the GGML_VK_DISABLE=1 Confirmation Was Wrong

Invalid Isolation Test

Phase 7 docs claimed GGML_VK_DISABLE=1 left decode at 42.90 t/s (unchanged), proving Vulkan contributed nothing. In Phase 9 testing, GGML_VK_DISABLE=1 produced identical results to without it — 527 t/s prefill, ~44 t/s decode. The env var does not disable the Vulkan backend in this build. The confirmation test was not a valid isolation.

4 — Physical Limits of NPU Decode

Actual NPU decode via XDNA was measured with timing instrumentation in dispatch_tile:

Stage	K=4096 tile	K=14336 tile
memcpy host→bo_a + bo_b	280 µs	1100 µs
bo.sync to NPU (DMA)	134 µs	250 µs
kernel (run.wait)	733 µs	~3000 µs
bo.sync from NPU	5 µs	5 µs
Total per dispatch	~1155 µs	~4400 µs

Per token: ~704 dispatches × 1155 µs = 813 ms/token → max ~1.2 t/s physical ceiling.

For 44 t/s (22 ms/token), each dispatch would need ~32 µs. The memcpy alone (8 MB bo_a) takes 280 µs at available memory bandwidth. NPU decode at 44 t/s is physically impossible with the current dispatch architecture.

Measured NPU decode: 0.65 t/s — consistent with the physics.

5 — Phase 7's Secondary Regression: Prefill Broken

Secondary Regression

Phase 7 also broke Phase 6 NPU prefill by:

Replacing all xclbin slots with decode-optimised kernels (TILE_N=64 decode xclbins replacing TILE_N=256 prefill xclbins)
Setting GGML_XDNA_MIN_N=1 GGML_XDNA_MAX_N=1 permanently in .zshrc, blocking all N>1 ops from XDNA

Preserved

The original Phase 6 4-col prefill xclbins (~/xclbin-4col/) were preserved on disk and never deleted.

6 — What Phase 7 Actually Achieved (Corrected Record)

Claim	Reality
NPU decode 43.84 t/s	Vulkan decode. NPU decode is 0.65 t/s.
`GGML_VK_DISABLE` confirms NPU	VK_DISABLE does not disable Vulkan in this build.
Decode +11.7× vs CPU	This was Vulkan vs CPU, not NPU vs CPU.
Vulkan prefill 930 t/s	Real, confirmed, intact.

7 — What Was Restored in Phase 9

Restored

Reverted .zshrc to Phase 6 config: 4-col TILE_N=256 xclbins from ~/xclbin-4col/, MIN_N=2
Removed Phase 7 dispatch timing instrumentation from dispatch_tile (was running two chrono::now() calls + floating-point accumulation on every hot dispatch)

Phase 6 Prefill Performance — Verified Restored

Test	Phase 6 measured	Phase 9 restored
pp38	16.0 t/s	11.29 t/s
pp160	22.0 t/s	18.37 t/s
pp512	13.7 t/s	12.22 t/s

Note: Phase 6 numbers were also measured without strict XDNA isolation (Vulkan may have assisted). Today's numbers are from clean env-i tests forcing XDNA via MAX_N. The practical baseline is ~12–18 t/s NPU prefill depending on prompt length.

8 — Corrected System Baseline

Backend	Prefill pp512	Decode
XDNA NPU 4-col (restored)	12.22 t/s	0.65 t/s (physics-limited)
Vulkan iGPU (ngl=99)	~530 t/s	44.09 t/s
CPU only (ngl=0, no XDNA)	—	3.76 t/s

9 — Phase 10 Outlook

Next Target

NPU decode to be viable (~15 t/s) requires pre-loading weight tiles (bo_a) into NPU BOs at model init — eliminating the 8 MB DMA copy per dispatch.

Estimated dispatch overhead with pre-staged bo_a: bo_b sync (~23 µs) + kernel (~67 µs) + bo_c sync (~5 µs) ≈ 95 µs per dispatch → 704 × 95 µs = 67 ms/token → ~15 t/s.

Still slower than Vulkan's 44 t/s, but achieves the original goal: inference on dedicated XDNA2 silicon without loading the iGPU. At 15 t/s the NPU becomes useful for sustained decode workloads where iGPU power draw is a concern.