AMD XDNA2 NPU Backend — Changelog

Phase 9 identified the 280 µs per-dispatch bo_a DMA (8 MB weight tile copy + sync) as a target for elimination. Phase 10 replaced the per-dispatch bo_a with a lazily-populated cache of pre-staged XRT buffer objects (WeightEntry::tile_bos). Weight tiles are copied and synced to device once on first use, then reused on every subsequent token with zero DMA cost.

Removed bo_a from XclbinSlot (saving ~112 MB pre-allocation). Added std::unordered_map<uint64_t, xrt::bo> tile_bos to WeightEntry, keyed by (slot_idx << 32) | tile_row_index. On cache miss (first token), a host_only BO is allocated via group_id(3), populated with the quantised weight tile, and synced once. On cache hit (all subsequent tokens), the pre-staged BO is passed directly to the kernel with no memcpy or sync. BOs are destroyed automatically when the weight cache entry is invalidated (M/K dimension change).

dispatch_tile signature changed from const int8_t * a_tile to xrt::bo & bo_a_staged.

The bo_a DMA cost (280 µs) is only 24% of the 1,155 µs per-dispatch total. Eliminating it saves ~197 ms per 704-dispatch token, but kernel compute (733 µs, 63%) dominates. Phase 10 is kept for code cleanliness and the ~112 MB memory saving, not for throughput.

Phase 9 measured ~1,155 µs per tile dispatch (K=4096). With 704 dispatches per token, host-side overhead (submission latency, redundant DMA syncs, per-dispatch bookkeeping) was hypothesised to be a significant fraction. Three approaches were investigated to reduce dispatch count or per-dispatch overhead:

A fourth approach — xrt::runlist batch dispatch — was researched but not tested. See section 5.

Built 8 new xclbins (4 K dimensions × 2 M values) using the mlir-aie whole_array example. All 4-col builds completed in 2–3 seconds each, confirming that xclbin compilation is not a bottleneck for experimentation.

Divisibility constraint: M must be a multiple of m × n_aie_rows = 64 × 4 = 256. Both M=4096 (16×256) and M=14336 (56×256) pass. No upper bound on M was encountered in the compiler.

All tests: Vulkan hidden (libggml-vulkan.so renamed), MIN_N=2, ngl=0, 3 repetitions, Meta-Llama-3.1-8B-Instruct-Q4_K_M. Backend reported XDNA only.

This conclusively disproves the Phase 9 projection that pre-staging + dispatch reduction could reach ~15 t/s. That projection assumed per-dispatch overhead (bo_b sync, kernel launch, readback) was the bottleneck. In reality:

Achieved throughput: 2.93 GOPS on int8 matmul (measured from 2.147B MACs / 733 µs kernel time).

Theoretical peak: 4 columns × 5 cores/column × 16 int8 lanes × 1.5 GHz = 480 GOPS.

Utilisation: 2.93 / 480 = 0.6%. The int8 matmul kernel achieves less than 1% of the AIE2P peak, indicating that the mlir-aie whole_array kernel is heavily bottlenecked by DMA-to-compute synchronisation, memory access patterns, or incomplete vectorisation for the int8 datapath.

XRT provides xrt::runlist (/usr/include/xrt/experimental/xrt_kernel.h) for batch dispatch: add N xrt::run objects, call execute() once, wait() once. A working example exists at mlir-aie/test/npu-xrt/add_one_two_runlist/test.cpp.

This was not tested because the Phase 11 tile-size experiments proved the bottleneck is kernel compute time, not submission overhead. Even if runlist eliminated all per-dispatch submission latency (~5–10 µs each × 704 = ~5 ms total), that represents <1% of the ~213 ms/token compute budget. The XRT docs also do not guarantee hardware pipelining — runs may execute serially on XDNA2.

NPU prefill (4.71 t/s) is now roughly equivalent to CPU-only performance (~4.3 t/s). The NPU adds no meaningful advantage over CPU for either prefill or decode with the current kernel.

The only path to meaningful NPU improvement is kernel-level optimisation — increasing AIE utilisation from 0.6% toward theoretical peak. This is outside the scope of the host-side llama.cpp backend and requires changes to the mlir-aie kernel code:

A full-project audit was completed alongside Phase 10, addressing 13 findings across 8 files. These were committed separately from Phase 10 to maintain clean git history:

Phase 11 produced no code changes. The bo_b sync deduplication was tested and reverted (no measurable benefit). The M=4096 and M=14336 xclbins remain in the mlir-aie build directory for future use but are not referenced by the backend.

A literature review (full survey) confirms our Phase 11 findings are consistent with independent research:

This project appears to be the furthest any independent developer has publicly reached with custom NPU workloads on XDNA2. No upstream llama.cpp NPU backend exists for any vendor. See the full research survey for 21 cited sources.