← Phase 11 · Literature review of NPU performance, AIE utilisation, and LLM inference across vendors. · GitHub
Key findings from independent research, academic papers, and community reports
Our Phase 11 discovery of 0.6% AIE utilisation is consistent with the academic literature for unoptimised kernels. Expert-optimised GEMM kernels on the same XDNA2 silicon achieve 38 TOPS (76% of peak) — a 127× improvement. The gap is entirely in the AIE kernel, not host-side dispatch.
No vendor’s NPU beats its companion GPU for LLM token generation. Intel, Qualcomm, Apple, and AMD all show the same structural mismatch: LLM decode is memory-bandwidth-bound, not compute-bound. TOPS ratings are irrelevant when every token must read the full model weights from shared DRAM.
This project appears to be the furthest any independent developer has publicly reached with custom NPU workloads on XDNA2. No other llama.cpp NPU backend exists for any vendor. Community sentiment is that NPU software stacks across all vendors remain immature.
What others have achieved on the same silicon
Ramezani et al. performed the most rigorous independent GEMM benchmark on both XDNA (Phoenix) and XDNA2 (Strix Point) NPUs, using optimised kernels with DMA-compute overlap and proper vectorisation.
| Precision | XDNA (10 TOPS peak) | XDNA2 (50 TOPS peak) | XDNA2 % of peak |
|---|---|---|---|
| int8 × int8 | 6.76 TOPS | 38.05 TOPS | 76.1% |
| int8 × int16 | 6.05 TOPS | 31.52 TOPS | 63.0% |
| int8 × int32 | 4.57 TOPS | 25.31 TOPS | 50.6% |
| bf16 × bf16 | 3.14 TOPS | 14.71 TOPS | 29.4% |
Key findings: L1 core memory utilisation 94–97%. Effective DRAM bandwidth ~50 GB/s (XDNA2). Without DMA-compute overlap, a 28% performance drop occurs. Optimal GEMM sizes were ~4K × 4K × 4K.
Our int8 × int32 pipeline maps to 25.31 TOPS achievable (50.6%) — not the full 38 TOPS of int8 × int8. Even so, that is ~8,600× our measured 2.93 GOPS. The kernel quality gap is enormous.
Direct comparison on Ryzen AI Max+ 395, 64 GB VRAM, using Ollama (CPU/GPU) vs FastFlowLM (NPU). Model: gpt-oss:20b.
| Device | Prompt eval (t/s) | Generation (t/s) |
|---|---|---|
| GPU (Vulkan) | 401.46 | 44.58 |
| CPU | 92.58 | 17.04 |
| NPU (FastFlowLM) | 18.20 | 10.94 |
Author’s conclusion: “The GPU was the fastest, the CPU showed higher performance than the NPU, and the NPU was the slowest.” GPU recommended for desktop LLM use. FastFlowLM is AMD-backed purpose-built NPU software — this represents the best-case NPU scenario.
| Model | Tokens/sec | NPU power |
|---|---|---|
| Llama 3.2 1B Q4_1 | 66 | <2W |
| Llama 3.2 3B | 28 | <2W |
| GPT-OSS 20B (MoE) | 19 | <2W |
The NPU’s sole advantage is power efficiency: ~2W vs ~25W for GPU. Over 10× better perf/watt claimed. For battery-powered, background inference this matters. For throughput, it does not.
Where unoptimised kernels sit in the literature
The 0.6% utilisation we measured is the expected result for a first-pass kernel without vectorisation, DMA double-buffering, and compute-communication overlap. It is not an anomaly — it is well-documented in the literature.
| Source | Hardware | Data type | Utilisation |
|---|---|---|---|
| AIE4ML (arXiv:2512.15946) | Versal VEK280, 304 tiles | i8 × i8 | 95.8% single-kernel |
| AIE4ML | VEK280, full array | i8 end-to-end | 82.2% (160 / 194 TOPS) |
| GAMA (arXiv:2504.09688) | Versal VEK280, 288 tiles | int8 | 85% (165 TOPS) |
| GAMA | VEK280 | bf16 | 86% (83 TFLOPS) |
| ARIES (FPGA 2025) | Versal VCK190 | INT8 | ~87% (45.94 TOPS) |
| Source | Hardware | Data type | Utilisation |
|---|---|---|---|
| MLIR-AIR (arXiv:2510.14871) | AIE | I16 | 78.7% |
| MLIR-AIR | AIE | I8 | 59.1% |
| FCCM 2024 (arXiv:2404.11066) | Versal VC1902 | int8 | 58.9–60.1% |
| MLIR-AIR | AIE | BF16 | 48.6% |
| Source | Hardware | Observation | Context |
|---|---|---|---|
| NPUEval (arXiv:2507.14403) | XDNA | ~10% mean vectorisation | LLM-generated AIE kernels |
| DPUV4E (arXiv:2506.11441) | Versal ACAP | <5% of theoretical | Small BERT MM layers |
| This project (Phase 11) | XDNA2 Strix Halo | 0.6% | mlir-aie whole_array example kernel |
The literature reveals a clear progression. Each step requires substantially more effort than the last:
| Optimisation level | Expected utilisation | Effort |
|---|---|---|
| Naive scalar kernel, single tile | 0.5–5% | Hours |
| Vectorised kernel, single tile | 10–30% | Days |
| Vectorised + double-buffered, single tile | 50–80% | Days–weeks |
| Expert-optimised, multi-tile GEMM | 80–98% | Weeks–months |
| Full array, end-to-end with overhead | 60–87% | Months (GAMA/ARIES level) |
DMA stalls, missing vectorisation, and compute-communication overlap
Five factors identified in the academic literature explain the gap between our 0.6% and the 76–98% achieved by expert kernels:
| # | Factor | Impact | Source |
|---|---|---|---|
| 1 | No DMA-compute overlap | 28% loss from this alone | Striking the Balance |
| 2 | Missing vectorisation | ~10% mean for auto-generated code | NPUEval |
| 3 | No double-buffering / ping-pong DMA | AIE stalls completely during data movement | GAMA, AIE4ML |
| 4 | Sub-optimal tile sizes for 64 KB SRAM | Wasted MACs on padding | DPUV4E |
| 5 | int8×int32 accumulation (our pipeline) | 50.6% peak vs 76.1% for int8×int8 | Striking the Balance |
The single biggest gain comes from DMA pre-fetch / double-buffering: overlap the next tile’s DMA transfer with the current tile’s compute. Without this, compute duty cycle drops to compute_time / (compute_time + dma_time), which can be <1% for bandwidth-limited transfers.
The GAMA paper quantifies compute-to-communication ratio (γ) ranging from 0.72 to 0.96. For int8×int32 (our pipeline), γ = 0.72 — meaning 28% of time is spent waiting for data even in the optimised case. Custom buffer placement recovers 12% of lost performance from memory stalls.
The structural mismatch is not AMD-specific
No vendor’s NPU beats its companion GPU for LLM token generation. This is a structural architectural mismatch, not a software maturity issue.
| Vendor | NPU | NPU decode | GPU decode | Source |
|---|---|---|---|---|
| AMD (this project) | XDNA2 (50 TOPS) | ~0.7 t/s | 44 t/s (Vulkan) | Phase 11 |
| AMD (FastFlowLM) | XDNA2 | 10.94 t/s | 44.58 t/s | 7shi / Zenn.dev |
| Intel | NPU 4 (48 TOPS) | ~2.9 t/s (Llama3-8B) | ~5.3 t/s | NITRO (arXiv:2412.11053) |
| Qualcomm | Hexagon (45 TOPS) | ~18–20 t/s | comparable | ASPLOS’25 |
| Apple | ANE (38 TOPS) | CPU beats ANE | 30–40 t/s (Metal) | Orion (arXiv:2603.06728) |
| Hailo | Hailo-10H | 6.9 t/s | 131.7 t/s (RTX 4050) | arXiv:2603.23640 |
Autoregressive LLM decode has fundamentally different characteristics from the workloads NPUs were designed for:
| LLM decode characteristic | NPU design assumption |
|---|---|
| Single token at a time (batch=1) | High batch parallelism |
| Variable sequence length | Fixed static shapes |
| Memory-bandwidth-bound | Compute-bound |
| Requires FP16/BF16 for LayerNorm, softmax | Optimised for INT8/INT4 |
| Dynamic KV cache growing per token | Static tensor shapes |
The memory bandwidth wall: During decode, the entire model weights must be read from DRAM for every single token. A 7B INT4 model is ~3.5 GB. At LPDDR5X-8533 bandwidth (~68 GB/s), the theoretical maximum is ~19 tok/s regardless of NPU compute capacity. The NPU shares DRAM bandwidth with the CPU and iGPU — it has no dedicated memory channel.
INT8 and INT4 symmetric quantisation compilation failed entirely on Intel NPU. GPU consistently 1.8–2.3× faster across all configurations. Intel’s own docs acknowledge decode is “mostly limited by DRAM bandwidth rather than compute capability.”
FP16 matmul on NPU is 159× slower than CPU INT8. Per-group quantisation (needed for activation outliers) incurs 10.7× overhead. Graph compilation takes 11+ seconds per prompt for variable-length inputs. NPU compiler lacks native RoPE, flash attention, dynamic KV caching.
ANE has 2.3 ms IOSurface round-trip overhead per dispatch. CPU baseline (283 tok/s on GPT-2 124M) outperforms ANE decode (170 tok/s). llama.cpp and MLX both ignore the ANE entirely and run on GPU through Metal.
What independent developers have experienced with XDNA NPUs
Across Reddit (r/LocalLLaMA, r/AMD), Phoronix, Hacker News, and developer blogs, the overwhelming sentiment is: “Great hardware, nonexistent software stack.”
Key observations from community reports:
| Source | Finding |
|---|---|
| r/LocalLLaMA | NPU limited to INT8 ONNX models through AMD’s pipeline. No general-purpose compute API. Multiple users report NPU sitting completely idle. |
| Phoronix | Kernel driver landed (Linux 6.11+) but userspace tooling essentially nonexistent. XRT described as “designed for FPGAs first, NPUs second.” |
| Framework Community | Hardware detection works but no practical inference path on Linux. Windows-only for LLM acceleration. |
| GitHub xdna-driver | Active issues about driver/toolchain version mismatches. Linux driver “lags about a year” behind AMD’s internal implementation. |
| AMD’s own forums | Support threads about Ryzen AI SDK installation failures. Setup process described as “fragile.” |
No upstream llama.cpp NPU backend exists for any vendor. A ggml-hsa branch for AMD was discussed in llama.cpp issue #14377 but auto-closed for inactivity. AMD has not contributed an NPU backend. The official AMD path is Windows-only ONNX Runtime GenAI.
This project’s approach — direct XRT dispatch with custom xclbin kernels on Linux — is essentially what no one outside AMD has successfully done for a production workload.
Kernel optimisation priorities from the literature
whole_array kernel may run scalar or under-vectorised. NPUEval shows ~10% mean vectorisation for auto-generated code.whole_array example is general-purpose. A kernel optimised for specific (M, K, N) shapes from LLaMA layers could achieve significantly higher utilisation.With DMA double-buffering and vectorisation (days–weeks of mlir-aie kernel work), the literature suggests 50–80% utilisation is achievable — translating to 12.5–20 TOPS for our int8×int32 pipeline. This would make NPU prefill competitive with Vulkan for small context. However, NPU decode would remain bandwidth-bound and uncompetitive.
Even with perfect kernel utilisation, NPU LLM decode cannot beat GPU decode. The 4 MB on-chip SRAM cannot hold model weights; every dispatch requires DRAM reads through shared LPDDR5 bandwidth. The NPU has no dedicated memory channel. This is a hardware constraint, not a software one.
Hardware specifications from technical deep-dives
| Feature | XDNA (Phoenix) | XDNA2 (Strix Point/Halo) |
|---|---|---|
| Array topology | 4×5 (20 tiles) | 4×8 (32 tiles) |
| MACs/tile/cycle (int8) | 128 | 256 (2×) |
| Peak INT8 TOPS | 10–16 | 50–55 |
| L2 SRAM | 2560 KB | 4096 KB |
| Tile local SRAM | 64 KB | 64 KB (unchanged) |
| Cascade bus | 384-bit | 512-bit |
| Concurrent contexts | 6 | 16 |
| BF16 / Block FP16 | BF16 only | BF16 + Block FP16 |
Per compute tile: 64 KB SRAM (4 banks × 16 KB). No caches — fixed-latency, software-managed. Dedicated DMA engines independent of compute.
Per memory tile (1 per column): 512 KB scratchpad, serves as L2 for 4 compute tiles in its column.
On-chip aggregate: ~800 GB/s SRAM bandwidth. 512-bit cascade bus at 80 GB/s.
Off-chip: Shared LPDDR5/5X with CPU and iGPU. No dedicated NPU memory channel. LPDDR5X-7500: ~120 GB/s shared.
Explicit data movement: The programmer must describe ALL data movement via circuit- or packet-switched routes. No automatic prefetching.
No caches, no stalling: The compiler must schedule all instructions to avoid hazards. Powerful for deterministic throughput but brutal for programmability.
Vectorisation is hard: NPUEval shows even GPT-4.1 achieves only 71.6% functional pass rate on AIE kernel compilation after 5 iterative attempts (29.4% on first try).
Papers, benchmarks, and community sources