AMD XDNA2 NPU Backend — Phase 12: Micro-Tile Optimization

Micro-Tile Optimization — 128×128×16 Decode Tiles

AIE micro-tile sweep · +23% NPU decode · SRAM budget analysis · instruction count as dominant factor · 2026-03-30

+23%

1 — Background

Phase 11 established that the NPU is compute-bound at 0.6% AIE utilisation with 64×64×64 micro-tiles (the per-core tile size within the AIE array). While Phases 10–11 focused on host-side dispatch overhead, Phase 12 investigates the AIE-side tile geometry — the m, k, n parameters passed to the mlir-aie whole_array builder that control how much work each AIE core does per DMA transfer.

The hypothesis: larger micro-tiles reduce the ratio of DMA setup to compute within the AIE array, potentially improving the 0.6% utilisation figure. The constraint: each AIE2 core has only 64 KB of data memory (SRAM), limiting how large the tiles can be.

2 — SRAM Budget Model

Each AIE2P core has 65,536 bytes of local data memory. The matmul kernel requires three tile buffers (A, B, C) with double-buffering (ping-pong), plus stack space:

SRAM constraint formula

2 × (m×k + k×n + m×n×4) + 3328 (stack) ≤ 65536

Where m, k, n are the per-core micro-tile dimensions. The ×4 on the C buffer accounts for int32 accumulation (4 bytes vs 1 byte for int8 A/B).

Additional constraints from the array geometry and xclbin compilation:

Constraint	Source	Effect
`n` is capped at 16	N=64, n_aie_cols=4, n%8==0	n ∈ {8, 16}
`m` must divide 512	M=2048, n_aie_rows=4	m ∈ {8, 16, 32, 64, 128, 256, 512}
`k` must divide all K dims	GCD(2048, 4096, 5632, 14336) = 512	k ∈ {8, 16, 32, 64, 128, 256, 512}

3 — Tile Sweep: 7 Configurations Tested

All tests used standalone whole_array.exe (the mlir-aie matrix multiplication example compiled for the decode path: M=2048, K=4096, N=64, n_aie_cols=4). Each configuration was run 10 times.

Round 1 — Varying m and k

Config	Micro-tile (m×k×n)	SRAM usage	Avg GFLOPS	Peak GFLOPS	vs Baseline
Baseline	64×64×16	33%	1,495	1,598	—
A	128×64×16	58%	1,315	1,347	−12%
B	64×128×16	49%	1,320	1,364	−12%
C	128×128×16	86%	1,833	2,135	+23%

Round 2 — Exploring k and n extremes

Config	Micro-tile (m×k×n)	SRAM usage	Avg GFLOPS	Peak GFLOPS	vs Baseline
D	64×256×16	80%	1,308	1,385	−13%
E	32×512×16	86%	1,438	1,532	−4%
F (bank-aware)	128×128×16	86%	1,835	2,061	+23%

4 — Why 128×128×16 Wins

Key Insight

The dominant factor is NPU instruction count (runtime sequence length), not compute-per-byte ratio. Config C halves the instruction count (1,292 vs 2,580 instructions) by doubling both m and k. The 4× larger compute volume per tile overcomes the additional DMA overhead per transfer.

Configs A (m=128 only) and B (k=128 only) each double one dimension but increase instructions to ~3,600–4,200 due to unbalanced DMA patterns. Config C doubles both dimensions symmetrically, yielding a compact instruction sequence. The SRAM at 86% is near-optimal — large enough for efficiency but within the 64 KB budget.

Bank-aware allocation (Config F, --alloc-scheme=basic-sequential) showed no meaningful difference vs the default allocator. The AIE2P memory controller handles bank conflicts transparently at this tile size.

5 — End-to-End Decode Benchmark

Result

LLaMA-3 8B Q8_0, NPU-only (GGML_VK_DISABLE=1), ngl=0, 32 tokens generated, 3 repetitions:

Configuration	Micro-tile	Decode throughput	Delta
Baseline	64×64×16	1.14 ± 0.09 t/s	—
Phase 12	128×128×16	1.40 ± 0.16 t/s	+23%

Model: Meta-Llama-3-8B-Instruct-Q8_0.gguf (8.0 GB). Q8_0 was chosen over Q4_K_M to minimise double-quantisation noise in the int8 pipeline (Q8→int8 is near-lossless vs Q4→F32→int8).

The +23% matches the standalone GFLOPS improvement exactly, confirming the tile geometry change propagates cleanly through the full llama.cpp dispatch path.

6 — Production Xclbin Builds

New decode xclbins were built for all 4 K dimensions used by LLaMA-3 8B:

K	File	Micro-tile	Build command
2048	`k2048_n64_decode.xclbin`	128×128×16	M=2048 K=2048 N=64 m=128 k=128 n=16
4096	`k4096_n64_decode.xclbin`	128×128×16	M=2048 K=4096 N=64 m=128 k=128 n=16
5632	`k5632_n64_decode.xclbin`	128×128×16	M=2048 K=5632 N=64 m=128 k=128 n=16
14336	`k14336_n64_decode.xclbin`	128×128×16	M=2048 K=14336 N=64 m=128 k=128 n=16

All built with: dtype_in=i8 dtype_out=i32 AIE_TARGET=aie2p n_aie_cols=4 via the whole_array example in mlir-aie.

A stale k12288_n64_decode.xclbin (old 64×64×16 tiles, not used by LLaMA-3 8B) was removed during cleanup.

7 — NPU Mode Toggle

With both prefill (4-col, TILE_N=256) and decode (128×128×16, TILE_N=64) xclbins now available, a shell-level mode toggle was added to switch between configurations without editing .zshrc:

Command	Effect
`npu-mode prefill`	Sources `~/.npu-prefill.env` — 4-col xclbins, TILE_N=256, MIN_N=2
`npu-mode decode`	Sources `~/.npu-decode.env` — 128×128×16 xclbins, TILE_N=64, MIN_N unset

Default mode at shell startup is prefill. The toggle was implemented as two static .env files sourced by a npu-mode shell function, avoiding the find_slot interference that would occur if both xclbin sets were loaded simultaneously (decode slots would steal small-batch prefill dispatches due to tile_n best-fit logic).

8 — Peano/Clang Limitation: opt_perf=1 Is a No-Op

Discovery

The mlir-aie build system exposes OPT_PERF_ENABLED which maps to AIE_LOOP_FLATTEN → chess_flatten_loop. Under the Peano/clang compiler (__AIECC__), AIE_LOOP_FLATTEN is defined as empty. The Chess compiler (xchesscc) that supports this pragma is proprietary and requires a commercial Vitis license.

This means the opt_perf=1 flag in the mlir-aie build has no effect on the generated code when using the open-source Peano compiler. Any performance difference from opt_perf requires the proprietary Chess compiler.

9 — Updated Performance Model

Backend	Prefill pp512	Decode	Status
XDNA NPU (Phase 12, decode mode)	—	1.40 t/s (+23% vs Phase 11)	Still compute-bound, but improved tile efficiency
XDNA NPU (Phase 10, prefill mode)	4.71 t/s	~0.7 t/s	Unchanged (different xclbin set)
Vulkan iGPU (ngl=99)	~530 t/s	44.09 t/s	31× faster decode than Phase 12 NPU

Phase 12 narrows the NPU-vs-Vulkan decode gap from 63× to 31×. Still not competitive, but the first Phase since Phase 6 to achieve a measurable throughput improvement from a tile geometry change — confirming that micro-tile tuning is a viable optimisation axis for the AIE kernel.

10 — Implications for Future Work

What Phase 12 teaches us

Micro-tile geometry matters more than host-side dispatch count. A 2× increase in both m and k (same total SRAM) yielded +23%, while Phase 11’s 7× dispatch reduction yielded 0%. The AIE kernel’s internal DMA-to-compute ratio is the bottleneck, not the XRT submission path.

This opens a specific optimisation path: kernel-level tile tuning within the mlir-aie ObjectFIFO pipeline. The 128×128×16 tile is optimal for the current SRAM budget, but the kernel itself may not be fully utilising the larger tiles (instruction count halved, but GFLOPS only improved 23%, not 2×). Profiling with --trace xclbins would reveal per-core stall cycles and confirm whether DMA latency or instruction scheduling is the remaining bottleneck.

The 86% SRAM utilisation also leaves little room for further tile growth without algorithmic changes (e.g., accumulation in the register file instead of SRAM, or single-buffered tiles for read-only data).

11 — Conclusion

Finding

128×128×16 is the optimal decode micro-tile for the mlir-aie whole_array kernel on AIE2P with 64 KB SRAM per core. It achieves +23% decode throughput (1.14 → 1.40 t/s) by halving the NPU instruction sequence length while staying within the SRAM budget at 86% utilisation. This is the first pure tile-geometry improvement since Phase 6 and confirms that micro-tile tuning is a more productive optimisation axis than host-side dispatch reduction.