Qwen3-8B on the Intel Core Ultra 5 226V NPU: 10.7 tok/s at 15 W and 24 dB of Noise

Intel's marketing calls Core Ultra an "AI computer," but the NPU usually has two real use cases: background blur in video calls and Copilot. The author of a Habr write-up decided to test something else: can the NPU run a full LLM, how many tokens per second does it produce, and how much energy does it burn? The test machine was a Ninkear L20 mini PC with an Intel Core Ultra 5 226V, 16 GB of LPDDR5X-8533 memory, and a working power range of 17-37 W. The answer: Qwen3-8B with 4-bit compression runs on the NPU at 10.7 tokens per second, drawing 15.4 W for the whole chip.

Three compute engines on one memory bus

Lunar Lake has three blocks for neural networks. The 8-core CPU (4 performance and 4 efficiency cores) is versatile but slow on neural networks. The Intel Arc 130V iGPU, built on the Xe2 architecture with hardware matrix multiplication, is the fastest for large models. The fourth-generation Intel AI Boost NPU is rated at 40 TOPS and computes INT8 and FP16 at low frequency and voltage, but it does not support dynamic shapes, takes a long time to compile, and does not support every operation. All three share the same RAM, so there is no need to copy a model between devices. But there is only one bus, and that becomes noticeable when they work at the same time.

Tokens, joules, and the NPU's honest defeat

The test: Qwen3-8B INT4, OpenVINO GenAI 2026.4, a 77-token prompt, a 256-token response, greedy generation, with the second run counted. Power was measured for the whole chip via the RAPL counter.

• Generation: NPU 10.7 tok/s, GPU Arc 130V 21.2 tok/s, CPU 10.0 tok/s
• First token: NPU 1.7 s, GPU 0.17 s, CPU 0.46 s
• Reading a 1329-token prompt: NPU 360 tok/s, GPU 117 tok/s, CPU 130 tok/s
• Chip power: NPU 15.4 W, GPU 20.6 W, CPU 16.0 W
• Energy per token: NPU 1.52 J, GPU 1.01 J, CPU 1.64 J

The author openly admits that the graphics are twice as fast and more efficient per token. On a 10 000-token task (15-20 pages), the NPU spends 15.6 minutes and 4.2 Wh; the GPU spends 7.9 minutes and 2.8 Wh. The 1.4 Wh difference is less than the charge of a single phone. The CPU matches the NPU in generation but reads a long prompt 12 times slower. By the author's estimate, the mini PC is 2-3 times more efficient per token than a computer with an RTX 5070 Ti.

Where the NPU is still needed

The NPU's strength is background work. It adds 1-2 W over idle on background tasks, and with the NPU at 99% load, the CPU is 7% busy and the graphics 2%. The fan does not spin up: 22-24 dB, versus 33-34 dB when the graphics and CPU are working. If two neural networks run on the same GPU, each slows down by half. If the LLM is moved to the NPU, a neural network on the GPU keeps 92% of its speed. All three blocks together draw 27.5 W, with the NPU at 75% load, the graphics at 92%, and the CPU at 62%.

Setup and pitfalls

The stack is built on OpenVINO: driver installation (the author used NPU driver 32.0.100.5540 and graphics driver 32.0.101.8331), Python 3.12, and the packages openvino, openvino-genai, nncf, and huggingface_hub. The NPU needs models with the -int4-cw-ov suffix: 4-bit symmetric weights with one scale per channel. In GenAI, it is enough to set MAX_PROMPT_LEN and MIN_RESPONSE_LEN. The first compilation is slow: 97 s for Qwen3-8B, 30-40 s for Depth Anything, 12 minutes for ESRGAN. With CACHE_DIR set, a repeat launch of Qwen3-8B takes 14.6 s.

Qwen3-TTS (1.7 billion parameters) turned out more interesting. The NPU does not accept the main part or the vocoder because of dynamic shapes and the KV-cache. But the author re-exported the code predictor, which is called 15 times per frame, to a static shape (input 1×16×2048), and it worked. Speed as RTF: everything on the GPU 1.73, GPU plus NPU 2.0-2.1, CPU plus NPU 6.7, CPU only 8.8-9.9. Intelligibility as measured by Whisper is 95-100%.

In Intel's plugins for GIMP 3, the NPU suits Super Resolution (sr_1033 in 46 ms, ESRGAN ×4 in 1.2 s) and deeplabv3 segmentation (19.5 ms). But SD 1.5 and SDXL Inpainting do not compile on it, and sseg-adas-0001 produces an incorrect result.

Practical takeaway

The NPU in Lunar Lake does not replace the graphics: for interactive chat, the Arc 130V is better. But for overnight batch photo processing, document summarization, book narration, and background agents, it lets the machine work almost silently, at just 1-2 W above idle, without occupying the GPU and CPU. The main condition: prepare models for fixed shapes and use the compilation cache.