Skip to main content

TECH VEDA

Corporate on-site training - Submit proposal Pick your modulesSharpen your kernel skills: deep dives, drivers, Yocto, CVEs, careers — updated daily. Read the blog →
Edge AI

Did It Run on the NPU? Tracing One Inference on an Embedded Linux Board

One inference on an embedded Linux board, traced through its file mapping, memory, threads, device node and driver on i.MX 8M Plus and RK3588.

Did It Run on the NPU? Tracing One Inference on an Embedded Linux Board

To the Linux kernel, one inference is not a special event. It is a process with a file mapping for the model, anonymous memory for the tensors, a few threads, and, when an accelerator is used, an open device file, a stream of ioctls and a train of interrupts. Each of these can be read from /proc, /sys and strace. This article shows what to read on a CPU-only run, on an NXP i.MX 8M Plus with the VX delegate, and on a Rockchip RK3588 with the RKNN runtime.

You have been handed a board and a .tflite file and asked to make it run. The runtime prints an inference time, and that is usually all anyone looks at. It does not tell you whether the model ran on the CPU or the NPU, which kernel driver was involved, how much memory the model holds, or why the first run took seconds when the rest take milliseconds. This article follows one inference through the process, the memory map, the device node and the driver, so you can tell what actually ran where on your own board.

What one inference is, from the kernel’s side

Every inference runtime on Linux, whether TensorFlow Lite, ONNX Runtime or a vendor SDK, reduces to the same four kernel objects.

  • A file mapping. The model file is opened and, in TensorFlow Lite’s case, mapped read-only into the process. Its pages are file-backed and count as RssFile.
  • Anonymous memory. Input, output and intermediate tensors live in memory the runtime allocates, and some runtimes also repack the weights into their own layout. This counts as RssAnon.
  • Threads. A CPU backend splits work across a thread pool. You see each thread under /proc/<pid>/task.
  • A device file, ioctls and interrupts. These appear only when an accelerator is in use. The delegate’s userspace library opens a node under /dev, submits jobs with ioctl(), and the kernel driver takes an interrupt when each job completes.

The fourth item is the one that answers “did it run on the NPU”. If the process has no accelerator device open, no inference ran on an accelerator, whatever the log says.

A CPU-only inference you can capture on any Linux machine

Start without an accelerator, so you know the baseline. The capture below comes from an x86-64 virtual machine with two CPUs running Linux 6.18.44, using ai-edge-litert 2.3.0 from PyPI and the quantised MobileNet V1 model from TensorFlow’s model archive. The script loads the model with two threads, calls allocate_tensors(), and then calls invoke() in a loop:

import sys, time, numpy as np
from ai_edge_litert.interpreter import Interpreter
it = Interpreter(model_path=sys.argv[1], num_threads=2)
it.allocate_tensors()
inp = it.get_input_details()[0]
x = np.zeros(inp['shape'], dtype=inp['dtype'])
while True:
    it.set_tensor(inp['index'], x)
    it.invoke()

With the loop running as PID 541, the model file is mapped once, read-only and shared, for its whole length:

raghu@techveda.org:~$ grep tflite /proc/541/maps
7fede6600000-7fede6a15000 r--s 00000000 fe:00 819370                     /opt/edge/mobilenet_v1_1.0_224_quant.tflite
raghu@techveda.org:~$ ls -l /proc/541/fd
total 0
lr-x------ 1 root root 64 Nov 18 05:15 0 -> /dev/null
l-wx------ 1 root root 64 Nov 18 05:15 1 -> /dev/null
l-wx------ 1 root root 64 Nov 18 05:15 2 -> /dev/null
lr-x------ 1 root root 64 Nov 18 05:15 3 -> /opt/edge/mobilenet_v1_1.0_224_quant.tflite
raghu@techveda.org:~$ cat /proc/541/task/*/comm
python
python
raghu@techveda.org:~$ grep -E 'RssAnon|RssFile|Threads' /proc/541/status
RssAnon:	   21244 kB
RssFile:	   37628 kB
Threads:	2

Read it item by item. The mapping is 0x415000 bytes, the 4,276,352-byte model rounded up to a page. File descriptor 3 stays open for as long as the interpreter exists. There are two threads: with two threads configured, the runtime added one worker to the main thread. The script also printed /proc/self/status at each stage in an earlier run: RssAnon rose from 15,032 kB to 21,536 kB across allocate_tensors(), the call that also logged Created TensorFlow Lite XNNPACK delegate for CPU. That 6.5 MB is the tensor arena and the CPU backend’s working data, held in anonymous memory beside the mapped file.

Now look at what the kernel is asked to do while one inference after another runs. Attach strace in counting mode to another run of the same program, PID 514, and interrupt it after two seconds:

raghu@techveda.org:~$ strace -f -c -p 514 -o sc.txt
strace: Process 514 attached with 2 threads
strace: Process 516 detached
strace: Process 514 detached
raghu@techveda.org:~$ cat sc.txt
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.019843           3      6293       386 futex
------ ----------- ----------- --------- --------- ----------------
100.00    0.019843           3      6293       386 total

Hundreds of inferences ran in that window, and the only system call was futex, the two threads waking and waiting for each other. A CPU inference makes no read() of the model, because it is mapped, and no ioctl(), because no driver is involved. This is the baseline signature of one inference on the CPU.

What a delegate adds: the i.MX 8M Plus and the VX delegate

On NXP’s BSP, the i.MX Machine Learning User’s Guide (UG10166, revision LF6.18.37_2.1.0, 24 September 2026) places the TensorFlow Lite 2.19.0 examples in /usr/bin/tensorflow-lite-2.19.0/examples and the VX delegate at /usr/lib/libvx_delegate.so. The guide says the VX delegate “directly uses the hardware accelerator driver (OpenVX with extension)”. Following that down the stack on the board gives three layers:

  • The delegate and OpenVX. libvx_delegate.so is loaded with dlopen() when you pass --external_delegate_path. It calls the Vivante OpenVX libraries that meta-freescale packages from imx-gpu-viv: libOpenVX, libGAL and libVSC among them.
  • The device node. The Vivante userspace libraries reach the kernel through /dev/galcore. In NXP’s kernel tree, gc_hal_kernel_linux.h defines DEVICE_NAME as "galcore" and CLASS_NAME as "graphics_class", and the driver registers a character device under that name.
  • One driver for three cores. In imx8mp.dtsi on NXP’s lf-6.18.y branch, the mix_gpu_ml node lists cores = <&gpu3d>, <&npu>, <&gpu2d>. The NPU at 0x38500000 is driven by the same galcore driver as the 3D and 2D GPUs. The driver names its interrupt handlers galcore:0, galcore:3d-1 and so on.

So on this board, one inference on the NPU looks like the CPU baseline plus an open /dev/galcore, ioctl() calls on that descriptor, and, in the normal interrupt-driven configuration, a rising count on a galcore line in /proc/interrupts. The driver also has a polling mode, so treat the ioctl() calls as the deciding signal and the interrupt count as confirmation. A run with no ioctl() on that descriptor has not used the NPU, even with the delegate loaded. The delegate may have declined every operator, which leaves the whole graph on the CPU.

The same guide explains the delay you will see first: “The first execution of the model inference using the delegate takes longer, because of the time required for computational graph compilation and initialization for the hardware accelerator.” Its sample benchmark_model output with the VX delegate reports First inference: 4586473 microseconds against Inference (avg): 2467.87. That is about 4.6 seconds against 2.5 milliseconds for the same model. The kernel sees that first inference mostly as userspace CPU time, spent while OpenVX compiles the graph, not as NPU work.

The RK3588 vendor stack: RKNN runtime and the rknpu driver

Rockchip’s stack is shaped differently. A model is converted on a PC with RKNN-Toolkit2 into an .rknn file, and the board loads it through the RKNN runtime, librknnrt.so, shipped in the rknpu2/runtime/Linux/librknn_api/aarch64/ directory of the airockchip/rknn-toolkit2 repository. The repository README on master names v2.5.0 as the latest version and says the kernel driver “can be found in the Rockchip kernel code”.

That driver is drivers/rknpu in Rockchip’s kernel. On the develop-5.10 and develop-6.1 branches it reports itself as RKNPU driver version 0.9.8, dated 20240828. Its Kconfig offers two memory managers, and the choice decides which device node you look for:

  • CONFIG_ROCKCHIP_RKNPU_DRM_GEM, the default. The driver registers a DRM driver named rknpu with DRIVER_GEM | DRIVER_RENDER, so the NPU appears as a render node under /dev/dri/, beside the GPU.
  • CONFIG_ROCKCHIP_RKNPU_DMA_HEAP. The driver registers a misc device named rknpu, so the node is /dev/rknpu.

This differs from the mainline rocket driver for the same NPU, which is an accel driver and appears as /dev/accel/accel0.

The vendor driver also gives you two debugfs files that answer the most common questions directly. With CONFIG_ROCKCHIP_RKNPU_DEBUG_FS, which defaults to y, it creates /sys/kernel/debug/rknpu/, with files that include version, load, power and freq. load prints NPU load: followed by a busy percentage for each core, so you can see which of the three cores one inference used.

Reading one inference on your own board

Run your inference program in a loop, as in the CPU example, and find its PID. Then work through the same four objects. On an i.MX 8M Plus EVK built from NXP’s BSP:

root@imx8mp-lpddr4-evk:~# pgrep -f label_image
root@imx8mp-lpddr4-evk:~# ls -l /proc/$(pgrep -f label_image)/fd
root@imx8mp-lpddr4-evk:~# grep -E 'tflite|vx_delegate|OpenVX|GAL' /proc/$(pgrep -f label_image)/maps
root@imx8mp-lpddr4-evk:~# grep galcore /proc/interrupts

Expect a descriptor pointing at /dev/galcore, mappings for the model, libvx_delegate.so and the OpenVX libraries, and at least one galcore interrupt line. Run the last command twice a few seconds apart. The count on the line that serves the NPU should rise while inferences run and stop when they stop, unless the driver was configured to poll, in which case the kernel log says so with a polling core message.

On a Rock 5B or another RK3588 board running a Rockchip vendor kernel:

root@rock-5b:~# cat /sys/kernel/debug/rknpu/version
root@rock-5b:~# cat /sys/kernel/debug/rknpu/load
root@rock-5b:~# ls -l /dev/rknpu
root@rock-5b:~# grep -l rknpu /sys/kernel/debug/dri/*/name
root@rock-5b:~# grep npu /proc/interrupts

The first line should read RKNPU driver: v followed by the driver version. Read load while the program runs and again when it is idle. /dev/rknpu exists only with the DMA heap option. With the default DRM GEM option the NPU is one of the render nodes, and the grep over the DRM debugfs name files shows which one. The driver requests its interrupts under the device’s own name, so the grep finds them by the unit address of the NPU node.

On either board, the system-call view separates CPU from NPU work in one command. Attach strace -f -c -p to the running process for a few seconds, as in the CPU capture. If the summary contains only futex, the inferences are running on the CPU. If it contains ioctl calls and the process holds the accelerator’s device file open, the work is reaching the driver.

Common wrong turns

Trusting the delegate’s load message. When one inference seems fast, a line saying the delegate was created looks like proof that the NPU is in use, because creating it is the step that names the hardware. It only proves the library loaded. The delegate can accept no operators and leave the whole graph on the CPU. The device file and the interrupt count tell you whether any job reached the driver.

Timing the first inference. A benchmark that times one run looks reasonable, because one inference is what the product does. On the VX delegate, NXP’s own sample shows the first run taking about 4.6 seconds against an average of 2.5 milliseconds, because the graph is compiled on that run. Measure one inference after warm-up, and report the first-run cost separately as a start-up cost.

Reading total RSS as model size. VmRSS is the obvious memory number. It adds file-backed pages to anonymous pages. The model mapping is clean and read-only, so the kernel can drop those pages and read them back from the file; the anonymous tensors can only go to swap, if there is any. In the CPU capture they were 37,628 kB and 21,244 kB. Report them separately.

Looking for the NPU under /dev/accel. Mainline accelerator drivers use /dev/accel, so it is the natural place to look. Vendor BSP drivers predate that subsystem or choose not to use it: galcore is a character device and rknpu is a DRM render node or a misc device. Check the driver’s source or Kconfig before deciding the NPU is missing.

Verification checklist

  1. The inference process has the model file mapped: grep for its name in /proc/<pid>/maps finds an r--s or r--p mapping.
  2. You have recorded RssFile and RssAnon separately from /proc/<pid>/status, after warm-up.
  3. The number of threads in /proc/<pid>/task matches the thread count you configured.
  4. For an accelerator run, /proc/<pid>/fd includes the accelerator’s device file: /dev/galcore on the i.MX 8M Plus BSP, a /dev/dri render node or /dev/rknpu on the RK3588 vendor kernel.
  5. The accelerator’s interrupt count in /proc/interrupts rises during the run and stops when it stops, unless the driver polls.
  6. strace -f -c -p during steady-state inference shows ioctl calls for an accelerator run, and only futex for a CPU run.
  7. Your reported latency excludes the first inference, and the first-inference time is recorded separately.

Key takeaways

  • One inference is a process with a file mapping, anonymous memory, threads and, for an accelerator, an open device file, ioctls and interrupts.
  • A CPU inference makes no system calls except futex in steady state; an accelerator inference makes ioctl() calls on a device file.
  • On the i.MX 8M Plus BSP the NPU is reached through /dev/galcore; on the RK3588 vendor kernel through a DRM render node or /dev/rknpu.
  • The first inference on a delegate includes graph compilation and should be measured separately.
Was this worth your time?

Frequently asked questions

How can I tell whether one inference ran on the NPU or the CPU?
Check the inference process for the accelerator’s open device file, watch the accelerator’s interrupt count in /proc/interrupts, and attach strace -c. An NPU run shows ioctl calls on the device file; a CPU run shows only futex calls in steady state.

Which device node does the NPU use on the i.MX 8M Plus BSP?
/dev/galcore. The galcore driver in NXP’s kernel serves the 3D GPU, the NPU and the 2D GPU, and the VX delegate reaches it through the Vivante OpenVX libraries.

Why does the RK3588 NPU appear under /dev/dri on a vendor kernel?
The default memory manager in Rockchip’s rknpu driver is DRM GEM, so the driver registers as a DRM render device. With the DMA heap option instead, it registers a misc device at /dev/rknpu.

Why is the first inference so much slower?
With a delegate such as NXP’s VX delegate, the first run includes compiling the graph and initialising the accelerator. NXP’s sample shows about 4.6 seconds for the first inference against an average of 2.5 milliseconds.

References

RB
Raghu Bharadwaj

Founder, TECH VEDA — 20+ years teaching the Linux kernel, device drivers and embedded systems.

Follow on LinkedIn

Get new posts by email

Kernel, embedded Linux and AI-era engineering — a few sharp reads a month. No spam.

We email occasionally and never share your address.