How to Move from Embedded Linux to Edge AI: What Carries Over and What to Learn First

A colleague asks whether you can get a vision model running on the board you brought up last year. You know that board well. You know its kernel, its device tree and its image. You do not know which file format the model should be in, why people keep mentioning quantization, or how to tell whether the accelerator is doing any of the work.

This article assumes you can already build an image and bring up a Linux-class SoC. It does not cover inference on microcontrollers or training models. It covers inference on boards that run Linux, which is where your background applies directly.

Which embedded Linux skills carry over into edge AI work?

In a shipped edge AI device, much of the engineering is the system around the model. Boot, device tree, drivers, CMA and the IOMMU apply to an AI accelerator the way they do to any DMA-capable peripheral, so the concepts are not new. The workload is different: large weight and activation buffers put real pressure on CMA, the IOMMU and cache coherency. Five areas deserve extra attention:

• Driver and board bring-up. An NPU (neural processing unit) is a hardware block that runs the arithmetic of a neural network, typically faster or at lower power than the CPU for the operators it supports. It needs clocks, a power domain, memory access and interrupts described correctly. On some SoCs the accelerator is a separate core, such as a DSP, that the kernel loads firmware onto through remoteproc. A patch series proposed in May 2026 to extend the upstream rocket driver to the Rockchip RK3568 NPU is largely SoC enablement work: power domain and bus setup, a 32-bit DMA address limit and device tree nodes. Its author reported that end-to-end inference was not yet working and was still looking for the cause, which shows how much has to work below the model before it can run.
• Thermal and clock behaviour. Sustained inference heats a board. A short benchmark can hide throttling that appears minutes later, through the thermal zones, cooling devices and cpufreq or devfreq governors you already configure.
• Power management. An accelerator that never powers down breaks the thermal or battery budget, so runtime power management matters for it as for any other device.
• The update path. A model changes after the device ships. The signed update process you already use for the root filesystem applies to model files as well.
• Real-time and camera paths. If inference runs beside a control loop, scheduling and CPU isolation decide whether deadlines hold. Most vision products also involve a camera, an image signal processor and buffer sharing through V4L2 and DMA-BUF, and a careless copy into user space can lose the zero-copy path.

On these boards the model is the part that gets replaced most often. Boot, memory, thermal and update behaviour are what you will still be debugging a year later.

Our guide to choosing an embedded Linux specialization treats edge AI as a direction that builds on core embedded Linux skills. This plan takes the same position: begin it after you can build an image and bring up a board.

What is actually new: the path a model takes to reach your board

The new work is a process you have probably not owned before. You will usually receive a trained model from someone else and be asked to deploy it. The steps are:

1. Get the model in an exchange format. ONNX is a framework-neutral model file. LiteRT is Google’s runtime for .tflite files; it is the new name for TensorFlow Lite, and the .tflite format did not change. ExecuTorch is PyTorch’s runtime for edge devices and has its own export flow.
2. Quantize it. Quantization converts the model’s weights and activations from 32-bit floating point to a smaller type, usually 8-bit integers. The model becomes smaller and faster, and it can lose accuracy. Post-training quantization needs only calibration data. Quantization-aware training retrains the model and is the fallback when the loss is too large. Calibration data is a small set of representative inputs, often a few hundred, that the toolchain runs through the model to measure the range of values in each tensor so it can choose the integer scale. It is not training data. Many NPUs run only quantized models, but some accelerators and GPUs run 16-bit floating point. On a GPU, 16-bit floating point is often the lower-effort first step. Check what your target supports, including the integer type and scheme its compiler accepts, before you start. Keep a separate set of labelled inputs for accuracy checking, because a model scored on its own calibration data looks better than it is.
3. Compile or convert for the target. On many vendor toolchains this step also performs the quantization: you give the compiler the floating-point model and the calibration inputs, and it quantizes as part of compilation. Check whether your toolchain wants a floating-point or an already quantized model, because quantizing twice, or giving the compiler a model it rejects, is a common first failure. Vendor toolchains usually run on a host PC. TI’s documentation for the AM62A processor, for example, states that the processor cannot compile a model for itself, and the compiled output is a directory of artifact files that the on-board runtime loads.
4. Run it through a runtime that hands supported operators to the accelerator. An operator is one step in the model, such as a convolution or an addition. Accelerators support only some operators, and often only some configurations of them, such as particular data types, kernel sizes or tensor shapes. In ONNX Runtime you pass an ordered list of execution providers, which are plug-ins for different hardware. LiteRT uses a similar mechanism called a delegate. Operators that the accelerator does not support normally run on the CPU, although some toolchains reject the model instead.
5. Reach the hardware through a kernel driver. You already understand this part, and it is where bring-up bugs appear. Firmware load failures, missing clocks or power domains, IOMMU faults and CMA exhaustion often show up at the application level as “the model will not load”, or worse, as a model that loads and quietly runs on the CPU.
6. Verify more than once. Check accuracy on the host right after quantization, because a model that lost too much accuracy is not worth compiling. Then check accuracy and speed again on the real board, because the board’s kernels can differ numerically from the host. The performance numbers in a vendor datasheet are often not the numbers you will measure.

The following pattern shows step 4 in ONNX Runtime. Replace the first provider name with the one your SoC vendor documents:

import onnxruntime as ort

session = ort.InferenceSession(
    "model.onnx",
    providers=["<accelerator provider for your board>", "CPUExecutionProvider"],
)
print(session.get_providers())

If the printed list contains only the CPU provider, the session has no accelerator registered, so nothing can run on it. If the accelerator is listed, it is registered, but some operators can still run on the CPU. Each switch between the two can cost a data copy or at least a synchronisation step, so a model split into many small pieces can be slower than the CPU alone. Our post on why an accelerated model still runs on the CPU shows how to count the pieces.

A model that runs is not a finished deployment. A finished deployment tells you which operators ran on the accelerator and how much accuracy the conversion cost.

The runtime your board uses is mostly decided by the SoC vendor. Which inference runtime reaches the accelerator on your SoC compares the common vendor paths, so this article does not repeat that comparison.

Where the Linux kernel fits in edge AI

The kernel has a subsystem for compute accelerators. Accelerator devices appear as /dev/accel/accel*, built on the DRM infrastructure but exposed separately from GPU nodes, so that graphics software does not treat them as GPUs. The kernel documentation says the common user-space layer will typically be deep learning frameworks such as PyTorch and TensorFlow, running on vendor-specific runtime stacks.

Two upstream NPU drivers are worth knowing on Linux-class SoCs: rocket for the Rockchip RK3588, and ethosu for Arm Ethos-U65 and U85, which was merged in Linux 6.19. Check your own kernel for what is actually present before relying on either. Mesa’s Teflon framework can run TensorFlow Lite models on Ethos-U NPUs, and the first Mesa support targeted the U65. When that Mesa driver was first merged, its developer said performance was very low because nothing had been optimised yet, so measure current results on your own board.

Many production SoCs reach their accelerator another way: a vendor out-of-tree driver with its own runtime, or a separate DSP or NPU core that the kernel loads firmware onto through remoteproc and talks to over rpmsg or shared memory. TI’s C7x subsystem on the AM62A is one example. Finding out which route your board uses is a normal first step. The mainline NPU driver on RK3588 and what it cannot do yet shows what the upstream route looks like in practice. How to choose an edge AI SoC explains why the TOPS figure on a datasheet, the vendor’s peak count of trillions of operations per second, is only an upper bound.

A staged plan to move from embedded Linux to edge AI

The plan has three stages. You move to the next stage when the result of the current one is in hand, not when a number of days has passed. As a rough guide, Stage 1 takes days and Stage 2 takes most of the total effort, because toolchain setup and unsupported operators happen there. The plan assumes steady part-time effort and a board whose vendor publishes a working model-compilation flow. If your vendor does not, choose a different board for this project instead of writing your own compiler.

The plan is built around one project: one model, two execution paths and one report. Before you start, write down the requirement the result will be judged against: the latency or frame rate the product needs, whether the accelerator is shared with other work, and the accuracy loss you will accept. Choose a model whose licence allows your use. MobileNetV2 for classification is the easier first model. A small YOLO variant is a better second one, because its decoding and non-maximum-suppression steps often run outside the accelerator, and some popular YOLO releases carry licence terms that matter for a product. Use a board you already know.

Stage 1: run the model on the CPU and measure it

• Run the model on the board’s CPU with LiteRT or ONNX Runtime.
• Measure latency at batch size 1 over many runs after a warm-up, with the CPU governor and thread count fixed and recorded. For example, discard the first 20 runs and then record 200. Report the median and a tail figure such as the 95th percentile, which is the latency that 95 percent of runs stay under.
• Time the whole path from file or camera to result, not only the model call. Pre-processing and post-processing, and the copies between the camera buffer and the model input, often become the largest cost once the model itself is fast.
• Record peak memory.

The result is a baseline table that you compare everything else against.

Stage 2: move to the accelerator and find which operators stay on the CPU

• Fix your accuracy requirement first, as a number on a named metric. For example, “no more than 1 point of top-1 accuracy lost” or “mAP within 2 points of the floating-point model”. Let the product requirement set it, not the result. A loss you did not define in advance becomes an excuse and not a result.
• Quantize the model with calibration inputs that resemble your product’s real inputs, using whichever step of your toolchain does it.
• Compile the model with the vendor toolchain and run it on the accelerator.
• Produce the operator report: which operators ran on the accelerator and which stayed on the CPU. The format varies by vendor, and on some stacks it is only a compile log. Use the vendor’s per-operator profiler or the runtime’s built-in profiling to see where the time goes, because the end-to-end figure alone does not tell you.
• Run the same labelled inputs, which must not be the calibration inputs, through three paths: the original floating-point model, the quantized model on your host, and the quantized model on the board. Use identical pre-processing in all three: resize method, colour order, normalisation and layout. Report the task metric for each path, such as top-1 accuracy for classification or mAP, the mean average precision, for detection. The gap between the first two is the cost of quantization. The gap between the last two is the cost of the deployment path. A few hundred inputs is enough to catch a gross failure, but differences of a point or two on a set that size are within noise, so set a threshold you can actually resolve.
• If accuracy is far worse on the board, check pre-processing before you blame quantization. Wrong channel order, normalisation or resize settings are the most common cause.

The result is an operator report and a measured accuracy difference for each path.

Allow this stage to take longer than you expect. If your board has no NPU, do the same comparison between the floating-point model and the integer-quantized model on the CPU, and measure the speed as well. Integer models are only faster if the runtime has optimised integer kernels for your core, and on some cores they are slower. The skills of quantizing, comparing and measuring are the same. When an operator is unsupported, a report that says so, with a CPU fallback you measured, is a good outcome.

Latency tells you how fast the model ran. Only the comparison against the original model tells you whether it still gives the right answers.

Stage 3: make it run reliably as part of a product

• Run inference continuously until temperature and clocks plateau, which is often 30 to 60 minutes and longer in a closed enclosure. Record clocks, temperature and latency throughout, for example from the cpufreq and devfreq cur_freq entries and the thermal zone temperatures in sysfs, plus whatever the vendor driver exposes for the accelerator. Include CPU utilisation and, if your product is limited by power or heat, power draw.
• Run the camera, the display and any other workload at the same time, because memory bandwidth is shared between them.
• Size memory for the real workload: the CMA or reserved-memory region for accelerator buffers, plus model weights and camera buffers, measured at peak and not at idle.
• Ship the model as a separate versioned artifact, either its own package or layer or a data partition updated through your normal signed update path, so that it can be updated and rolled back independently of the root filesystem. Keep each compiled artifact paired with the exact runtime and toolchain version that produced it, because an artifact built for one version can fail to load on another. Record the toolchain version in the file name or a manifest, and have the loader refuse a mismatch with a clear error.
• Decide what happens when inference hangs or the accelerator stops responding: a watchdog on the inference process, a defined fallback to the CPU or to a safe state, and a recovery path such as a driver reset or a reboot. Test it by forcing the failure. If the product has real-time tasks, pin the inference threads and isolate cores.
• Write a one-page report: model and version, quantization method, calibration data, operators on the accelerator and on the CPU, accuracy for each path, latency, memory, and a verdict against the requirement you set at the start.

That report is your evidence. It shows where the accelerator helped and what the conversion cost, which a list of installed tools does not.

Common mistakes when moving into edge AI

• Starting with a machine learning course. For inference deployment you rarely train the model, so learn the machine learning that deployment forces you to learn, and no more at first.
• Treating the TOPS figure as a speed. It is a peak count, usually of 8-bit integer operations, under the vendor’s assumptions. Check the vendor’s footnote. It says nothing about memory bandwidth, supported operators or how well your model uses the hardware. Your measured throughput is the number that matters.
• Stopping when the output looks right. A correct result on one image proves little. The model may be running mostly on the CPU, or the quantized version may be less accurate than the original on inputs you did not test.
• Benchmarking for thirty seconds. Short runs finish before the board warms up. The clock reduction that follows is part of the product’s real behaviour.
• Treating your systems experience as the old skill set. Tool names and flows change with each vendor release. Boot, memory, thermal and update skills remain useful across those changes.

If you can, review your operator report with a mentor. A question such as “why is this operator still on the CPU?” often leads to the next thing to learn.

How to check that you have made the transition

Each item is something you can observe. An item you cannot do marks what to work on next.

1. You can state which operators of your model ran on the accelerator and which ran on the CPU, from a report and not from memory.
2. You can state the accuracy difference between the original and the quantized model on inputs that were not used for calibration, on the host and on the board.
3. You have latency, memory and temperature figures from a sustained run on the real board, with the clock settings recorded, and a verdict against the requirement you set at the start.
4. You can name the kernel driver and device node that the accelerator uses on your board, or explain why it uses a vendor route.
5. A teammate can repeat your result from your scripts and your report.

A first check to run on your own board

You can begin before you choose a model. On a board you already know, ask the runtime which hardware it can use:

python3 -c "import onnxruntime as ort; print(ort.get_available_providers())"

The output lists the execution providers that your installed ONNX Runtime build supports. Look for a provider that targets your accelerator. Providers such as XNNPACK or Arm Compute Library are still optimisations for the CPU, and an available provider is not proof that your model’s operators will run on it or that the hardware is present and working.

If Python reports that onnxruntime is not installed, or only the CPU provider appears, check whether your SoC vendor ships its own build. A generic package for your architecture may offer only CPU providers. If your vendor uses a different runtime, the equivalent check is that runtime’s own list of delegates or devices.

Then check the kernel side:

ls /dev/accel
dmesg | grep -i -E 'npu|accel|ethos|rocket|tidl'

These show whether an accelerator device node exists and whether a driver reported itself at boot. Neither is conclusive, because many vendor drivers do not use /dev/accel. Whatever the result, you now have a first fact about your board. Finding out why it looks the way it does takes you through the vendor documentation, the runtime build and the driver, which is the same route the staged plan follows.

You do not need to become a machine learning researcher. You need to be able to show, with measurements, that a model works on your board.

Further reading

• Compute Accelerators introduction, Linux kernel documentation
• TensorFlow Lite is now LiteRT, Google Developers Blog
• LiteRT delegates, Google AI Edge documentation
• Execution providers, ONNX Runtime documentation
• ExecuTorch documentation
• Deep learning models, TI Edge AI documentation for the AM62A

— Raghu Bharadwaj