How to Choose an Edge AI SoC: NPU, GPU, DSP, CPU, Memory, Power and the Software Stack

An Edge AI SoC selection usually starts as a spreadsheet with a TOPS column, and that spreadsheet is where the project goes wrong. Two years later the same team is explaining why a part rated at four times the throughput of its competitor delivers half the frame rate, or why a product cannot take a security update because the accelerator driver only exists for a kernel that reached end of life.

None of that is bad luck. Every one of those outcomes is visible at Edge AI SoC selection time, in documents the vendors publish, if you know which questions have answers and which do not. This guide goes through the seven axes in the order that they actually decide the outcome, which is close to the reverse of the order they appear in a datasheet.

The TOPS number is not a unit

Start here, because it clears away most of the spreadsheet. Below are throughput figures as the vendors themselves publish them, with the qualifier each one attaches. Every row is quoted from the vendor’s own current documentation.

Put those side by side and the comparison collapses. Renesas’s 80 and NVIDIA’s 275 are sparse; TI’s 8 is dense 8-bit at 125 °C; Hailo-10H’s 40 is INT4; NXP’s 8 is a unit called eTOPS that has no published definition. Rockchip’s 6 has no data type at all. There is no arithmetic that turns these into one another.

The NVIDIA row carries the sharpest lesson, because it is the only vendor here that publishes the split. On AGX Orin 64GB, 105 of the headline 275 comes from the two deep learning accelerator engines rather than the GPU. That is thirty-eight per cent of the number on the box sitting behind a fixed-function engine with a restricted layer set. If your network does not map onto that engine, the number you compared against is not the number you get. And Orin Nano, the cheapest member of the family, has no such engine at all — the developer guide lists its DLA core count as zero — so its entire figure is GPU.

Start with the driver, not the silicon

The question that decides an Edge AI SoC selection is not how fast the accelerator is. It is what runs it, on which kernel, maintained by whom.

Linux has a subsystem for exactly this hardware. It is drivers/accel, added in v6.2, and it is not a separate subsystem so much as a branch of DRM: the core is drm_accel.c, devices appear under major number 261 as /dev/accel/accel*, and drivers set DRIVER_COMPUTE_ACCEL in their DRM feature flags. In the master tree today it contains exactly six drivers.

raghu@techveda.org:~$ ls drivers/accel/
Kconfig  Makefile  amdxdna  drm_accel.c  ethosu  habanalabs  ivpu  qaic  rocket

Read that list carefully, because it is the whole upstream story. amdxdna and ivpu are x86-only, for the NPUs inside AMD and Intel laptop processors. habanalabs and qaic both depend on PCI: they are add-in cards for data centre and edge servers, and qaic in particular is a trap, because it is Qualcomm and it is upstream but it drives the Cloud AI 100 card, not the Hexagon NPU inside any Qualcomm SoC you would put in a product.

That leaves two in-tree drivers for NPUs embedded in application-class SoCs: rocket, which supports the Rockchip RK3588 and nothing else, and ethosu, which supports the Arm Ethos-U65 and U85 IP. The device tree binding directory tells the same story from the other side — Documentation/devicetree/bindings/npu/ contains two files, one for Arm Ethos and one for rockchip,rk3588-rknn-core.

Everything else on the market reaches its NPU through an out-of-tree vendor module. That is not automatically disqualifying, and for several of these parts it is the only realistic choice, but it must be priced honestly rather than discovered later.

Three more drivers have been posted to the lists and are not merged: accel/qda for Qualcomm DSPs, which reached a second revision in August 2026 with open questions in its own cover letter about device tree binding and privilege separation; accel/neutron for the NXP Neutron NPU; and an accel/npac request for comments. Posted is not merged, and merged is not shipped in a kernel you can use.

It is also worth being precise about what an in-tree driver gives you, because it is less than people assume. The kernel documentation for rocket says it plainly: the driver powers the hardware on and off, allocates and maps buffers, and submits jobs, and everything else happens in userspace in a Mesa driver. The kernel has no idea what a convolution is. Every question about which operators are supported is a userspace question on every one of these platforms, in-tree or not. We went through what that means in practice for one part in The Mainline NPU Driver on RK3588, and What It Cannot Do Yet.

The kernel version outlives the product

This is the axis that turns a good Edge AI SoC choice into a bad one three years later, and it is arithmetic rather than judgement.

The current longterm kernels and their projected end-of-life dates, from kernel.org:

Those dates were last revised in February 2026, and the revision is instructive: 6.6, 6.12 and 6.18 all gained a year or two because enough companies depend on them, while 5.10 and 5.15 kept the dates they already had. The series you most want extended are the ones least likely to be.

Now combine that with when the drivers landed. The accel subsystem did not exist before v6.2, so a BSP based on 5.10, 5.15 or 6.1 cannot contain an in-tree accelerator driver by construction. rocket first appears in v6.18. ethosu first appears in v7.0, which is not a longterm series at all. So of the six maintained longterm kernels, exactly one contains an in-tree NPU driver for an embedded SoC, and it contains only the Rockchip one.

Set that against what the vendors actually ship. Rockchip publishes BSP branches for 4.4, 4.19, 5.10, 6.1, 6.6 and 6.12, with 6.1 as the default. NVIDIA currently ships two Jetson branches side by side: Jetson Linux 39.2.1 on kernel 6.8 with Ubuntu 24.04, and Jetson Linux 36.5.2 on kernel 5.15 with Ubuntu 22.04, both released in August 2026 and both supporting the full Orin range. “The Jetson kernel is 5.15” and “the Jetson kernel is 6.8” are both true statements, which is why you have to name the branch.

The consequence is concrete. Two of the six longterm series lose upstream stable fixes in December 2026, which from now is a matter of months. If your BSP is pinned there and your accelerator driver is an out-of-tree module written against those headers, then your security update path, your kernel upgrade path and your silicon vendor’s roadmap are one single dependency. Ask, at selection time and in writing, which kernel the vendor will support at the end of your product’s life, not which one they ship today.

NPU, GPU, DSP and CPU are one question

These four are usually presented as four separate rows in the comparison. They are better understood as a single question: when the accelerator refuses a layer, where does that layer go, and what does it cost?

Every Edge AI SoC here has a restricted accelerator and a fallback path. NVIDIA documents the restrictions on its DLA in unusual detail, and the list is instructive because it is representative rather than unusual: no dynamic dimensions, so minimum, maximum and optimum profile values must be equal; convolution kernel dimensions in the range 1 to 32; deconvolution padding must be zero; activation functions limited to ReLU, Sigmoid, TanH, clipped ReLU and leaky ReLU; the unary operations limited to absolute value, sine, cosine and arctangent; static slicing only; and at most sixteen loadables resident per core.

What happens to a layer outside that set is the part that hurts. NVIDIA’s own documentation states that with GPU fallback enabled, layers that cannot run on the DLA fall back to GPU execution silently, without an error. A pipeline you believe is DLA-accelerated can be running entirely on the GPU, competing with everything else you put there, and nothing in the logs says so.

The same pattern repeats one level up. TensorFlow Lite’s delegate documentation is explicit that if you hand a floating-point model to a delegate that only supports 8-bit quantized operations, it rejects all of them and the model runs entirely on the CPU. Not partially — entirely. ONNX Runtime’s execution provider mechanism has the same shape, and it is worth knowing how thin the edge support tier is: the Rockchip NPU provider is marked preview and community-maintained, as are the Arm Compute Library, Arm NN and Vitis-AI providers.

This is where the DSP earns its place in the comparison, and TI is the clearest illustration because it publishes the split. On AM62A the matrix multiply accelerator does 2 TOPS of dense 8-bit work, and beside it sits a C7x vector DSP quoted at 40 GFLOPS. The MMA does dense integer matrix multiply; the C7x runs what the MMA cannot and provides the floating-point path; anything neither can take falls back to the Cortex-A cores. Renesas has the same two-part structure on RZ/V2H, where the vendor’s own driver documentation defines DRP-AI as a reconfigurable processor plus an AI-MAC block described as hardware for high-speed 8-bit matrix operations.

So the useful question is not “how many TOPS”, it is “what fraction of my graph lands on the fast path, and how fast is the slow path”. A part with a modest accelerator and a strong vector DSP can beat a part with an impressive accelerator and nothing behind it, on a real network with unusual layers. You cannot answer this from a datasheet. You answer it by compiling your actual model with the vendor toolchain and reading the operator assignment report, which every one of these toolchains produces.

One practical warning about quantization while you are there. NXP documents that the i.MX 8M Plus NPU is optimised for per-tensor quantization, and that per-channel quantized models require additional compute and may carry a small accuracy error. Modern post-training quantization in both TensorFlow and PyTorch defaults to per-channel. The default export path is the slow path, and nothing tells you.

Memory decides more than the accelerator does

Three separate memory questions matter when sizing an Edge AI SoC, and teams routinely check only the first.

Capacity. Model weights, activation buffers and the working set of everything else on the board come out of the same DRAM. The part of that footprint the kernel cannot reclaim is invisible until an allocation fails, which we covered separately in The Half of an AI Model the Kernel Cannot Reclaim.

Bandwidth. This is the one that decides sustained throughput, and disclosure varies enormously. NVIDIA publishes a figure per module: 204.8 GB/s for AGX Orin on a 256-bit LPDDR5 interface, 102.4 GB/s for Orin NX on 128-bit, and 51 GB/s for the 4GB Orin Nano on 64-bit. That is a four-fold spread inside one product family, and it tracks the SKU price far more closely than the TOPS figure does. Qualcomm, by contrast, publishes bus width and clock — 2 x 16 LPDDR5 at 3200 MHz for QCS6490, 6 x 16 for the IQ-9075 — and no GB/s figure at all, so any Qualcomm bandwidth number you see has been computed by whoever wrote it down. TI publishes LPDDR4 at up to 3733 MT/s for AM62A and 4266 MT/s for TDA4VM. Rockchip’s datasheet lists LPDDR4, LPDDR4X and LPDDR5 support across four 16-bit channels without a speed grade in the chapter I could retrieve.

Movement. This is the one that gets missed, and on a camera pipeline it is decisive. The question is whether a buffer captured by the camera can be handed to the accelerator without a copy. On Linux that means dma-buf, and it is not a given. The mainline rocket driver does not implement dma-buf export and registers no import hook, so a V4L2 capture buffer cannot be handed to the NPU as a dma-buf at all — you copy it into a driver buffer first, at every frame, on top of whatever else that memory is doing.

Underneath that sit two kernel facts worth checking on any candidate. First, whether the accelerator is behind an IOMMU, because that decides whether it can work from scattered pages or needs physically contiguous memory. The contrast is visible in the two in-tree drivers: rocket depends on ROCKCHIP_IOMMU, requires an iommus property in its binding and uses the shmem GEM helpers, while ethosu has no IOMMU path at all, uses the DMA GEM helpers and pulls its scratch memory from an on-SoC SRAM pool. Second, if you need contiguous memory, you need CMA, and CMA can only be used in sleeping context, deliberately bypasses single-page allocations and works by migrating movable pages, so it fails under fragmentation rather than degrading.

One more trap in the same area, because it will bite during a BSP upgrade rather than at selection: the default CMA dma-buf heap only got a stable name in Linux 6.17. Before that it could appear as reserved, linux,cma or default-pool depending on the platform, and it is now default_cma_region. Userspace that hardcodes the old name breaks on a new kernel, and vice versa.

Power is the axis nobody documents

Ask what an Edge AI SoC consumes and you discover that most vendors do not answer.

Rockchip’s RK3588 datasheet — the only revision I could retrieve is Rev 0.1 from July 2021, marked as an initial release for special reference — has no SoC power section at all, only absolute ratings, recommended operating conditions and package thermal resistance. NXP publishes estimated maximum currents per supply rail for the i.MX 8M Plus and states explicitly that these are a guideline for power supply selection and that actual consumption for typical use cases is lower. TI publishes no typical figure for AM62A or TDA4VM, only the relative claim of the lowest power envelope in the industry at a 125 °C junction. Renesas publishes an efficiency ratio for RZ/V2H, 10 TOPS per watt, whose numerator is the pruning-inflated 80 rather than the dense 8.

The exceptions are worth noting because they show what disclosure looks like. Qualcomm states 6 to 9 W typical for QCS6490 and 3.8 to 20 W for the IQ-9075 SoC. Hailo states 2.5 W typical for both Hailo-8 and Hailo-10H — the same figure for parts of very different throughput, one of which carries external LPDDR4, which is worth treating with some caution rather than repeating.

NVIDIA documents this axis better than anyone here, through nvpmodel, and the detail rewards reading. In the r39.2.1 developer guide, AGX Orin 64GB offers MAXN, 15 W, 30 W and 50 W modes with 30 W as the default; Orin NX 16GB offers MAXN, 10 W, 15 W and 25 W with 15 W as the default. Two things in that table catch people out. MAXN carries a power budget of “n/a” rather than a wattage, and NVIDIA states in as many words that it is not the maximum performance mode, because hardware throttling engages when module power exceeds the TDP budget. And on Orin NX 16GB the number of available DLA cores is mode-dependent: two in MAXN, 25 W and 40 W, but one in the 10 W and 15 W modes — including the default. Choosing a power mode is also choosing how much accelerator you have.

The practical position is that for most of these parts you will have to measure power yourself, on your workload, in your enclosure. Budget for that. It is a week of work, and it is cheaper than discovering the thermal design is wrong after tooling.

The discrete accelerator is a real option

Before settling on a more expensive Edge AI SoC, price the other path: keep a modest SoC and attach a dedicated accelerator over PCIe. Hailo is the clearest example, and it has an unusual architecture — Hailo-8 integrates all its memory on die and requires no external DRAM, which means model weights must fit on chip. Hailo does not publish that capacity, and the practical evidence for the constraint is that Hailo-10H added a DDR interface specifically to reach larger models.

Two things decide whether this path works for you, and neither is the TOPS figure.

The first is the slot. Hailo publishes PCIe Gen 3 with four lanes for the M.2 Key M module, but two lanes for the Key B+M and Key A+E variants, and one lane for the 8R mPCIe module. That is a four-fold spread across parts that all carry the same headline number, and it interacts with what your carrier board actually routes, which is frequently narrower than the connector suggests.

The second is the driver, and here the accounting must be honest. The Hailo PCIe driver is GPL-2.0 and actively maintained on GitHub, and the HailoRT runtime is open source. But it is not in mainline, and — this is a verified negative rather than a claim about quality — no driver series has ever been posted to the kernel lists at all: a subject search and a diff-filename search on lore both return nothing. So this is an out-of-tree module you build and maintain against every kernel you ship, with the usual consequence that a kernel update can break it until the vendor catches up. That consequence is not theoretical: the module taints the kernel on load, and the vendor’s own community forums carry recurring threads from people whose accelerator stopped building after a routine kernel upgrade. Note also that the driver’s master branch supports only the Hailo-10 and Hailo-15 families, while Hailo-8 needs a separate branch.

That is a real cost, but it is a bounded and well-understood one, and it buys you the freedom to change the accelerator without changing the SoC, the BSP or the rest of the product. For many designs that is the better trade.

An Edge AI SoC checklist to run before you commit

None of the above requires a purchase order. Most of it can be answered in a day, on an evaluation board or against a kernel tree, and the answers are far more reliable than any comparison table.

On your build host, against the kernel source you intend to ship, find out whether the accelerator has an in-tree driver at all and which compatible strings it binds to:

raghu@techveda.org:~$ ls drivers/accel/
raghu@techveda.org:~$ ls Documentation/devicetree/bindings/npu/
raghu@techveda.org:~$ grep -rn "compatible" drivers/accel/*/[a-z]*_drv.c

If the answer is that there is no in-tree driver, that is a legitimate outcome and not a reason to stop. It tells you the vendor module is now a line item in your maintenance plan.

On the evaluation board itself, establish what is actually present at runtime. These read the board, so they are target-side:

root@evk:~# uname -r
root@evk:~# ls /dev/accel/ /sys/class/accel/ 2>/dev/null
root@evk:~# ls /dev/dma_heap/
root@evk:~# ls /sys/class/iommu/
root@evk:~# grep -i cma /proc/meminfo

Those five lines answer more than a week of datasheet reading. The kernel version tells you where you sit against the end-of-life table above. An empty /dev/accel confirms the accelerator is reached through a vendor module rather than the upstream framework. The heap listing tells you what buffer types you can actually allocate and whether a CMA region exists at all. The IOMMU listing tells you whether you are going to be fighting for contiguous memory. And the CMA figures in /proc/meminfo tell you how much of it you have before you have written any code.

Then do the one test that cannot be substituted: compile your real model with the vendor toolchain and read the operator assignment report. Every toolchain here produces one, and every one of them requires this step to happen on an x86 host rather than on the target — TI states plainly that model compilation cannot be performed on the SoC, and Renesas, Hailo and Rockchip all have the same offline structure. Count the operators that land on the accelerator and the ones that do not. That ratio, on your network, is the specification. The TOPS figure is marketing.

Finally, ask the vendor three questions in writing: which kernel version will be supported at the end of our product’s lifetime; is the runtime library source-available or a binary; and what is the licence and distribution channel for the compiler. On several of these platforms the compiler is the closed part of an otherwise open stack, which is manageable, but only if you know before you commit.

What this means in practice

If you take one thing from this, make it the ordering. An Edge AI SoC is chosen on its software stack first, then memory, then power, and only then the accelerator’s headline figure — because the first three are what you cannot change later, and the fourth is the one that gets all the attention.

A useful way to sanity-check a shortlist is to ask what happens in the worst case rather than the best. What is the frame rate when the model does not fit the accelerator’s operator set? What is the update path when the kernel this driver was written against goes end of life? What is the memory bandwidth per inference when the display and camera pipelines are also running? Those three answers separate real candidates from spreadsheet candidates, and none of them appears on a product page.

The teams that get this right tend to do the same unglamorous thing: they buy two evaluation boards early, compile their actual model on both, and read the operator reports before anyone writes a requirements document. It costs a fortnight. It routinely saves a redesign. And it is the same principle we applied to quantization in 4-Bit Weight Quantization: Why the Memory System Decides — the constraint that governs is almost never the one on the front of the datasheet.

Key takeaways

• TOPS figures are published in incompatible units: sparse versus dense, INT4 versus INT8, NXP’s undefined “eTOPS”, TI’s figure at a 125 °C junction, and Rockchip’s with no data type at all. They cannot be placed in one column.
• Mainline drivers/accel holds six drivers, and only two serve NPUs in an embedded Edge AI SoC — rocket for the RK3588 alone, and ethosu for Arm Ethos-U. Everything else needs an out-of-tree vendor module.
• The accel subsystem did not exist before v6.2, rocket arrived in 6.18 and ethosu in 7.0, so of the six longterm kernels only 6.18 carries an in-tree embedded NPU driver. The 5.10 and 5.15 series reach end of life in December 2026.
• Every accelerator here has a restricted operator set and a fallback path, and the fallback is frequently silent: NVIDIA documents that DLA layers fall back to the GPU without an error, and a TensorFlow Lite delegate that rejects your operators runs the whole model on the CPU.
• Memory has three questions, not one — capacity, bandwidth and whether a buffer can move without a copy. Mainline rocket has no dma-buf export, so a camera pipeline copies every frame.
• Most vendors publish no SoC power figure at all. Plan to measure it yourself, on your workload, in your enclosure.
• Compile your real model with the vendor toolchain and read the operator assignment report before you commit. That ratio is the specification.

Frequently asked questions

Can I compare TOPS figures between vendors at all?
Only after normalising them, and often not even then. The published figures use sparse and dense counts, INT4 and INT8 precisions, one undefined vendor-specific unit, and in one case a temperature qualifier. Several vendors attach no data type to the number. Treat the figure as an upper bound under the vendor’s most favourable assumptions, and compare measured throughput on your own model instead.

Which embedded NPUs actually have an upstream Linux driver?
In the current mainline tree, two families: the Rockchip RK3588 through the rocket driver, and Arm Ethos-U65 and U85 through the ethosu driver. The other four drivers in drivers/accel are x86 laptop NPUs or PCIe cards. Drivers for Qualcomm DSPs and the NXP Neutron NPU have been posted to the mailing lists but are not merged.

Is an out-of-tree vendor NPU driver a reason to reject a part?
No, but it is a cost to price rather than discover. It means you build and maintain a module against every kernel you ship, a kernel update can break it until the vendor responds, and your upgrade path is tied to the vendor’s roadmap. For most Edge AI SoC options on the market today it is the only available choice.

Why does my accelerated model still run slowly?
Most often because part of the graph never reaches the accelerator. Restricted operator sets mean unsupported layers fall back to the GPU, the DSP or the CPU, and that fallback is usually silent. Compile the model with the vendor toolchain, read the operator assignment report, and check how many operators actually landed on the fast path.

Should I use a discrete accelerator instead of a more capable SoC?
It is often the better trade, because it lets you change the accelerator without changing the SoC or the BSP. Two things decide it: the PCIe lane count your carrier board actually routes, which varies from one to four lanes across modules carrying the same headline figure, and the fact that the driver will be out of tree and maintained by you.

Further reading

• The Linux compute accelerators subsystem — what drivers/accel is for and the contract a driver signs.
• Documentation/accel/rocket — the RK3588 NPU driver, including the plain statement of what the kernel does and does not do.
• dma-buf heaps userspace API — which heaps exist, and the naming change at Linux 6.17.
• kernel.org releases — the longterm series and their projected end-of-life dates.
• TensorRT DLA layer restrictions — a rare example of a vendor documenting exactly what its accelerator will not run.
• LiteRT delegates — the delegate mechanism, and the full-CPU fallback behaviour.
• ONNX Runtime execution providers — the provider list and its support tiers.