TinyML vs Edge AI on Linux is not a performance comparison, and TOPS is the wrong number to argue about. It is a decision about whether your product has an operating system underneath the model at all. TinyML runs the model on a microcontroller with no MMU, no dynamic memory allocation and often no OS, in a few kilobytes of RAM, and it optimises energy per inference. Linux Edge AI runs the model as a userspace process on an SoC with an MMU, a scheduler and a filesystem, and dispatches work to an NPU through a kernel driver. Pick the machine from the duty cycle, the power budget and the update model — not from the accelerator’s marketing figure.
Teams keep asking the wrong question about on-device AI. They compare a microcontroller running a keyword spotter with an SoC running a detection model, and they compare them on throughput. The two are not competing implementations of the same design; they are different machines with different contracts. This article sets out what separates them, what the current measured numbers look like, a decision guide you can apply to a real product, and three situations where the correct choice is not ambiguous.
Two machines, not two model sizes
TinyML means models small enough — typically under two million weights — to run on microcontrollers and similar constrained devices drawing sub-milliwatt to low-milliwatt power. That is the MLCommons definition, and the important consequences are architectural, not numerical.
The TensorFlow Lite Micro paper states the constraint directly: embedded processors sit 100 to 1000 times below their mobile counterparts in compute, memory and power, the inference framework must “operate in a few kilobytes of memory”, and system vendors routinely omit features that mainstream systems assume, including dynamic memory allocation and virtual memory. MLCommons puts the memory gap against smartphone-class ML at roughly six orders of magnitude.
Read that again, because it is the whole article. On a TinyML target there is:
- No MMU, so no virtual memory and no process isolation.
- No
malloc()in the inference path. TFLite Micro takes a fixed arena that you size at build time; if the model does not fit, the build fails rather than the device failing in the field. - No filesystem. The model is part of the firmware image, usually a C array flashed with the binary.
- Often no operating system, or a small RTOS such as Zephyr or FreeRTOS, with no demand paging and no page cache.
On the Linux side, all of those exist. The model is a file on disk, loaded at runtime by a process, into memory the kernel manages, on hardware that a driver powers up and feeds. That flexibility is the whole reason to run Linux, and it is also the whole cost.
What the numbers look like
The MLPerf Tiny v1.4 round, published by MLCommons in July 2026, is the cleanest current evidence for the TinyML side. It fixes the model, dataset and accuracy target for each task and measures accuracy, latency, and — this is the part Linux teams almost never measure — energy per inference. Nine organisations submitted 25 system configurations.
Some results worth holding on to:
- ASYGN’s ColibriNPU completed a Visual Wake Words inference (is a person present in a 96×96 image) using 22.2 microjoules. MLCommons notes that this is low enough for a CR2032 coin cell to run one inference per second for more than three years.
- Syntiant’s NDP120 runs the streaming wake-word benchmark at a 3.3% duty cycle — the device is idle and listening about 97% of the time.
- STMicroelectronics reports that enabling the hardware signal processor on the STM32U3 reduces image-classification inference time by up to 76%, with 23.3% lower power, against the same Cortex-M33 configuration.
Now the Linux side, measured with an equally open stack. Linaro benchmarked the Arm Ethos-U65 NPU on an NXP i.MX93 board using mainline Linux 6.19-rc1, which carries the Ethos-U accelerator driver, with TensorFlow Lite and Mesa’s Teflon delegate. Supported models ran 6× to 11× faster than the CPU baseline, and that baseline was not a weak one: for INT8 and UINT8 models TFLite uses ruy and gemmlowp with NEON, not XNNPACK.
The caveats in that report matter more than the speedup. The Ethos-U65 accepts only INT8 or UINT8 per-tensor quantisation; per-channel quantised models often fall back to the CPU, and float models do not run on the NPU at all. The fallback is silent — you get correct-looking output at CPU speed, and in one object-detection case, wrong detections. This is a class of failure that does not exist on a microcontroller, where an unsupported operator is a build-time error.
Where the model actually executes under Linux
Linux has a Compute Accelerators subsystem, and the kernel’s own documentation for the Rockchip NPU driver (accel/rocket, supporting the RK3588) describes the contract in one sentence: the driver “just powers the hardware on and off, allocates and maps buffers to the device and submits jobs to the frontend unit. Everything else is done in userspace“, as a Mesa driver.
That is the Linux Edge AI stack in miniature. The kernel does power management, buffer allocation and mapping, and job submission. The compiler, the runtime, the delegate and the model live in userspace.
Because buffer allocation and mapping sit on the kernel side, the buffer path is the DMA path. NPU input and output buffers are shared with a device that reads and writes memory independently of the CPU, so cache maintenance, coherency and mapping lifetime apply here exactly as they do to any other DMA-capable peripheral — the ordinary coherent and streaming DMA mapping problem, not a new one. This is also where an accelerated pipeline can lose much of its advantage: if every frame is copied into and out of an intermediate buffer instead of being shared with the device, a model that runs several times faster on the NPU delivers a far smaller end-to-end improvement.
So the questions that decide whether your board actually works are: is there an upstream driver for this NPU, is there an open userspace to drive it, and which operators does that path support? A 6 TOPS NPU whose only driver lives in a vendor kernel fork from 2022 is a maintenance liability for the life of the product, not a feature.
TinyML vs Edge AI on Linux: a decision guide
Compare the two on the properties that constrain a product, not on inference speed. MCU here means a microcontroller-class TinyML target; Linux means an application-class SoC with an NPU or other accelerator.
| Property | MCU (TinyML) | Linux + accelerator |
|---|---|---|
| Software underneath the model | Bare metal or a small RTOS (Zephyr, FreeRTOS) | Full kernel: scheduler, MMU, drivers, filesystem |
| Memory model | Fixed arena sized at build time; no malloc(); no virtual memory | Virtual memory, page cache, dmabuf, DMA-mapped NPU buffers |
| Working memory | Kilobytes | Hundreds of megabytes to gigabytes |
| Where the model lives | Compiled into the firmware image | A file, loaded at runtime by a process |
| Updating the model | A firmware release, signed and flashed | Replace a file; no change to the boot chain |
| Power | Sub-milliwatt to low milliwatts; runs for years on a cell | Hundreds of milliwatts to watts; usually mains or a large battery |
| The metric that decides success | Energy per inference (microjoules) and duty cycle | Throughput and end-to-end latency under load |
| Timing behaviour | Deterministic by construction; no scheduler jitter | Jitter from the scheduler and the page cache; needs PREEMPT_RT and CPU isolation to bound it |
| Unsupported operator | Build-time failure | Silent CPU fallback at runtime, sometimes with wrong output |
| What can go wrong in the field | Arena overflow, sensor drift, battery exhaustion | Driver and BSP rot, CVEs, cache coherency bugs, OTA failure |
| Long-term maintenance cost | Low: a frozen firmware image | Real: kernel LTS, security updates, an NPU driver that must stay alive upstream |
| Choose it when | Always-on sensing, hard energy budget, one fixed task, no network stack needed | Vision or audio at scale, several concurrent tasks, networking and OTA, models that change often |
Reading the table, the rule is simple. If the energy budget is the product, choose the MCU. If the software around the model is the product, choose Linux.
There is one asymmetry the table does not show: the two mistakes do not cost the same. Scope tends to grow in one direction. A device that ships as a sensor acquires a network connection, then remote updates, then a second model, then a security requirement it did not have at design time. Moving from a microcontroller to Linux part-way through a programme is a re-architecture — new silicon, new boot chain, new skills on the team. Carrying an SoC that turned out to be larger than you needed is a line on the bill of materials. If you are genuinely undecided, those two errors are not equally expensive, and that argues for deciding on the device you will have in three years rather than the one on the bench today.
Regulation pulls on this decision as well, and it pulls both ways. The EU Cyber Resilience Act obliges manufacturers to handle vulnerabilities and ship security updates across a product’s supported life, which argues for a platform that already has a real update mechanism in it; what the Act requires of device makers is a subject in its own right. It also argues against carrying software you do not need, because a frozen firmware image has a much smaller attack surface and far fewer published CVEs to triage than a kernel plus a Linux userspace. The Act raises the cost of the Linux path and raises the bar for choosing it. It does not settle the choice.
Three situations where the choice is not ambiguous
Scenario 1 — A battery-powered vibration sensor on a motor, deployed for years. The device samples an accelerometer, runs a small anomaly-detection model, and raises an alert when a bearing starts to fail. It must last years on a primary cell in a place nobody wants to visit. Pick TinyML. The energy budget is the product. A Linux SoC spends more energy booting than this device is allowed to spend in a week, and once you are counting microjoules per inference, an MMU, a page cache and a scheduler are overhead you pay for and do not use. The MLPerf Tiny anomaly-detection task exists precisely because this is one of the most commonly shipped TinyML products.
Scenario 2 — A retail camera that detects people, reads shelf labels and uploads events, with a five-year field life and monthly model updates. Pick Linux with an accelerator. Not because the model is large — a person detector is not large — but because of everything around it: a camera pipeline, a network stack with TLS, secure OTA updates, a filesystem, log rotation, and a model you intend to replace repeatedly without reflashing firmware. On a TinyML target a model update is a firmware release; on Linux it is a new file. Once the model changes monthly, that difference outweighs any latency figure. This is also where the driver question decides the platform: choose the SoC whose accelerator has an in-tree driver and an open userspace, not the one with the largest number on the datasheet.
Scenario 3 — A hearable, or any always-on wake word in front of a larger system. Use both, in two tiers. The Syntiant result — a streaming wake word at a 3.3% duty cycle — is the shape of the answer. A microcontroller-class tier stays awake at milliwatts, listening. When it triggers, it wakes the Linux tier, which does the expensive work with the full runtime and the NPU. Qualcomm’s MLPerf Tiny submission on the Snapdragon 8 Elite Gen 5 Sensing Hub, reporting sub-0.30 ms latencies for keyword spotting and visual wake words, is the same architecture inside one package. If your product is always sensing but only occasionally reasoning, a single-tier design is usually wrong in either direction: a Linux system that never sleeps drains the battery, and a microcontroller alone cannot run the model that justifies the product.
The questions that decide it
In order of how much they constrain the design:
- What is the duty cycle? Always-on sensing at low milliwatts is TinyML territory. Frequent or continuous heavy inference is Linux territory.
- What is the energy budget per inference? If you cannot state it, you are not ready to choose. TinyML teams measure microjoules; if that number does not matter to you, it is a signal that Linux is affordable.
- How will the model be updated, and how often? Firmware release versus file replacement.
- What else must the device do? Networking, storage, a display, several sensors, containers, TLS and OTA all argue for an OS.
- How likely is the scope to grow? If networking, remote updates or a second model are plausible within the product’s life, weigh that now. Adding them later to a microcontroller design means starting again.
- Does the accelerator have an upstream driver and an open userspace? On Linux this decides your maintenance cost for the life of the product.
None of these questions is “how many TOPS”. The accelerator figure is the last thing to look at, not the first.
Both sides of this decision are engineering disciplines in their own right, and the Linux side is the one teams underestimate. The driver, the buffer path, the power management and the update chain are what make an edge AI product work in the field, long after the model has stopped being the interesting part of the system.
Key takeaways
- TinyML and Linux Edge AI are different machines, not different model sizes. The dividing line is the absence of an MMU, dynamic allocation and a filesystem, not the parameter count.
- TinyML optimises energy per inference — MLPerf Tiny reports figures such as 22.2 µJ for a Visual Wake Words inference, and a streaming wake word at a 3.3% duty cycle. If that metric does not matter to your product, Linux is probably affordable.
- On Linux, the kernel powers the NPU, maps its buffers and submits jobs; the runtime, compiler and model live in userspace. Upstream driver support, not TOPS, decides the maintenance cost.
- The NPU buffer path is a DMA path, so cache coherency and mapping lifetime are ordinary DMA concerns, and avoidable copies erode the accelerator’s speedup end to end.
- The two mistakes do not cost the same. Moving a microcontroller design to Linux part-way through a programme is a re-architecture, while an oversized SoC is a bill-of-materials cost, so an undecided team should decide for the device it will have in three years.
- NPU operator support is a real constraint: the Ethos-U65 runs only INT8/UINT8 per-tensor models and falls back to the CPU silently when it cannot.
- Many products are correctly built as two tiers: an always-on microcontroller tier that wakes a Linux tier when something interesting happens.
Frequently asked questions
What is the real difference between TinyML and Edge AI on Linux?
TinyML runs inference on a microcontroller with no MMU, no dynamic memory allocation in the inference path and often no operating system, with the model compiled into the firmware image and a fixed memory arena sized at build time. Edge AI on Linux runs the model as a userspace process on an SoC with virtual memory, a scheduler and a filesystem, dispatching work to an NPU through a kernel driver. The difference is the machine underneath the model, not the size of the model.
How much power does TinyML actually use?
MLPerf Tiny measures energy per inference. In the v1.4 round, one submission completed a Visual Wake Words inference using 22.2 microjoules, which MLCommons notes is low enough for a CR2032 coin cell to run one inference per second for more than three years. Streaming wake-word systems can run at duty cycles around 3%, meaning the device is idle most of the time.
How does the Linux kernel run a model on an NPU?
It does not run the model. The kernel’s compute accelerator drivers power the NPU on and off, allocate and map buffers for it, and submit jobs to it; the compiler, runtime, delegate and model all live in userspace. The kernel documentation for the Rockchip accel/rocket driver states this explicitly.
Should I choose an SoC by its TOPS figure?
No. Check whether the accelerator has an upstream kernel driver and an open userspace stack, and which operators and quantisation formats that path supports. An NPU whose only driver lives in an old vendor kernel fork becomes a maintenance problem for the entire life of the product, whatever its TOPS number.
Does the Cyber Resilience Act favour TinyML or Linux?
Neither on its own. The Act obliges manufacturers to handle vulnerabilities and ship security updates across a product’s supported life, which favours a platform that already has an update mechanism. It also penalises unnecessary software, and a frozen firmware image has a smaller attack surface and fewer CVEs to triage than a kernel and a Linux userspace. It raises the bar for choosing Linux rather than deciding the question.
Further reading
- MLPerf Tiny: Benchmarking AI at the Edge (v1.4 results) — MLCommons, July 2026
- TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems — David, Duke, Jain, Janapa Reddi
- accel/rocket Rockchip NPU driver — kernel.org, Compute Accelerators subsystem
- Direct kernel access to the Ethos-U65 NPU on i.MX9: benchmarks — Linaro
- MLPerf Tiny benchmark results — MLCommons
