Some edge AI accelerators are not driven by the Linux kernel directly: firmware on a companion core sits between the two. At Linux Plumbers Conference 2026, Tomeu Vizoso described how a driver for the C7x DSPs in Texas Instruments SoCs divides the work between userspace, the kernel and that firmware, and where each part of client isolation is enforced. A third revision of the kernel series was posted on 23 October 2026; on 15 November 2026 it was under review and not in mainline.
On many embedded SoCs the neural accelerator is a DSP or a small processor core that runs its own firmware, and Linux reaches it only by passing messages to that firmware. The talk reported here calls these firmware-mediated accelerators and asks what userspace, the kernel and the firmware must each do when the silicon has no front end to do it for them. This article reports what the slides say, then adds what has changed since the talk and what a product team should take from it. It was written from the slides, not the recording.
The talk
The talk is “Firmware-mediated accelerators for edge AI”, given by Tomeu Vizoso, whom the programme lists as “NPU drivers – Independent contractor”. It was part of the LPC Refereed Track of Linux Plumbers Conference 2026 in Prague, scheduled for Monday 5 October 2026 at 17:00 local time in a 45-minute slot. The slide deck has 30 numbered slides after the title slide.
The problem: why firmware-mediated accelerators are hard to support in mainline
The kernel has a place for compute accelerator drivers, which TECH VEDA described in The accel Subsystem: What Linux Standardises About Your NPU, and What It Does Not. The slides list seven accelerator drivers already in mainline and three under review: Qualcomm QDA for Hexagon DSPs, NXP Neutron, and the Texas Instruments C7x driver, named Thames. The talk abstract says all three were submitted in the first half of 2026.
The slides define firmware-mediated accelerators by two properties. The kernel driver interacts with the hardware through inter-processor communication with the firmware, and userspace compiles payloads that run mostly unmodified on the hardware. The difficulty follows from the second property. Code supplied by an unprivileged process runs on a processor that shares memory with the rest of the system, and according to the slides there is no front-end processor in silicon to separate one client from another, so the other components have to take over that work.
The abstract states the aim as letting users accelerate workloads without integrating a vendor BSP or vendor-specific software.
The argument
The slides begin with goals for Thames: a simple userspace interface, open source firmware, every component distributable by Yocto, Debian or Fedora, readiness for Vulkan and OpenCL, and a design that can serve as a model for other drivers. The priority is given as “Security, performance and simplicity, in that order”.
The hardware is the J722S SoC with two C7524 DSP cores at 1 GHz. Per the slides, each core has an MMU with address space identifiers (ASIDs), on-chip SRAM and one DMA engine, the Data Routing Unit (DRU).
The central slide assigns responsibilities. Userspace, a Gallium driver in Mesa, compiles each model graph into one DSP program, which the slides call a kernel, and plans the DMA transfers. The Linux driver validates those transfers, balances jobs across the cores, manages the DSP address spaces and bans a context after repeated faults. The firmware accepts jobs, switches contexts, runs each DSP kernel in user mode, and handles faults and timeouts. Remoteproc powers the core and loads the firmware.
The interface to userspace has nine ioctls, covering buffers, jobs, device queries and buffer synchronisation. The firmware is open source and redistributable but needs a closed source compiler. The speaker states that the driver fulfils the DRM requirements for mainline. He reports SSDLite MobileDet with TensorFlow Lite at 14.4 ms per frame, with a target of about 5 ms after userspace performance work. These are the speaker’s figures on the J722S; we have not reproduced them.
The points worth your attention
1. Isolation is enforced in two places, because the DMA engine bypasses the MMU
Memory reached by the DSP program is protected by the DSP’s MMU. Each client has its own page table, which the Linux driver manages, while the firmware supervisor manages its own address space. The program runs in the DSP’s user execution mode, where, per the slides, it has no access to other page tables or to privileged registers, and a violation produces a clean fault.
The DMA engine is the exception. The slides say the DRU does not go through the MMU, and add: “Errors don’t raise any faults: must be validated”. Userspace writes the DRU records, because planning data movement into the fast SRAM is part of compiling the model. The Linux driver therefore checks every record, translates its addresses from DSP virtual to physical, and returns an error for an invalid address or length. This matters because the security boundary is not only the MMU. It includes a kernel routine that parses transfer descriptions written by an untrusted process, and that routine has to be correct for the isolation to hold.
2. The cost of sharing the DSP is given in cycles and milliseconds
The slides give the speaker’s measurements for a context switch between clients: 172 cycles to switch page tables, with no TLB invalidation because of the ASIDs; 223k cycles to clear the SRAM; at most 2k cycles to close streams and purge the matrix multiply unit; and 18 cycles to wipe registers. At the stated 1 GHz clock, clearing the SRAM takes about 0.22 ms by our arithmetic.
Jobs are ordered between clients by the DRM scheduler, which also chooses the core, and completion is reported through a fence: a sequence number that the device writes and the host reads.
When a DSP kernel faults or exceeds its time limit, the firmware supervisor clears state and reports a fault record to the driver. The driver bans a context that has faulted three times in its last 10 jobs. A reset of the DSP core is given as 94 ms, and the slides reserve it for faults and hangs of the firmware itself. For a product that runs two models from two processes on one DSP, these figures bound how much one process can delay the other.
3. The kernel interface contains nothing specific to neural networks
One slide says: “UAPI and IPC know nothing about ML”. A job consists of a DSP kernel, a parameter buffer and a list of DMA transfers; the DSP kernel is treated as untrusted; each job has its own completion fence. The consequence for a team is that the model format, the supported operators and the quantisation all live in the Mesa driver, so a new operator needs a userspace update and no kernel change.
What has happened since
We checked the following on 15 November 2026.
- Version 3 of the kernel series was posted on 23 October 2026 with 16 patches; versions 1 and 2 date from January. The v3 change list matches the slides: a shared-memory ring and a mailbox doorbell replace rpmsg, the Linux driver writes the client page tables, DSP kernels run in user mode, and the driver checks and translates DRU transfers.
- Version 3 differs from the slides in one respect that we found. The slides say buffers are mapped to the DSP on creation. In v3, creation returns only a handle, and a separate bind ioctl maps the buffer at a DSP address chosen by the client.
- The v3 cover letter lists limitations, among them no runtime power management and the J722S as the only supported SoC.
- Review is in progress. On 26 October a devicetree bindings maintainer asked for one binding patch to be reworked and gave a Reviewed-by tag on another. A maintainer of the DMA buffer sharing framework asked how 2 MiB mappings are guaranteed when memory is fragmented and wrote, after a brief reading, that the job submission patch looked sound with respect to fence requirements. We found no version 4 in the dri-devel archive up to 15 November.
- The series is not in mainline. The accelerator makefile in the mainline tree, whose latest tag was 7.4-rc2 of 9 November, builds six drivers and Thames is not one of them. We did not check the DRM development trees.
- The Mesa documentation page for its inference front end did not list the thames driver.
- Of the other two firmware-mediated accelerators the slides place under review, Qualcomm QDA reached version 4, with 25 patches, on 1 November 2026; its documentation patch describes FastRPC over RPMsg. For NXP Neutron, the latest revision we found is version 2, from March 2026.
What it means for a product team
This section is TECH VEDA’s reading, not the speaker’s.
First, there is nothing to deploy yet. The kernel driver is under review, the v3 cover letter points to the userspace driver as a Mesa merge request, and the slides say the firmware needs a closed source compiler. A product on a J722S board that needs the DSP for inference today still depends on the vendor’s software.
Second, the design in the slides suggests questions to put to any vendor of firmware-mediated accelerators. Which component stops one client from reading another’s buffers? Is there a data path, such as a DMA engine, that does not pass through an MMU, and what validates it? What happens to other clients when one job faults, and how long does recovery take? Does the kernel interface change when a new operator is added?
Third, plan for one firmware image per DSP core. The v3 cover letter says a core found running other firmware is left out. In our reading, this stack and a vendor inference stack are alternatives on a given core.
Fourth, measure your own model. The slides list more operators as future work and report one detection model on one SoC. They contain no comparison with the vendor’s runtime and no power figures.
The talk does not settle when the driver will be merged, or whether the Qualcomm and NXP drivers will move towards the same division of work.
What was checked and what could not be verified
We worked from the PDF slide deck attached to the conference programme, read on 15 November 2026. The programme links a recording; we did not watch it, so anything said only aloud is absent, and the three quotations are slide text. Several slides are diagrams with a title only, and we did not interpret them. The main sources read for later developments are listed in References. We could not read the firmware repository or the Mesa merge request, and we ran no code on a board.
Key takeaways
- In firmware-mediated accelerators, the kernel talks to firmware on a companion core, and userspace supplies the program that the core runs.
- In the Thames design, the DSP’s MMU and user mode isolate clients, and the Linux driver validates DMA transfers because the DMA engine bypasses the MMU.
- The speaker’s figures on the J722S include 223k cycles to clear SRAM on a context switch, 94 ms for a core reset and 14.4 ms per frame for SSDLite MobileDet.
- Version 3 of the kernel series was posted on 23 October 2026 and was under review, not in mainline, on 15 November 2026.
Frequently asked questions
What are firmware-mediated accelerators?
Firmware-mediated accelerators are devices that the Linux kernel does not program directly. The kernel driver communicates with firmware running on a companion core, and userspace compiles payloads that run mostly unmodified on the hardware. The slides name Qualcomm QDA, NXP Neutron and the Texas Instruments C7x as examples.
Why does the kernel driver validate DMA transfers in the Thames design?
The DMA engine of the C7x, the Data Routing Unit, does not go through the MMU, and according to the slides its errors raise no faults. Userspace writes the transfer records, so the Linux driver checks each record and translates its addresses before the firmware uses it.
Is the Thames driver in the mainline kernel?
No. Version 3 of the series was posted on 23 October 2026 and was under review on 15 November 2026. The mainline tree at that date, whose latest tag was 7.4-rc2, did not build it.
Which hardware does the talk cover?
The talk presents firmware-mediated accelerators through one example, the Texas Instruments J722S SoC with two C7524 DSP cores. The slides list the TDA4VM, AM68, AM69 and AM62A7 as hardware to be supported later, and the version 3 cover letter names the J722S as the only supported SoC.
References
- Recording: Firmware-mediated accelerators for edge AI (linked from the conference programme)
- Slides: Firmware-mediated accelerators for edge AI (PDF)
- Linux Plumbers Conference 2026 programme entry
- [PATCH v3 00/16] New DRM accel driver for Texas Instruments’ C7x DSPs (cover letter, 23 October 2026)
- Review reply to v3 patch 07/16, buffer creation and mapping (26 October 2026)
- [PATCH v4 01/25] accel/qda: Add QDA driver documentation
- drivers/accel/Makefile in the mainline tree




