The Hidden Cost of Out-of-Tree Drivers and Private Kernel Patches

No team decides to become unable to upgrade its kernel. The decision that produces that outcome is always smaller and always reasonable: a patch to make a peripheral behave, a revert to work around a regression before a deadline, a driver copied from another tree and adjusted. Each is a day of work and each solves a real problem. The cost is never in the patch you are writing. It is in the fiftieth one, three years later, when someone needs to move to a supported kernel and cannot say what half of them are for.

This is the argument for treating that accumulated difference as a design decision with a running cost. It is a different question from where a single driver should live, which we covered in In-Tree vs Out-of-Tree Driver: Where Your Code Should Live. This is about the whole of it, drivers and patches together, and what it does to your ability to take the next kernel.

What counts as downstream delta

The term needs defining, because teams usually hear it as “our patch files” and it is much larger than that.

Downstream delta is every meaningful difference between the kernel baseline you consume and the exact kernel stack your product ships. The baseline might be an upstream stable or long-term release, an Android Common Kernel, or a silicon vendor’s branch. The delta includes patches, Kconfig and defconfig changes, device tree changes, out-of-tree modules, firmware version assumptions, build system changes, and any private interface that user space depends on.

Two of those are routinely forgotten. A device tree that describes hardware differently from the upstream binding is delta, and it becomes visible the day the driver goes upstream with a binding that does not match yours. A closed application calling a vendor-specific ioctl is delta too, and it is the kind that blocks upgrades rather than only adding cost to them.

Ten kinds of downstream delta

Not all downstream code is equally expensive. Sorting it is the single most useful thing an architect can do here, because the categories have genuinely different exits.

Medium risk — you will pay, and you can predict it.

Out-of-tree modules

Code in files that are yours, built against the kernel. There is no stable in-kernel API, so an interface change means a fix-up. The work is proportional to how much the interfaces you use have moved, which makes it predictable in a way the other categories are not. The choice of whether a given driver should live here at all is covered separately in the in-tree versus out-of-tree discussion linked above.

Configuration drift

Less discussed and more annoying. CONFIG_ symbols get renamed, merged or removed upstream, and a defconfig fragment that selected one silently stops selecting anything. There is no build error. The feature is simply absent in the next image, and the failure appears either during testing or after the product has shipped. The control is to validate the generated configuration rather than the fragment, which is covered below.

High risk — the cost grows without you doing anything.

Patches to files you do not own

This is the category teams underprice most consistently, and the reason is a genuine asymmetry described in the next section.

Reverts of upstream commits

A revert can be the correct short-term response to a verified regression, and treating every one as a failure is wrong. Treat it instead as an incident artefact, and record the reverted commit, the affected hardware, the failure signature, the upstream bug link, the test that catches it, the owner, and the condition under which it is removed. A revert carrying those fields is a managed workaround. A revert carrying none of them is untracked divergence that reverses whatever the original commit fixed, with no record that it has done so, which is a serious matter when that commit was a security fix.

Forked drivers

Someone copies an upstream driver, renames it, and changes it. It works immediately, which is why it is popular. The danger is not the copy itself but losing the connection to the source: from that moment you receive neither upstream bug fixes nor vulnerability notices for that code, and nothing tells you. If a downstream copy is genuinely necessary, record the upstream base commit, preserve the original commit metadata, and schedule a comparison against the upstream driver. A tracked fork is a maintenance task. An untracked one is the only item on this list whose failure mode is complete silence.

Backports of features that are not upstream yet

You take a series off a mailing list because you need it now. You then own that merge indefinitely, and when the feature is merged upstream in a different form, you own the conflict between your version and the real one. Track the series and its landing status so the replacement is planned rather than discovered.

Critical risk — the upgrade does not happen.

Binary-only modules

You cannot rebuild what you have no source for. The upgrade is gated on the supplier shipping a build for the new kernel, on their schedule. If the supplier stops, that kernel version is the last one your product will ever run. The only real control is commercial: a written support commitment, with the supported kernel versions and duration named in it.

Private user-space interfaces

Vendor ioctls that never went upstream, with closed user-space depending on them. The new kernel does not have the interface, and you cannot change the application. This has the same effect as a binary-only module, at the application level rather than the kernel level.

Lost provenance

A patch with no commit message, no author and no ticket. You cannot drop it, because you cannot prove it is safe to drop. You cannot forward-port it properly, because nobody knows what it was for. In practice this is what stops a tree being upgraded at all: not one impossible patch, but forty unexplained ones that nobody will take responsibility for removing.

Certification lock

The safety case, the security certification or the customer qualification is bound to one specific build. The upgrade may be technically trivial and still be impossible for the year, because requalifying costs more than the vulnerability appears to.

Why separate code costs less than modified code

The most useful question about a piece of downstream code is not how many lines it is. It is who controls the rate at which its surroundings change.

An out-of-tree module conflicts when the interfaces it uses change. A patch that modifies an existing upstream file conflicts whenever anybody touches that file. Your maintenance cost in the second case is therefore set by upstream’s activity in that area, not by yours, and you have no influence over it. A patch in a rarely modified driver may need no work for years. The same size of patch in the scheduler, in memory management, in DRM or in networking will need attention on almost every rebase.

That argument is often over-read, so it is worth stating the limit. A module boundary does not give you a stable in-kernel API, and there is no such thing to be had — the kernel documentation is explicit that a stable internal interface is not on offer. An external module still requires maintenance whenever the interfaces it depends on change, and for fast-moving areas such as media, DRM, tracing or power management that maintenance can be substantial. The advantage of a module is not stability. It is separability: the code, its ownership, its tests and its divergence are all clearly bounded, which is what makes the cost predictable rather than small.

What this costs at scale: the Android case

Most organisations cannot say what their delta costs them, because nobody measures it. Android is the exception, and its published account is worth borrowing precisely because it is the largest downstream in the world. The figures below are from Android’s own Generic Kernel Image documentation, checked in September 2026.

Before the Generic Kernel Image project, device-specific changes by silicon vendors and manufacturers “could result in as much as 50% of kernel code being out-of-tree code and not from upstream Linux kernels or ACKs”. The consequences Android lists are the ones this article is about. Security patches had to be backported into every device kernel, and fragmentation made that “prohibitively expensive”. The custom modifications made it “difficult to just merge the LTS fixes into device kernels”. And there was a delay of up to eighteen months between an upstream long-term release and it reaching a device.

One figure is directly relevant to anyone who treats upgrades as optional. On Pixel devices, Android found that ninety per cent of the kernel security issues reported in its security bulletin had already been fixed for devices that stayed up to date. Those fixes required no backporting work at all; they arrived with the update. A large delta is what prevents an organisation from collecting them.

The structural response is instructive, and needs describing accurately. The Generic Kernel Image moves silicon and board support out of the common kernel into loadable vendor modules, and defines a Kernel Module Interface that Android commits to keeping compatible for the supported lifetime of a kernel. From Android 12, devices shipping with kernel 5.10 or later must use it. That is not a general stable module ABI in the upstream Linux sense, and it does not exist upstream; it is a compatibility guarantee that holds inside Android’s tightly constrained build, with a single configuration and a controlled toolchain. It reduces the scope of divergence. It does not remove vendor integration, testing or lifecycle work.

Measure your delta before you argue about it

The discussion is more productive with a measurement in it, and counting private kernel patches is not difficult. On your build host, against the vendor tree you actually ship, measure the distance to the nearest upstream tag:

raghu@techveda.org:~$ git log --oneline v6.12..vendor/release | wc -l
raghu@techveda.org:~$ git diff --shortstat v6.12..vendor/release

Then find the part of the delta that has no explanation attached. In an OpenEmbedded or Yocto layer the patches are files, so this is a one-line question:

raghu@techveda.org:~$ grep -rL "Upstream-Status" --include="*.patch" recipes-kernel/

Every file that command prints is provenance debt. The Yocto contributor guide requires an Upstream-Status tag on each patch, with one of six values: Pending, Submitted, Accepted, Backport, Denied or Inappropriate, the last two carrying a written reason. The project takes this seriously enough to publish metrics counting patches by status, including a category for tags that are missing or malformed.

That tag costs almost nothing to add, and it addresses the most damaging category in the table above. A patch marked Backport with an upstream commit id has a defined end: it disappears when you move past that version. A patch marked Denied with a reason is a decision someone can revisit. A patch marked Pending for three years is a decision nobody made. The Yocto guidance is blunt about that state, saying it should only be used in exceptional cases where there is a genuine obstacle to submitting upstream.

Validate the generated configuration, not the fragment

Check the result rather than the input. A fragment that still contains a symbol is not evidence that the symbol is still set in the resulting build:

raghu@techveda.org:~$ make olddefconfig
raghu@techveda.org:~$ scripts/diffconfig .config.old .config
raghu@techveda.org:~$ grep -E "CONFIG_(YOUR_SYMBOL|RELATED_SYMBOL)=" .config

Keep the generated configuration as a build artefact and fail the build when a required symbol disappears. That converts a failure which would otherwise appear in the field into a failed build.

Confirm the runtime inventory, then compare it with the build

On the device itself, confirm what is loaded that did not come from the kernel build:

root@evk:~# cat /proc/sys/kernel/tainted
root@evk:~# lsmod
root@evk:~# find /lib/modules/"$(uname -r)" -type f -name "*.ko*" | sort

Read those results with their limits in mind. Downstream code that was built into the kernel rather than loaded as a module will not appear in the module list at all, and the taint value tells you that something external was loaded without telling you which files. Runtime inspection confirms what you already expect; it does not by itself establish what is present. The authoritative source is your build manifest and software bill of materials, which is also what a customer or regulator will ask for.

The decision: keep private kernel patches small, bounded and labelled

The recommendation is not that private kernel patches are forbidden. Products ship on real schedules and some of this is unavoidable. It is that they should be a managed quantity with three properties.

Small enough to move

The test is not the line count. It is whether one engineer can rebase the tree onto a new long-term kernel inside a normal sprint. If the answer is no, you have already lost the ability to respond to a vulnerability quickly, and you will find that out during an incident rather than during planning.

Separated rather than interleaved

Prefer a module boundary over a patch wherever the hardware allows it, for the separability reason above rather than any promise of stability. This is the structural insight behind Android’s approach: it did not ask vendors to write less code, it moved that code out of the common kernel into loadable modules with a defined compatibility contract, so the kernel and the modules can be updated independently.

Labelled without exception

Every item records why it exists, who to ask about it, and what would remove it. At minimum: the upstream base commit or tag, an owner, the product reason, the upstream status, the test that proves it still works, and the condition that removes it. Organisations working under formal traceability regimes will need more — licence and redistribution status, security relevance, and a bill-of-materials entry for every module and firmware blob — but those six fields are what separate an item that has an end date from one that does not.

Consequences

Accepting this costs upfront effort in places that feel unproductive. Writing a patch header takes longer than writing the patch. Submitting upstream introduces review latency you cannot schedule. Structuring hardware support as a module sometimes means a worse-fitting design than the patch you would otherwise write. These are real costs, and they fall on the current release while the benefit falls on a later one, which is exactly why this decision usually loses when it is made implicitly.

What you buy is the ability to take an upgrade when you need it rather than when it is convenient. That matters most in the situation nobody plans for: a vulnerability with a known fix in a newer stable release, a customer asking when you will ship it, and a tree that will not rebase. At that point the delta is not technical debt in the abstract. It is the reason the answer is measured in quarters. The same lifecycle question decides silicon selection, which is why kernel support and its end-of-life date belong in that decision too, as covered in How to Choose an Edge AI SoC.

There is also an organisational consequence worth naming. A large unexplained delta concentrates knowledge in whoever last touched it, and that person eventually leaves. A labelled delta, with upstream status recorded per patch, keeps that knowledge in the repository rather than in one person’s memory.

Key takeaways

• Nobody chooses to be unable to upgrade. Private kernel patches accumulate one reasonable change at a time, and the cost appears at the fiftieth, not the first.
• Downstream delta is more than patch files. It includes Kconfig, device tree, modules, firmware assumptions, build changes and private interfaces that user space depends on.
• Classify before you argue. Out-of-tree modules and configuration drift are predictable; patches to upstream files, reverts, forks and pre-upstream backports compound; binary modules, private user-space interfaces, lost provenance and certification lock can stop an upgrade entirely.
• Where the code sits matters more than how much of it there is, because the maintenance cost of a patch to a file you do not own is set by upstream’s activity in that file. A module is separable, not stable — it still needs maintenance when its interfaces move.
• Android published what this costs at scale: up to half the kernel code out of tree, security backporting that was “prohibitively expensive”, and ninety per cent of reported kernel security issues already fixed for devices that simply stayed current.
• Measure it. Commit distance against an upstream tag, a grep for missing Upstream-Status, and a check of the generated .config rather than the fragment. All three take minutes.
• Give every item six fields: base commit, owner, reason, upstream status, the test that covers it, and the condition that removes it. A patch with an exit condition is a task. A patch without one is permanent.

Frequently asked questions

Are private kernel patches always a mistake?
No. Hardware ships before its upstream support does, and some patches are legitimately not suitable for upstream. The mistake is carrying them without recording why they exist and what would remove them, because an unexplained patch cannot be dropped safely or forward-ported correctly.

Why is a patch to an existing kernel file worse than an out-of-tree module?
Because of who controls the conflict rate. A module conflicts when the interfaces it uses change. A patch to an upstream file conflicts whenever anyone modifies that file, so your maintenance cost is set by upstream’s activity in that area rather than by your own work. A module is not maintenance-free, though; it is simply separable.

Is reverting an upstream commit downstream ever acceptable?
Yes, as a short-term response to a verified regression. Record the reverted commit, the affected hardware, the failure signature, the upstream bug, the test that catches it, an owner and a removal condition. Without those fields it is not a workaround, it is untracked divergence, and it silently un-does whatever the original commit fixed.

What makes an upgrade genuinely impossible rather than just expensive?
Four things: binary-only modules you cannot rebuild, private user-space interfaces that closed applications depend on, patches whose purpose nobody can explain and therefore nobody will remove, and a certification or qualification bound to one specific build.

How do I find the undocumented part of my delta quickly?
In an OpenEmbedded or Yocto layer, list every patch file with no upstream status recorded, using grep with the files-without-match option against the recipes directory. Each result is a patch nobody has taken responsibility for, and that list is usually the honest measure of the problem.

Further reading

• Generic Kernel Image (GKI) project — Android’s own account of what fragmentation cost it, and the structural fix.
• Maintain a stable kernel module interface — the constraints that make Android’s KMI possible, and why they do not apply to mainline.
• Yocto Project Recipe Style Guide — the Upstream-Status values and what each one means.
• The Linux Kernel Driver Interface — why there is no stable in-kernel API, in the maintainers’ own words.
• Building External Modules — how an out-of-tree module is built and installed.