Kernel Memory Attribution Is a Product Decision, Not a Debugging Trick

A device that loses memory over days is one of the least tractable failures in embedded Linux. The symptom is slow, the reproduction window is long, and by the time the OOM killer starts taking user-space processes the evidence that would have identified the cause is gone. The tools most products ship narrow the problem to a slab cache and stop there, which leaves a field engineer holding a size class rather than an owner.

Memory allocation profiling closes that last step. This article is not about how to run it — it is about what it costs, what it can be wired into, and why the kernel version in your BSP decides whether the option exists at all.

Where meminfo and slabinfo stop, and memory allocation profiling starts

Every product can already read /proc/meminfo and /proc/slabinfo. Both are useful and both stop one step short.

root@beaglebone:~# grep -E "MemFree|Slab|SUnreclaim" /proc/meminfo
MemFree:           61236 kB
Slab:              97412 kB
SUnreclaim:        85324 kB
root@beaglebone:~# sort -k3 -rn /proc/slabinfo | head -3
kmalloc-512        61440  61440    512    8    1 : tunables ...
kmalloc-256        18944  18944    256   16    1 : tunables ...
dentry              9114   9114    192   21    1 : tunables ...

Eighty-five megabytes of unreclaimable slab, most of it in kmalloc-512. That names a size class, not an owner, and the gap is structural rather than a reporting deficiency. Every caller in the kernel requesting between 257 and 512 bytes lands in that cache: network drivers, filesystem code, the block layer, and whatever the product added. Attribution to an owner is a different question, and the slab statistics were never designed to answer it.

The usual escalation is CONFIG_DEBUG_KMEMLEAK, which does record stack traces. But kmemleak answers whether an object is unreachable, which is narrower than which code owns the memory currently in use. On a fleet device the second question is asked far more often, because a large share of memory regressions are not leaks in the free-the-memory sense. They are caches that grow without bound under a workload nobody tested, and a leak detector reports those as healthy.

How memory allocation profiling makes the attribution

Memory allocation profiling is built on a small library called code tagging, and the mechanism explains both why the numbers can be trusted and what they cost.

At each tagged allocation site the compiler emits a static struct alloc_tag into a dedicated ELF section. The tag carries the file name, line number, function and module name, filled in from __FILE__, __LINE__, __func__ and KBUILD_MODNAME. The attribution is therefore a compile-time fact rather than a runtime stack walk, which is why it is cheap and why the result is a line rather than a symbol.

That distinction is the design decision worth understanding. A return address resolves to a symbol, and a symbol is a function, not a line, so a driver allocating in three places inside one function would collapse into a single entry. Placing a static tag at the call site keeps the file and line, and it survives the compiler inlining the surrounding code.

Counting is per-CPU and lock-free. Each tag points at a per-CPU pair of 64-bit counters, bytes and calls; an allocation adds and a free subtracts. There is no shared cache line on the fast path, which is the basis for the kernel documentation’s claim that the overhead suits production use. I have not benchmarked that claim on hardware, and neither the documentation nor the changelog is a measurement of your workload.

The last piece is a static key. When the feature is compiled in but switched off, the added instructions are patched out of the stream rather than skipped by a branch. That is what makes it reasonable to ship the code on every unit and leave it dormant.

Reading the file on a real board

The output is already machine-readable and already access-controlled. The file is created with mode 0400, so it is root-only and cannot be read by an unprivileged process that has been compromised.

root@beaglebone:~# head -3 /proc/allocinfo
allocinfo - version: 1.0
#     <size>  <calls> <tag info>
       262144      512 mm/filemap.c:1919 func:__filemap_get_folio

The version line exists so the format can change without breaking parsers, which means a collection agent can refuse an unfamiliar format instead of silently misreading it. Each data line is a byte count, a call count, then the file path, line number, module name in square brackets when the site is in a loadable module, and the function name.

Filtering to one module is the common case, and the module name is the stable key:

root@beaglebone:~# grep -F "[acme_eth]" /proc/allocinfo | sort -g
         4096        8 drivers/net/acme_main.c:92 [acme_eth] func:acme_probe
        98304      192 drivers/net/acme_rx.c:311 [acme_eth] func:acme_rx_refill
      2097152     4096 drivers/net/acme_ring.c:184 [acme_eth] func:acme_ring_alloc

Two things about that output will trip up anyone building tooling against it. The path is whatever the compiler recorded, so it reflects the build directory and differs between builders; the bracketed module name does not. And the byte count is the size of the slab object the allocator handed out, not the size requested, because the slab path passes the cache’s object size into the accounting call. A 488-byte structure served from kmalloc-512 is accounted as 512 bytes, so a total computed from sizeof() will never match the file. That is the number you want for a memory budget, but it will be read as a tool defect the first time someone checks the arithmetic.

Finally, one reading identifies what is large; most large entries are legitimately large. Two readings around a representative workload identify what only ever grows, and core memory-management entries such as the page-extension allocation and the slab allocator’s own page requests are large on every healthy system. Because the accounting nests, those are already counted separately from whatever asked for the memory, so they must never be added to a driver’s total.

What memory allocation profiling costs

Memory allocation profiling is not free, and on a memory-constrained board the price needs stating in bytes.

The dominant cost is page extensions. Enabling the feature selects PAGE_EXTENSION unconditionally, and allocation tagging registers a page-extension client whose per-page size is exactly one pointer. Every page of RAM therefore acquires four extra bytes on a 32-bit target or eight on a 64-bit one. On a 512 MB BeagleBone Black that is roughly half a megabyte; on a 2 GB arm64 board, roughly four megabytes. Page extensions carry their own bookkeeping in addition, so treat those as a floor rather than a total.

Kernels from 6.13 can avoid that cost by packing the tag reference into spare page flags instead, requested with a boot parameter. The documented failure mode deserves a second reading before anyone enables it across a fleet: if there are not enough spare page flags, compression fails, a warning is issued, and memory allocation profiling is disabled entirely. It does not fall back to page extensions. A device can therefore boot with the option set and quietly have no profiling at all, which is exactly the kind of silent capability loss a fleet should be checking for rather than assuming.

The second cost is static and scales with the number of tagged call sites rather than with RAM: a tag structure plus a per-CPU counter pair for every annotated site in the build, present whether or not the counters are running.

There is also a configuration trap. The debug variant looks like a harmless extra, but it forces profiling on and makes the runtime control read-only, which removes the ability to switch the counters off on a running device. It belongs on a bench kernel while annotating a new allocator, not on anything you ship.

The shipping configuration follows from all of this: compile the feature in, and leave the enabled-by-default symbol unset so the static key patches the work out until a boot parameter turns it on.

CONFIG_MEM_ALLOC_PROFILING=y
# CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT is not set
# CONFIG_MEM_ALLOC_PROFILING_DEBUG is not set

Put those in a kernel configuration fragment rather than a hand-edited config, so the setting survives a clean build and is visible in review. The page-extension memory is still spent either way; that is the one part of the bill that cannot be deferred.

The unload check you can put in CI

The behaviour that gets the least attention is the one most worth automating. When the feature is enabled, the allocation-tag machinery registers a module unload callback. On rmmod it walks every tag belonging to that module and, for any tag whose counter is not zero, emits a warning naming the file, line, module, function and byte count, then reports that the module cannot unload cleanly.

root@beaglebone:~# rmmod acme_eth
[  418.552147] drivers/net/acme_ring.c:184 module acme_eth func:acme_ring_alloc has 2097152 allocated at module unload
[  418.552389] acme_eth: memory allocation(s) from the module still alive, cannot unload cleanly

What happens next is a correctness decision rather than an oversight. Because the allocation tags live in the module’s own data and the leaked objects still reference them, freeing that data would leave dangling pointers, so the kernel deliberately keeps it allocated. A load-unload loop over a leaking driver therefore loses more memory per iteration than the leak itself.

For a team maintaining out-of-tree drivers this is a ready-made test. A CI job that loads the module, exercises it, unloads it and fails the build when the kernel log contains an unload warning turns an entire class of driver defect into a deterministic pre-merge check. The signal is exact: the counter for that call site is not zero, and the message names the line. There is no sampling and no false positive from a structure the scanner could not walk. If you carry driver patches outside the mainline tree, this pairs directly with the maintenance cost discussed in The Hidden Cost of Out-of-Tree Drivers and Private Kernel Patches.

The version gap that decides everything

None of this is available to a large share of shipped devices, and it is the first thing to check rather than the last. Memory allocation profiling reached mainline in 6.10, which was never a longterm release, so the question that matters is which maintained branches carry it. Each row below was checked by fetching the documentation file from the branch tip rather than reasoning from the merge date.

A product on the 6.1 or 6.6 longterm series does not have this and cannot be configured into having it. That belongs in the upgrade case alongside the security and hardware-enablement arguments, with the advantage of being concrete: a named diagnostic capability the older kernel cannot provide is easier to put in front of people who do not read changelogs.

Two further gates are worth knowing. From 6.18 the option depends on MMU, so no-MMU targets are excluded, but there is no architecture restriction of any kind and both 32-bit ARM and arm64 are eligible. And a vendor BSP fork based on 6.6 does not acquire the feature by carrying backports of other things, so the kernel version in your BSP is the number that decides, not the mainline release it was branched from.

What to enable, and in what order

1. Audit which kernel each product line actually ships. If any are on 6.6 or older, this capability is unavailable to them, and that is an argument for the upgrade rather than an item for the observability backlog.
2. Compile the feature into every image with the enabled-by-default symbol unset, so it ships dormant and can be switched on by boot parameter on a single unit under investigation.
3. Add a collection step that captures the file twice around a representative workload and reports the delta, not the snapshot. Parse the version line and refuse unknown formats.
4. Add a CI job that loads, exercises and unloads every out-of-tree module the product ships, and fails on an unload warning.
5. If you enable the compressed mode, assert at boot that the file exists and is non-empty, so a failed compression cannot silently remove the capability from a fleet.

What not to do

• Do not enable the counters by default across a fleet on the assumption that the overhead is negligible. The documentation supports the claim; your workload has not been measured against it.
• Do not treat the largest entry as the fault. Core memory-management sites are large on healthy systems, and a reading without a baseline cannot separate large from growing.
• Do not build tooling around the file path in a tag. The path reflects whoever built the module; the module name is the stable key.
• Do not compute expected totals from sizeof(), because the accounting records the slab object size handed out.
• Do not enable the debug variant on anything you ship; it removes the ability to switch the counters off at runtime.
• Do not assume a vendor BSP has memory allocation profiling because its nominal mainline base does.

Key takeaways

• Memory allocation profiling attributes outstanding kernel memory to a file, line and function, which is the step /proc/slabinfo cannot take. The file is root-only at mode 0400.
• Attribution comes from a static tag emitted at each call site, not a stack walk, so it survives inlining and separates several allocations inside one function.
• The fixed price is one pointer of page-extension data per page of RAM. Kernels from 6.13 can avoid it with the compressed mode, which disables memory allocation profiling outright if it cannot find spare page flags.
• Ship it compiled in and dormant: a static key patches the work out until a boot parameter enables it.
• The unload warning turns leak detection into a pre-merge CI check for out-of-tree modules.
• Present from 6.12 onward and absent from 6.6, 6.1, 5.15 and 5.10, so the kernel version in your BSP decides whether any of this is available.

Frequently asked questions

Which kernels have memory allocation profiling?
It reached mainline in 6.10 and is present in the 6.12, 6.13, 6.18 and 7.2 stable series. It is absent from the 6.6, 6.1, 5.15 and 5.10 longterm branches, where it cannot be configured in at all. A vendor BSP based on 6.6 does not gain it from unrelated backports.

Can I ship it enabled on a production device?
The kernel documentation states the overhead is low enough for production use, and the design supports that with per-CPU lock-free counters. That is the documentation’s claim rather than a measurement of your workload, so benchmark before enabling memory allocation profiling by default.

What does it cost in RAM?
Enabling it selects page extensions, and allocation tagging adds one pointer per page of RAM: four bytes per page on a 32-bit target, eight on a 64-bit one. There is also a static tag and a per-CPU counter pair for every annotated allocation site in the build.

Why does the byte count not match my sizeof calculation?
The accounting records the size of the slab object the allocator handed out rather than the size requested, so a 488-byte structure served from the kmalloc-512 cache is accounted as 512 bytes.

How do I use it in continuous integration?
Load the module, exercise it, unload it, and fail the build if the kernel log contains an unload warning. The kernel names the file, line and byte count for any allocation site whose counter is not zero at unload.

Further reading

• Memory allocation profiling — Linux kernel documentation
• Linux 6.10 changelog — KernelNewbies
• Commit 22d407b164ff, “lib: add allocation tagging support for memory allocation profiling”
• lib/alloc_tag.c — the procfs interface and the module unload check
• include/linux/codetag.h — struct codetag and union codetag_ref
• kernel/module/main.c — free_module and module_memory_free
• lib/Kconfig.debug — the MEM_ALLOC_PROFILING entries