Shrinker Debugfs Shows Which Kernel Cache Holds Memory

If you have ever watched kernel memory grow on a board over days and had no way to say which cache was holding it, the shrinker debugfs interface is what you were missing. It has been in the kernel since 6.0, it is in every longterm series your product is likely to ship on, and it is almost certainly not compiled into your image.

The memory you cannot account for

Start with the situation as it actually appears. A gateway has been up for three weeks. Free memory has been falling steadily and the application has not grown. The first thing you read is /proc/meminfo:

raghu@techveda.org:~$ grep -E 'MemFree|Slab|SReclaimable|SUnreclaim' /proc/meminfo
MemFree:           41284 kB
Slab:             119640 kB
SReclaimable:      96228 kB
SUnreclaim:        23412 kB

Ninety-six megabytes are marked reclaimable on a board with half a gigabyte of RAM. The word “reclaimable” is reassuring and almost useless: it says the kernel believes it can get that memory back, not which subsystem holds it, nor whether that is legitimate caching or a reference never to be dropped.

The next step is normally /proc/slabinfo or slabtop:

raghu@techveda.org:~$ head -4 /proc/slabinfo
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables ...
dentry             184512  184884    192   21    1 : tunables ...
ubifs_inode_slab    31204   31320    736   22    4 : tunables ...

This is better, but it answers a different question. Slab accounting is organised by allocator cache, not by the code that decides when those objects may be freed. It tells you that 184,512 dentries exist, not how many a reclaim pass would release, and it does not connect them to the subsystem that owns the release decision — on a board with several filesystems mounted, that dentry count is one pooled number covering all of them.

So the usual next move is the blunt one, echo 3 > /proc/sys/vm/drop_caches. That drops every reclaimable cache at once. It is destructive to performance, it is not something you run on a device in the field, and when memory falls afterwards you have learned almost nothing: some cache gave memory back, but not which one and not how much. If memory does not fall, you still do not know which cache ignored the request. This is the gap the shrinker debugfs files close.

What a shrinker is, and why the count is not a number the kernel already keeps

A shrinker is the kernel’s registration mechanism for “I am holding objects that can be freed if memory gets tight”. A subsystem fills in a struct shrinker with two callbacks: count_objects() answers how many freeable objects exist right now, and scan_objects() is asked to free a given number of them. Reclaim walks the registered shrinkers, calls count_objects() on each to size the pool, works out a proportional amount of pressure, then calls scan_objects() with that number.

The important consequence is that the object count is not a statistic the kernel keeps in a variable; it is produced on demand by calling into the owning subsystem. That is why no file existed before 6.0 to read it from — there was nothing accumulated to print — and why reading the new file is not free.

Two flags change what the count means: SHRINKER_NUMA_AWARE for per-node tracking and SHRINKER_MEMCG_AWARE for per-cgroup tracking. A superblock shrinker sets both; the zsmalloc pool behind zram sets neither, and that difference produces one of the traps below.

The interface Linux 6.0 added

Commit 5035ebc644ae, from Roman Gushchin on 31 May 2022, created /sys/kernel/debug/shrinker/ with one directory per registered shrinker. Two follow-ups in the same series completed it: e33c267ab70d gave shrinkers readable names, replacing bare numbers, and bbf535fd6f06 added scan so you could drive a shrinker rather than only observe it.

Each directory is named by joining the shrinker’s name and an allocated id with a hyphen, from snprintf(buf, sizeof(buf), "%s-%d", shrinker->name, id). The id comes from an IDA allocator in registration order, so it is not stable across boots or configurations. A script that hardcodes sb-ubifs-19 breaks on the next boot; match on the name prefix instead.

Inside each directory there are two files with deliberately asymmetric permissions: count at mode 0440, readable only, and scan at mode 0220, writable only. You cannot read scan, and that is intentional, because the write is an action rather than a setting.

The names tell you immediately whether the board is holding filesystem metadata, compressed swap or transparent huge pages. Read from the registering call sites at v6.12.113 and v7.2.11: superblock shrinkers register as sb-<fstype>, the workingset shadow-entry shrinker as mm-shadow, the zsmalloc pool as mm-zspool:<poolname>, which for zram is the disk name, so mm-zspool:zram0. Transparent huge pages register thp-zero and thp-deferred_split, and UBIFS registers ubifs-slab. Some subsystems rename their entry after registration to append the backing device, which is why the kernel’s documented listing shows sb-xfs:vda1-36 rather than plain sb-xfs.

Reading the count file

Each line of count is a cgroup identifier followed by one object count per NUMA node:

raghu@techveda.org:~$ cat /sys/kernel/debug/shrinker/sb-btrfs:vda2-24/count
1 212 0
21 97 0
55 802 5
2367 2 0
225 13 0

Three behaviours in that output are not obvious and all three are visible in the source. First, a line whose total across all nodes is zero is omitted entirely, which the introducing commit explains as keeping the output small on machines with thousands of cgroups. An empty count file therefore means the cache is holding nothing, not that the file is broken.

Second, if the shrinker is not NUMA-aware, only node 0 is actually queried and zeros are printed for every other node: the counting loop tests nid == 0 || (shrinker->flags & SHRINKER_NUMA_AWARE) before calling into the subsystem. On a single-node board, which covers most embedded systems, there is one number per line and the distinction never shows. Third, SHRINK_EMPTY is normalised to 0, so “no objects” and “declined to answer” look the same.

Making one cache give memory back

The scan file takes exactly three numbers: the cgroup identifier, the NUMA node id, and the number of objects to scan. The parse is a single sscanf(kbuf, "%llu %d %lu", ...) that returns -EINVAL unless all three convert. The kernel buffer is 72 bytes, so there is no room for anything elaborate.

The worked before-and-after, taken from the kernel’s own documentation for this interface, is the whole point of the feature:

raghu@techveda.org:~$ cd /sys/kernel/debug/shrinker/sb-btrfs:vda2-24
raghu@techveda.org:~$ head -3 count
1 212 0
21 97 0
55 802 5
raghu@techveda.org:~$ echo "55 0 200" > scan
raghu@techveda.org:~$ head -3 count
1 212 0
21 96 0
55 752 5

Two hundred objects were offered for scan in cgroup 55 on node 0, and that cgroup’s node-0 count fell from 802 to 752; the single-object drop on cgroup 21 is unrelated background activity in the same capture. Fifty were released, not two hundred, because scan_objects() frees what it can and keeps the rest. That gap is itself diagnostic: a cache that returns almost nothing when scanned is holding pinned objects that no amount of memory pressure will recover.

One silent behaviour to know about: a scan count of zero is accepted and does nothing, because the write returns success before reaching the shrinker. So echo "55 0 0" > scan looks like it worked and had no effect — a confusing five minutes if the value came from an empty variable.

The first field is a trap on two kinds of board

The documentation states that for a shrinker which is not memcg-aware, or on a system with no memory cgroups, 0 is printed as the cgroup identifier and 0 is what you pass to scan. The write path matches that: for a non-memcg-aware shrinker, any first field other than 0 is rejected with -EINVAL. The read path does not. The counting function prints mem_cgroup_ino(memcg) using the loop variable, which is the root memory cgroup whenever memory cgroups are compiled in, and the root cgroup’s identifier is 1, not 0. It prints 0 only when mem_cgroup_iter() returns NULL, which happens when CONFIG_MEMCG is off or the controller was disabled at boot. So for a non-memcg-aware shrinker such as mm-zspool:zram0 or ubifs-slab, write 0 regardless of what count showed you. This conclusion comes from reading the source at the tags listed below and was not confirmed on hardware.

The second trap is a version difference. In Linux 7.0 the first column stopped being a cgroup inode number and became a cgroup ID. Commit ea73e3647160, from Shakeel Butt on 25 December 2025, switched the helper from cgroup_ino() to cgroup_id() and widened it to u64; commit 95296536eb19 then renamed the helpers to match. The commit message is explicit about the consequence: “The cgroup_ino() returns the kernfs inode number while cgroup_id() returns the kernfs node ID. For 64-bit systems, they are the same.” On a 64-bit board nothing changes. On a 32-bit Arm board the two values differ, so a script that obtains a cgroup inode number itself and feeds it to scan works on 6.19 and earlier, and from 7.0 onwards either addresses the wrong cgroup or fails with -ENOENT. The kernel documentation at v7.2.11 still describes the field as a cgroup inode id, so the documentation has not caught up with the code.

Both traps have the same defence: read the first column out of count and pass it back verbatim, except for non-memcg-aware shrinkers where you pass 0. Never compute it yourself.

Which kernels have the shrinker debugfs interface

Checked by reading mm/shrinker_debug.c and lib/Kconfig.debug at real point-release tags rather than inferring from release dates:

• 5.10.272 and 5.15.223 — the file does not exist. The interface is not available at any configuration on these series.
• 6.1.190, 6.6.159, 6.12.113, 6.18.57 — present, with the cgroup inode number as the first field.
• 7.2.11 — present, with the cgroup ID as the first field.

On the mainline side, mm/shrinker_debug.c is absent at v5.19 and present at v6.0, so 6.0 is the introducing release, and the identifier change brackets just as cleanly: v6.19 uses the inode number, v7.0 the ID. Within 6.1 through 6.18 the count format, the scan parse, the zero-line omission, the buffer size and both file permissions are unchanged, so there is nothing to upgrade for inside that span.

The Kconfig entry is the reason the shrinker debugfs interface is not already on your board, and it has an oddity worth recording. The commit that introduced CONFIG_SHRINKER_DEBUG added it with default y, and its message says the feature is “enabled by default”. No released tag from v6.0 through v7.2 has a default line at all, which makes the symbol default-off in every shipped kernel. Which commit removed the default was not identified, so treat the discrepancy as observed rather than explained. The dependency is just depends on DEBUG_FS, with no DEBUG_KERNEL requirement.

How to check your own board

Start with the configuration, because the directory will simply be absent if the symbol is off and that looks identical to an old kernel:

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E 'CONFIG_MEMCG=|CONFIG_DEBUG_FS=|CONFIG_SHRINKER_DEBUG'
CONFIG_MEMCG=y
CONFIG_DEBUG_FS=y
# CONFIG_SHRINKER_DEBUG is not set
raghu@techveda.org:~$ uname -r
6.12.113

That last line is the answer, and on a 6.12 kernel it is the expected default. Without /proc/config.gz, the same grep against the .config in your build tree works. To turn it on, add CONFIG_SHRINKER_DEBUG=y alongside CONFIG_DEBUG_FS=y to a Yocto kernel configuration fragment or the Buildroot kernel fragment file, and rebuild.

On a kernel that has it, mount debugfs and list what is registered:

raghu@techveda.org:~$ mount -t debugfs none /sys/kernel/debug
raghu@techveda.org:~$ ls /sys/kernel/debug/shrinker/
mm-shadow-18  mm-zspool:zram0-34  sb-proc-25  sb-tmpfs-1  thp-deferred_split-10  ubifs-slab-19
raghu@techveda.org:~$ for d in /sys/kernel/debug/shrinker/*/; do echo "$(basename "$d"): $(tr -s '[:space:]' ' ' < "$d/count")"; done
mm-shadow-18:
mm-zspool:zram0-34: 1 18442
sb-proc-25:
sb-tmpfs-1: 1 96
thp-deferred_split-10:
ubifs-slab-19: 1 48213

That last loop is the one to keep: it prints every shrinker beside everything it holds, and the three empty lines are the zero-omission rule at work rather than a failure. On a board whose kernel memory has been climbing, the line with an unexpectedly large number is your answer. Then scan a bounded number of objects from that cache:

raghu@techveda.org:~$ cat /sys/kernel/debug/shrinker/ubifs-slab-19/count
1 48213
raghu@techveda.org:~$ echo "0 0 1000" > /sys/kernel/debug/shrinker/ubifs-slab-19/scan
raghu@techveda.org:~$ cat /sys/kernel/debug/shrinker/ubifs-slab-19/count
1 47380

Note the 0 in the write even though count printed 1, because ubifs-slab is registered with no flags and so is not memcg-aware. Eight hundred and thirty-three objects came back out of a thousand offered, which says this cache is behaving and the memory really is available under pressure. A cache that gives back nothing is the one to investigate. Attribution work of this kind on real hardware is ground we cover in our training programmes.

What it costs, and where it misleads

Reading count on a memcg-aware shrinker iterates every online memory cgroup and calls into the subsystem once per cgroup per node, which is why the loop checks signal_pending() and returns -EINTR so you can interrupt it. On an embedded board with a handful of cgroups it is cheap, but it is still a call into live code holding locks rather than a counter read, so it does not belong in a one-second polling loop shipped to the field. The code path has also had real bugs over its life, including a deadlock fix, a double-free on the shrinker name and, more recently, a NULL pointer dereference fix — normal history for a debug interface, and a reasonable argument for enabling it on engineering images rather than production ones.

Three limits are worth stating plainly. Writing to scan frees objects, so you are modifying the system you are measuring; a before-and-after comparison is valid, a repeated scan is not. Enabling it requires CONFIG_DEBUG_FS, and debugfs on a production image is a security decision. And the shrinker debugfs files report what is held, not what leaked: a cache that is large because reclaim has never been under enough pressure to visit it looks exactly like a cache with a reference leak. The scan file is how you tell those apart, which is why the write side matters more than the read side.

What was checked, and what was not

No hardware was used. The count and scan transcript for the btrfs superblock shrinker is the kernel’s own documented example and is reproduced as such. The /proc/meminfo, /proc/slabinfo, directory listing and UBIFS transcripts are constructed to the verified output formats rather than captured from a board, and their numbers are illustrative. Everything about the interface itself — directory naming, both file permissions, the count format and zero-line omission, the NUMA gating, the three-field scan parse, the zero-scan no-op, the non-memcg first-field rejection, the Kconfig text, and per-series presence — was read from mm/shrinker_debug.c, lib/Kconfig.debug, include/linux/cgroup.h and mm/memcontrol.c at the tags named above. The claim that the root cgroup identifier reads as 1 rather than 0 follows from the source and from the documented example output, and was not confirmed on a board. Which commit removed default y was not established.

Key takeaways

• The shrinker debugfs interface reports per-cache reclaimable object counts and lets you scan one cache on demand, which is what /proc/slabinfo and drop_caches cannot do.
• It has existed since Linux 6.0 and is in every longterm series from 6.1 onwards, but CONFIG_SHRINKER_DEBUG is default-off in every released kernel, so you have to build it in. On 5.10 and 5.15 the code is absent entirely, so there it is not a configuration question.
• Always copy the first column out of count rather than computing it, and pass 0 for shrinkers that are not memcg-aware.
• From Linux 7.0 that column is a cgroup ID rather than an inode number. The two differ only on 32-bit kernels, which is exactly where many embedded products still live.

Related: kernel memory attribution as a product decision and the half of an AI model the kernel cannot reclaim.

Frequently asked questions

Does my kernel have the shrinker debugfs interface?
If your kernel is 6.1 or newer the code is present, including every longterm series from 6.1 through 7.2. Whether the interface exists on your board depends on CONFIG_SHRINKER_DEBUG, which has no default value in any released kernel and so is off unless someone set it. Kernels on the 5.10 and 5.15 series do not contain the code at all.

Why is the count file empty?
Because a line whose object total across all NUMA nodes is zero is left out of the output on purpose, and if every line would be zero the file reads as empty. An empty count file means the cache is holding nothing reclaimable at that moment, not that the interface is broken.

Is the shrinker debugfs interface safe to leave enabled on a production device?
It requires CONFIG_DEBUG_FS, and the files are root-only, so the real question is whether debugfs belongs on your production image at all. Reading count calls into live subsystem code rather than reading a counter, so it is not suitable for frequent polling, and writing to scan frees memory. Enabling it on engineering images is the safer default.

References

• mm: shrinkers: introduce debugfs interface for memory shrinkers (5035ebc644ae)
• mm: shrinkers: provide shrinkers with names (e33c267ab70d)
• mm: shrinkers: add scan interface for shrinker debugfs (bbf535fd6f06)
• memcg: use cgroup_id() instead of cgroup_ino() for memcg ID (ea73e3647160)
• memcg: rename mem_cgroup_ino() to mem_cgroup_id() (95296536eb19)
• mm: shrinkers: fix debugfs file permissions (2124f79de6a9)
• Shrinker Debugfs Interface — Linux kernel administrator’s guide
• kernel.org active release list (releases.json)