Dynamic Debug in Linux: Turn On pr_debug Messages at Runtime

Most driver source is full of debug messages that never appear. pr_debug() and dev_dbg() print nothing unless the kernel was built with CONFIG_DYNAMIC_DEBUG, and even then each callsite starts disabled. Engineers often respond by adding printk() calls and rebuilding, which on an embedded target means a build, a copy to the board, and a reboot for every guess. Dynamic debug removes that cycle: the messages are already compiled in, and you enable exactly the ones you want at runtime. Everything below is reproducible in about fifteen minutes.

What you need

• A kernel built with CONFIG_DYNAMIC_DEBUG=y (most desktop distributions and most vendor BSP kernels have it).
• Root access — the control file is writable only by root.
• Kernel headers for the running kernel, to build the demonstration module: usually linux-headers-$(uname -r) on Debian and Ubuntu, or kernel-devel on Fedora and RHEL.
• Basic familiarity with make, insmod and dmesg.

Check that dynamic debug is available

The quickest check is whether the control file exists:

raghu@techveda.org:~$ ls -l /proc/dynamic_debug/control
-rw-r--r-- 1 root root 0 Jul 21 09:12 /proc/dynamic_debug/control

If that file is missing, look for it under debugfs instead. Older kernels expose the control file only there:

raghu@techveda.org:~$ sudo mount -t debugfs none /sys/kernel/debug
raghu@techveda.org:~$ ls -l /sys/kernel/debug/dynamic_debug/control

If neither path exists, the feature is not compiled in. Confirm from the running kernel’s configuration:

raghu@techveda.org:~$ zcat /proc/config.gz | grep DYNAMIC_DEBUG
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DYNAMIC_DEBUG_CORE=y

Not every kernel exports /proc/config.gz. If yours does not, check /boot/config-$(uname -r) instead.

Read the catalog before you change anything

The control file is both the command interface and a catalog of every debug callsite the kernel knows about. Reading it first tells you what you are allowed to select:

raghu@techveda.org:~$ sudo head -n 4 /proc/dynamic_debug/control
# filename:lineno [module]function flags format
init/main.c:1179 [main]initcall_blacklist =_ "blacklisting initcall %s\012"
init/main.c:1218 [main]initcall_blacklisted =_ "initcall %s blacklisted\012"
init/main.c:1424 [main]run_init_process =_ "  with arguments:\012"

Each line carries the source file, line number, module name in brackets, function name, current flags, and the format string. The flags field is preceded by = so it is easy to filter. =_ is the default and means the callsite prints nothing; =p means it is enabled. The newline inside a format string is shown as the octal escape \012, so searching the catalog for a literal newline will not work.

Build a small module to practise on

Working on a module you wrote makes the effect easy to see. Create a directory with two files. First hello_dyndbg.c:

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

#include <linux/module.h>
#include <linux/init.h>
#include <linux/kernel.h>

static int __init hello_dyndbg_init(void)
{
	pr_info("module loaded\n");
	pr_debug("init: first debug callsite\n");
	pr_debug("init: second debug callsite, value = %d\n", 42);
	return 0;
}

static void __exit hello_dyndbg_exit(void)
{
	pr_debug("exit: about to unload\n");
	pr_info("module unloaded\n");
}

module_init(hello_dyndbg_init);
module_exit(hello_dyndbg_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("raghu@techveda.org");
MODULE_DESCRIPTION("Dynamic debug demonstration module");

The pr_fmt macro prefixes every message from this file with the module name — a habit worth keeping in real drivers. Then the Makefile, whose recipe lines must begin with a real tab character, not spaces:

obj-m := hello_dyndbg.o

KDIR ?= /lib/modules/$(shell uname -r)/build

all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules

clean:
	$(MAKE) -C $(KDIR) M=$(PWD) clean

Build and load it:

raghu@techveda.org:~$ make
raghu@techveda.org:~$ sudo insmod hello_dyndbg.ko
raghu@techveda.org:~$ dmesg | tail -n 2
[  912.443210] hello_dyndbg: module loaded

Only the pr_info() line appears. Both pr_debug() callsites ran and printed nothing, because their flags are still =_. That is the situation dynamic debug exists to fix.

Enable the messages with a control file query

A command written to the control file is a set of match specifications followed by a flags change. The match keywords are module, file, func, line, format and class; those you give are combined with AND, and a keyword you leave out means “any”. The flags change is + to add, - to remove, or = to set exactly. Confirm the callsites are catalogued, then turn them on:

raghu@techveda.org:~$ sudo grep hello_dyndbg /proc/dynamic_debug/control
hello_dyndbg.c:10 [hello_dyndbg]hello_dyndbg_init =_ "init: first debug callsite\012"
hello_dyndbg.c:11 [hello_dyndbg]hello_dyndbg_init =_ "init: second debug callsite, value = %d\012"
hello_dyndbg.c:17 [hello_dyndbg]hello_dyndbg_exit =_ "exit: about to unload\012"

raghu@techveda.org:~$ sudo sh -c "echo 'module hello_dyndbg +p' > /proc/dynamic_debug/control"

The line numbers in your own catalog will differ; use whatever your build reports. The sh -c wrapper is needed because sudo applies to the command, not to the shell redirection. The module’s init function has already run, so reload it to see the effect:

raghu@techveda.org:~$ sudo rmmod hello_dyndbg
raghu@techveda.org:~$ sudo insmod hello_dyndbg.ko
raghu@techveda.org:~$ dmesg | tail -n 3
[  980.112040] hello_dyndbg: module loaded
[  980.112044] hello_dyndbg: init: first debug callsite
[  980.112047] hello_dyndbg: init: second debug callsite, value = 42

The timestamps are illustrative. To narrow the selection, add more match keywords. These are all valid queries:

raghu@techveda.org:~$ sudo sh -c "echo 'file hello_dyndbg.c func hello_dyndbg_exit +p' > /proc/dynamic_debug/control"
raghu@techveda.org:~$ sudo sh -c "echo 'module hello_dyndbg line 10-11 +p' > /proc/dynamic_debug/control"
raghu@techveda.org:~$ sudo sh -c "echo 'format \"second debug callsite\" +p' > /proc/dynamic_debug/control"

A line range must not contain spaces: 10-11 is accepted, 10 - 11 is not. The format keyword matches any part of the format string, not the whole of it, which makes it useful when you know the text of a message but not the file it lives in.

Add context with decorator flags

p is the flag that enables a callsite. The rest are decorators that prepend context to each printed message. On current kernels the set is:

• t — thread ID, or <intr> for interrupt context
• m — module name
• f — function name
• s — source file name
• l — line number
• d — call trace
• _ — no flags

So +pmf enables the callsite and prefixes the module and function names, and +pt adds the thread ID, which is what you want when you suspect a message is coming from interrupt context. The decorators are prepended in the order listed above, but the exact separator characters have changed between kernel versions, so read your own dmesg output rather than assuming a layout.

The flag set itself is version-dependent. Current kernels accept ^[-+=][fslmptd_]+$; kernels from the 4.x and 5.x era accept only ^[-+=][flmpt_]+$, with no s and no d. An unsupported flag character is a common cause of a rejected write, and the kernel logs a precise reason:

raghu@techveda.org:~$ sudo sh -c "echo 'module hello_dyndbg +po' > /proc/dynamic_debug/control"
sh: echo: write error: Invalid argument
raghu@techveda.org:~$ dmesg | tail -n 2
[ 1042.556119] dyndbg: unknown flag 'o'
[ 1042.556121] dyndbg: flags parse failed

Always read dmesg after a failed write — the parser names the keyword or flag it rejected, which is more useful than the shell’s message.

Apply it to a driver you did not write

The same commands work on any in-tree code. If a USB device is not enumerating, or an I2C transfer is failing, the subsystem almost certainly already has the messages you need:

raghu@techveda.org:~$ sudo sh -c "echo 'file drivers/usb/core/* +p' > /proc/dynamic_debug/control"
raghu@techveda.org:~$ sudo sh -c "echo 'module i2c_core +p' > /proc/dynamic_debug/control"

Quoting matters here. The wildcard is expanded by the kernel’s matcher, not by your shell, so the pattern stays inside quotes. Enable a narrow selection, reproduce the failure, then disable it again — a broad query such as +p with no match specification turns on every callsite in the kernel and produces more log output than you can read.

For probe or timing problems rather than message-level ones, dynamic debug pairs well with function tracing; see our hands-on ftrace tutorial. Both are part of the driver debugging practice we teach in the Linux Device Drivers course.

Getting the messages onto the console

Enabling a callsite sends its output to the kernel log ring buffer. Whether it also reaches your console depends on the log level: debug messages are KERN_DEBUG, level 7, and most systems do not print level 7 to the console. The reliable approach on a development machine is to watch the ring buffer directly:

raghu@techveda.org:~$ sudo dmesg -w

If you need the messages on a serial console during boot, raise the console log level with the loglevel=8 kernel parameter, or use ignore_loglevel to print everything regardless.

Enabling messages at boot and at module load

Some failures happen before you can log in. For those, pass the query on the kernel command line: a bare dyndbg= applies to built-in code, and module.dyndbg= applies to one module.

dyndbg="file drivers/base/* +p"
i2c_core.dyndbg="+p"
pc87360.dyndbg="func pc87360_init_device +p"

The query must not exceed 1023 characters, and your bootloader may allow less. If the named module is not built in, the parameter is stored at boot and reapplied when the module is loaded later. You can also pass it at load time:

raghu@techveda.org:~$ sudo modprobe hello_dyndbg dyndbg=+pmf

Or place a line such as options hello_dyndbg dyndbg=+pt in a file under /etc/modprobe.d/. These sources are applied in order — modprobe.d first, then boot arguments, then modprobe arguments — with the last one taking effect. Note that dyndbg is a fake module parameter: every module accepts it, and it does not appear under /sys/module/<name>/parameters/.

Dynamic debug on space-constrained targets

The catalog costs memory, roughly in proportion to the number of debug callsites in the image, which is a real consideration on a small target. The kernel provides a middle path: set CONFIG_DYNAMIC_DEBUG_CORE=y without CONFIG_DYNAMIC_DEBUG, which builds the mechanism but not the full catalog, then opt individual modules in by adding ccflags := -DDYNAMIC_DEBUG_MODULE to their Makefile. You keep runtime control over the drivers you are working on and pay nothing for the rest.

Turn it off and clean up

Disable what you enabled, then remove the module:

raghu@techveda.org:~$ sudo sh -c "echo 'module hello_dyndbg -p' > /proc/dynamic_debug/control"
raghu@techveda.org:~$ sudo sh -c "echo 'module hello_dyndbg =_' > /proc/dynamic_debug/control"
raghu@techveda.org:~$ sudo rmmod hello_dyndbg
raghu@techveda.org:~$ make clean

The first command removes the p flag but leaves any decorators; the second clears the flags entirely, which is the cleaner reset. Settings written to the control file do not survive a reboot, so anything you want to keep belongs on the kernel command line or in /etc/modprobe.d/.

Key takeaways

• Dynamic debug controls existing pr_debug() and dev_dbg() callsites at runtime, so there is no rebuild and no reboot in the loop.
• Read the catalog at /proc/dynamic_debug/control first — it tells you which module, file, function and line values are selectable.
• A query is match specifications plus a flags change; leaving out a keyword means “any”, so always give at least one match specification.
• Flag p enables a callsite; t m f s l d are decorators, and the older flag set is only f l m p t _.
• After a rejected write, read dmesg: the parser names the keyword or flag it did not accept.
• Use dyndbg= on the kernel command line for problems that occur before userspace is up.

Frequently asked questions

Why does my pr_debug message still print nothing after I enabled it?
Two common reasons. The message may have gone to the kernel log but not to your console, because debug messages are log level 7 and most consoles are set lower; check with dmesg. Or the code that contains the callsite has not run again since you enabled it — reload the module or repeat the operation that triggers it.

What is the difference between CONFIG_DYNAMIC_DEBUG and CONFIG_DYNAMIC_DEBUG_CORE?
CONFIG_DYNAMIC_DEBUG builds the full catalog of callsites for the whole kernel and enables the core mechanism. CONFIG_DYNAMIC_DEBUG_CORE enables the mechanism only, without the full catalog, so modules opt in individually with ccflags := -DDYNAMIC_DEBUG_MODULE. The second is useful when image size matters.

Do my control file settings survive a reboot?
No. Anything written to the control file is lost on reboot. To make a setting persistent, put the query in the kernel command line as dyndbg= or module.dyndbg=, or add an options <module> dyndbg=... line under /etc/modprobe.d/.

Can I enable messages in a module before it is loaded?
Yes. Pass <module>.dyndbg="QUERY" on the kernel command line. The kernel stores the query and applies it when the module is loaded. You can also pass dyndbg= directly as a modprobe argument.

Further reading

• Dynamic debug — Linux kernel documentation (admin-guide)
• Dynamic debug — Linux 4.19 documentation, useful for comparing the older flag set and debugfs-only control file
• The kernel’s command-line parameters, for loglevel, ignore_loglevel and dyndbg