Reading a Kernel Oops, Part 2: Decode to the Source Line and Reason Backward

In Part 1 we read a kernel oops by eye: the header as triage, the faulting address as a fingerprint, and the call trace split into confirmed and unreliable frames. This part turns that reading into an exact location and then into a cause. The running example is the same NULL-dereference module from Part 1, whose init function faults at line 10, the statement *p = 42;.

What a kernel oops gives you to work with

You need three things: the saved oops text (from dmesg or journalctl -k), a kernel or module built with CONFIG_DEBUG_INFO, and the standard tools addr2line, nm, readelf, and gdb. Without debug information the tools can name the function but not the file and line. For an out-of-tree module, that means building the .ko with -g (for example ccflags-y := -g) so it keeps a .debug_line section.

Why raw addresses lie: KASLR and the build ID

Two facts make people decode the wrong thing. First, kernel address space layout randomisation (KASLR) relocates the kernel by a random offset at every boot, so a raw hex address like 0xffffffff8102abcd means nothing unless you also know that boot’s base. This is precisely why the tools work on symbol+offset instead: an offset within a function is invariant across boots. Copy the symbol+offset/size string, not the raw address.

Second, the symbols must come from the exact binary that crashed. Matching by version string is not enough; the reliable match is the build ID, a hash the toolchain embeds in every kernel and module. You can read it, and modern tooling can fetch the matching debug symbols by it:

raghu@techveda.org:~$ readelf -n ./oops_demo.ko | grep -A1 'Build ID'
    Owner                Data size 	Description
    GNU                  0x00000014	NT_GNU_BUILD_ID
        Build ID: 3f0a1c9b6e2d... 

If two builds share a version string but differ by even one config option, their build IDs differ, and decoding one oops against the other gives confident but wrong line numbers. When debug symbols are served by a debuginfod server, decode_stacktrace.sh can locate them automatically from the build ID in the dump.

Resolve one frame with faddr2line

scripts/faddr2line takes the symbol+offset/size string directly, which is exactly what the oops prints. Point it at the object that contains the symbol; for a fault in your module, that is the .ko you built with debug info:

raghu@techveda.org:~$ /usr/src/linux-headers-$(uname -r)/scripts/faddr2line ./oops_demo.ko oops_demo_init+0x11/0x1e
oops_demo_init+0x11/0x1e:
oops_demo_init at /home/raghu/oops_demo/oops_demo.c:10

Line 10 is *p = 42;. The size after the slash (/0x1e) is optional but recommended, because faddr2line uses it to disambiguate when two static functions share a name. For a fault inside the kernel image rather than a module, pass vmlinux instead of the .ko, for example faddr2line vmlinux 'do_one_initcall+0x5b/0x340'.

Decode the whole trace with decode_stacktrace.sh

Resolving frames one at a time is fine for a single fault, but a long trace is faster to read translated all at once. scripts/decode_stacktrace.sh reads an oops on standard input and rewrites each symbol+offset into a file and line. Its argument form, taken from the script itself, is:

decode_stacktrace.sh [-R] [<vmlinux> [<base_path>|auto [<modules_path>]]]

Pass the matching vmlinux, let the base path default to auto, and give the directory holding your module as the third argument so module frames resolve too:

raghu@techveda.org:~$ SCRIPTS=/usr/src/linux-headers-$(uname -r)/scripts
raghu@techveda.org:~$ $SCRIPTS/decode_stacktrace.sh /usr/lib/debug/boot/vmlinux-$(uname -r) auto /home/raghu/oops_demo < oops.txt
RIP: 0010:oops_demo_init (/home/raghu/oops_demo/oops_demo.c:10) oops_demo
do_one_initcall (init/main.c:1236)
do_init_module (kernel/module/main.c:2606)
load_module (kernel/module/main.c:3020)

A useful detail: the script keeps the unreliable ? frames from Part 1 but still resolves them, so you can see what they were without being misled into treating them as real callers. If a frame cannot be resolved you get WARNING! No debugging info in module oops_demo, rebuild with DEBUG_KERNEL and DEBUG_INFO, which means the object it found has no .debug_line section.

Follow the inlining

Part 1 noted that the function on the RIP line may be an inlined caller, with the real fault inside a small static helper folded into it. This is where you recover that. Both decode_stacktrace.sh and a direct addr2line -i print the full inline chain:

raghu@techveda.org:~$ addr2line -i -f -e vmlinux ffffffff81002abb
kmalloc_node
__kmem_cache_alloc_node
device_add

Read that from the bottom: the outer function is device_add, but the code that ran was an inlined kmalloc_node. Without the -i chain you would go looking in device_add for a bug that lives in the allocation helper. On heavily optimised kernels this is a routine source of confusion.

Cross-check with gdb

For a single address you can confirm the result against the vmlinux image with gdb, which is the method the kernel documentation recommends for faults in the core kernel:

raghu@techveda.org:~$ gdb /usr/lib/debug/boot/vmlinux-$(uname -r)
(gdb) list *do_one_initcall+0x5b
0xffffffff81002abb is in do_one_initcall (init/main.c:1236).
(gdb) quit

For the Code: line, scripts/decodecode disassembles the bytes around the faulting instruction and marks the trapping one, which confirms the exact operation and, read together with the register dump, the bad pointer value.

Reason backward from the crash to the defect

You have an exact line, but that line is where execution crashed. For our toy module the defect and the crash are the same statement, which is why the module is a poor teacher for this step and real bugs are not. In practice the pointer that faulted was made bad somewhere upstream, and your job is to find that point.

Three inputs drive the search. The faulting-address fingerprint from Part 1 tells you the bug class: 0x6b6b6b6b6b6b6b6b sends you looking for a free that happened too early, while a small offset sends you to a structure that was never allocated. The confirmed frames (the ones with no ?) give you the real path into the crash, so you can inspect each caller for where the pointer or object was set up. And ftrace can show how execution actually reached the crash site when the static call graph is not obvious; see our tutorial on function tracing with ftrace. For lifetime bugs specifically, enabling KASAN turns a vague use-after-free into a precise report naming both the free and the later access. Working through faults like these on real hardware is a core part of the TECH VEDA Linux Device Drivers course.

Why the machine is now unreliable

An oops is often called recoverable, which is misleading. When the fault is in process context and panic_on_oops is not set, the kernel kills the faulting task (through make_task_dead) and keeps running. But it does not unwind what that task was doing: any locks it held are still held, and any reference counts it raised are not restored. The result is leaked mutexes and refcount leaks that surface later as hangs or a second, unrelated-looking oops. That is the concrete reason to reboot after an oops rather than trust the system, and it is why a reproducer sometimes only fails on the second attempt.

For capturing a full crash for offline analysis, set panic_on_oops=1 so the first oops becomes a panic, and pair it with kdump to save a vmcore. On a healthy investigation you decode on a fresh boot, against build-ID-matched symbols, and treat the first oops as the real one.

Key takeaways

• Decode symbol+offset, not raw addresses: KASLR makes raw kernel addresses meaningless across boots.
• Match symbols by build ID, not just version string; readelf -n shows it and debuginfod can fetch by it.
• faddr2line resolves a single frame; decode_stacktrace.sh translates the whole trace; addr2line -i recovers inlined frames the dump hides.
• The decoded line is the crash site; reason backward using the fingerprint, the confirmed frames, ftrace, and KASAN to find the defect.
• After an oops the machine holds leaked locks and refcounts, so reboot before trusting it; use panic_on_oops with kdump to capture a full crash.

Frequently asked questions

Why decode symbol+offset instead of the raw address from the oops?
KASLR relocates the kernel by a random offset at each boot, so a raw address is only meaningful for that one boot. An offset within a named function does not change between boots, which is why faddr2line and decode_stacktrace.sh work on symbol+offset.

Is the same kernel version enough to decode an oops correctly?
No. Two builds can share a version string but differ in configuration, which changes addresses. The reliable match is the build ID embedded in the kernel and modules; decoding against a different build ID gives confident but wrong line numbers.

Why does the decoded line not look like where the bug is?
Because the oops marks where execution crashed, not where the defect was introduced. The faulting pointer was usually made bad earlier. Use the faulting-address fingerprint and the confirmed call frames to work backward to the point where the pointer or object went wrong.

Why is the system unreliable after an oops if it kept running?
The kernel kills the faulting task but does not release the locks it held or restore the reference counts it raised. Those leaks can cause later hangs or a second oops, so you should reboot rather than continue on the same boot.

Further reading

• Reading a Kernel Oops, Part 1: What the Dump Is Telling You
• Bug hunting — The Linux Kernel documentation
• The Kernel Address Sanitizer (KASAN) — The Linux Kernel documentation
• scripts/faddr2line (torvalds/linux)
• scripts/decode_stacktrace.sh (torvalds/linux)