Reading a Kernel Oops: What the Dump Is Telling You

A kernel oops is not a debugger sitting at a breakpoint; it is the kernel’s last report from a machine that has already violated one of its own invariants. The register and stack values were captured at the moment of the fault, and by the time they reach your screen the system may be inconsistent. Reading an oops well means treating it as evidence at a crime scene, not as an answer, and most of that reading can be done before you reach for a single tool.

The most important idea to hold onto is this: the oops locates the instruction that crashed, which is rarely the line that contains the defect. A NULL pointer dereference faults at the access, but the bug is wherever that pointer was set to NULL, left uninitialised, or freed earlier. Everything below is aimed at extracting, from the dump alone, as much as possible about what kind of bug you are chasing before you decode anything.

Reproduce a real kernel oops

To read a dump you need one in front of you. This small module dereferences a NULL pointer in its init function. Run it only on a test machine or VM you can crash freely.

/* oops_demo.c */
#include <linux/module.h>
#include <linux/init.h>

static int __init oops_demo_init(void)
{
        int *p = NULL;

        pr_info("oops_demo: about to dereference NULL\n");
        *p = 42;                /* faulting instruction */
        return 0;
}

module_init(oops_demo_init);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Deliberate NULL-pointer oops for teaching");

Build it against your running kernel’s headers and load it:

raghu@techveda.org:~$ make -C /lib/modules/$(uname -r)/build M=$(pwd) modules
raghu@techveda.org:~$ sudo insmod ./oops_demo.ko
raghu@techveda.org:~$ dmesg | tail -n 20

The insmod command does not return cleanly, and the report appears in the kernel log. On x86-64 it looks similar to this (addresses and offsets will differ on your system):

BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
Oops: 0002 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 3921 Comm: insmod Tainted: G           OE      6.8.0-45-generic
RIP: 0010:oops_demo_init+0x11/0x1e [oops_demo]
Code: ... c7 00 2a 00 00 00 ...
RAX: 0000000000000000 RBX: ffffb3c1c0a0fc80 RCX: 0000000000000000
Call Trace:
 <TASK>
 ? __die_body+0x1a/0x60
 ? page_fault_oops+0x15c/0x290
 do_one_initcall+0x5b/0x340
 do_init_module+0x97/0x290
 load_module+0x2a1b/0x2c40
 </TASK>
---[ end trace 0000000000000000 ]---

Read the header as triage information

The first block is not boilerplate. It tells you what kind of fault occurred and in what context, which shapes everything you do next.

• #PF: supervisor write access and error_code(0x0002) decode the fault: bit 1 set means a write, bit 0 clear means the target page was not present, and no user bit means it happened in kernel mode. A read fault, or a fault on a present page, points at a different class of problem.
• Oops: 0002 [#1] — the [#1] is an oops counter. A [#2] means this is the second fault since boot, so you may be looking at a cascade rather than the original problem.
• RIP: 0010:... — the 0010 is the code segment selector, the kernel code segment, confirming the fault was in kernel mode at privilege ring 0.
• Tainted: G ... OE — these flags are diagnostic. O is an out-of-tree module, E an unsigned one. If you see W, a warning already fired earlier in this boot, and the oops may be a downstream effect. P means a proprietary module is loaded, which changes how much you can trust the environment.
• PREEMPT SMP NOPTI and the Comm: insmod field tell you the preemption model, that it is a multiprocessor build, and which process was running. Faults in interrupt or atomic context are read very differently from faults in an ordinary process.

One quick sanity check on the RIP line: it is printed as symbol+offset/size. If the offset is larger than the size, the symbol is only the nearest preceding one, and the real code is likely a static function the linker did not export. Do not trust the name blindly in that case.

The faulting address is a fingerprint

The address in the BUG: line frequently identifies the class of bug on its own, before you read any source. A few patterns are worth memorising.

• Exactly 0x0 — a genuine NULL dereference.
• A small value such as 0x0000000000000018 — you dereferenced a member of a NULL structure. The address is the field offset, so 0x18 tells you which member was accessed.
• 0x6b6b6b6b6b6b6b6b — this is POISON_FREE (the byte 0x6b, ASCII ‘k’) repeated. You read through memory that was freed: a use-after-free. The related pattern 0x5a5a... (POISON_INUSE, ‘Z’) is memory that was allocated but never initialised.
• 0xdead000000000100 or 0xdead000000000122 — these are LIST_POISON1 and LIST_POISON2 on x86-64. list_del() writes them into a node’s next and prev pointers, so seeing them means a list node was used after it was removed. Unlike the slab poison, this one is written on ordinary kernels, not just debug builds.
• A non-canonical address (bits 63:48 not all identical) does not raise a page fault at all; the CPU raises a general protection fault. That is the signature of a wild or corrupted pointer.
• An address in the userspace range (low canonical, for example 0x00007f...) that faults inside kernel code — a driver dereferenced a user-controlled pointer directly instead of copying it in through the safe accessors.

Five dumps, read by their fingerprint

The value of these patterns is that they change what you look for before you open an editor. Here are five faults, each read from its header alone. Boot the test machine with slub_debug=FZP so the slab allocator poisons memory, which makes the memory-lifetime fingerprints appear reliably.

1. A plain NULL dereference

The reproducer above wrote through a NULL pointer. The address is 0x0 and the fault is a write, so the read is simply: some pointer was NULL at a store. The register dump confirms it — RAX: 0000000000000000 is the pointer, and the Code: bytes c7 00 2a 00 00 00 are movl $0x2a,(%rax), the write of 42.

2. A member of a NULL structure

Change the fault so it reads a field of a NULL structure instead of the pointer itself:

struct widget {
        int   id;            /* offset 0 */
        char  name[16];      /* offset 4 */
        void *ops;           /* offset 24 (0x18) */
};

static int __init oops_demo_init(void)
{
        struct widget *w = NULL;

        return w->ops != NULL;   /* reads 8 bytes at offset 0x18 */
}

BUG: kernel NULL pointer dereference, address: 0000000000000018
#PF: supervisor read access in kernel mode
RIP: 0010:oops_demo_init+0x9/0x20 [oops_demo]

The address is not 0x0 but 0x18, and that is the whole clue: 0x18 is offsetof(struct widget, ops). The pointer w was NULL and the code reached for its ops field. In real drivers this immediately narrows the search to the one structure whose field sits at that offset, which you can confirm with pahole or offsetof. Note also that this fault is a read (supervisor read access), unlike the write in the first example.

3. A use-after-free

Now allocate an object, free it, and use it:

struct thing {
        void (*fn)(void);
        char  pad[8];
};

static int __init oops_demo_init(void)
{
        struct thing *t = kmalloc(sizeof(*t), GFP_KERNEL);

        if (!t)
                return -ENOMEM;
        kfree(t);
        t->fn();                 /* read fn from freed memory, then call it */
        return 0;
}

BUG: unable to handle page fault for address: 6b6b6b6b6b6b6b6b
#PF: supervisor instruction fetch in kernel mode
#PF: error_code(0x0010) - not-present page
RIP: 0010:0x6b6b6b6b6b6b6b6b
Code: Unable to access opcode bytes at 0x6b6b6b6b6b6b6b6b.

The fault address and the RIP are both 0x6b6b6b6b6b6b6b6b: the freed object was poisoned with POISON_FREE (0x6b), the read of fn returned that value, and the indirect call jumped to it. The error_code(0x0010) instruction-fetch bit and the Unable to access opcode bytes line confirm the CPU tried to execute at a poison address. One precise detail: the slab allocator fills a freed object with 0x6b except its final byte, which is set to POISON_END (0xa5). A pointer sitting at the very end of an object therefore reads as 0xa56b6b6b6b6b6b6b rather than a clean run of 0x6b; the pad field above keeps fn away from that last byte.

4. A wild pointer and a general protection fault

A pointer that is not merely NULL but non-canonical faults differently:

static int __init oops_demo_init(void)
{
        char *p = (char *)0xbadc0ffee0ddf00d;   /* non-canonical */

        *p = 0;
        return 0;
}

general protection fault, probably for non-canonical address 0xbadc0ffee0ddf00d: 0000 [#1] PREEMPT SMP NOPTI
RIP: 0010:oops_demo_init+0x11/0x20 [oops_demo]

There is no #PF line and no page-fault address, because the CPU rejected the address before any page lookup: the top bits of 0xbadc0ffee0ddf00d are not a sign-extension of bit 47, so it is non-canonical and raises a general protection fault instead. The header even names the likely cause. Seeing general protection fault rather than NULL pointer dereference tells you the pointer held garbage, which points at corruption or a use-after-free rather than a missing NULL check.

5. An unchecked userspace pointer in a driver ioctl

This is the one you are most likely to meet in real driver work. A character-device ioctl receives arg, which for most commands is a userspace address, and the driver dereferences it directly instead of copying it in:

static long demo_ioctl(struct file *file, unsigned int cmd,
                       unsigned long arg)
{
        int value = *(int __user *)arg;   /* WRONG: raw deref of a user pointer */

        return value;
}

When user space calls the ioctl with a pointer that is invalid or simply not mapped, the driver faults:

BUG: unable to handle page fault for address: 00000000cafef00d
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 PID: 4012 Comm: demo_app Tainted: G           OE      6.8.0-45-generic
RIP: 0010:demo_ioctl+0x1e/0x40 [demo]
Call Trace:
 <TASK>
 __x64_sys_ioctl+0x9a/0xe0
 do_syscall_64+0x5b/0x120
 entry_SYSCALL_64_after_hwframe+0x6e/0x76
 </TASK>

Three fields read together tell the story. The faulting address 0x00000000cafef00d is in the userspace range, the Comm: demo_app field names the process that made the call, and the RIP is inside the driver’s ioctl handler, reached through __x64_sys_ioctl. That combination means the kernel followed a pointer that user space controlled. It is not memory corruption; it is missing validation. On modern CPUs with SMAP, a direct kernel read of user memory faults even when the address is mapped, because supervisor code may not touch user pages without the safe accessors. The fix is to copy the value in, which validates the address and returns an error instead of crashing:

        int value;

        if (copy_from_user(&value, (int __user *)arg, sizeof(value)))
                return -EFAULT;

copy_from_user, copy_to_user, and get_user/put_user check the range and carry an exception-table entry, so a bad user address yields -EFAULT rather than an oops. A userspace-range address in an oops whose RIP sits in a driver is almost always this bug.

The call trace is partly fiction

The Call Trace looks like a clean list of callers, but on x86-64 with the ORC unwinder (the default for years) it is not. Lines prefixed with ? are unreliable: the unwinder found a text address on the stack but could not confirm it belongs to the active call chain. Almost always these are breadcrumbs left over from an earlier call path, and they should be ignored. In the first dump, the ? __die_body and ? page_fault_oops lines are the fault-handling machinery itself, not part of your code’s path.

Lines without a ? are confirmed frames. In that example those are do_one_initcall, do_init_module and load_module, which is the genuine module-load path. Reading every ? line as a real caller is one of the most common ways engineers waste hours on an oops.

There is a second, subtler trap: inlining. The function named on the RIP line may be an inlined caller, and the code that actually faulted can be inside a small static helper that was folded into it. The dump cannot show that; recovering the inline chain needs the decoding tools this series turns to next.

You can read the corrupted value directly

The register block and the Code: line are not just for experts. The Code: field is the raw bytes around the faulting instruction; when disassembled it shows the exact operation, for example a write through RAX. Cross-reference that with the register dump, see RAX: 0000000000000000, and you have confirmed both the kind of access and the offending pointer value without opening an editor. In the first example the store movl $0x2a,(%rax) writing through a zero RAX is the NULL write, and 0x2a is the value 42. A later article in this series shows how scripts/decodecode turns the Code: line into readable assembly with the trapping instruction marked.

Key takeaways

• An oops is a snapshot of an already-broken machine; it locates the crash, not the defect.
• The header is triage data: the error code, oops counter, code segment, taint flags, and context all narrow the search before you decode anything.
• The faulting address is a fingerprint. 0x18 is a member of a NULL struct, 0x6b6b... is a use-after-free, 0xdead0000... is list poison, a non-canonical address raises a general protection fault, and a userspace-range address in kernel code means a driver dereferenced a user pointer without copy_from_user.
• In the Call Trace, only lines without a ? are confirmed frames; ? lines are usually stale and misleading.
• The register block and Code: line let you read the bad pointer and the faulting operation straight from the dump.

Frequently asked questions

What does an address of 0x6b6b6b6b6b6b6b6b in an oops mean?
It is the byte 0x6b (POISON_FREE) repeated, which the slab allocator writes over freed memory when slab debugging is enabled. Reading through it means a use-after-free: you followed a pointer into memory that had already been released.

Why does a small address like 0x18 appear instead of 0x0 for a NULL bug?
Because the code dereferenced a member of a NULL structure, not the pointer itself. The faulting address equals the field’s offset within the structure, so 0x18 means the access was to the field at offset 24, which you can match with offsetof or pahole.

Why does my driver oops when it dereferences a pointer passed to ioctl?
Because the ioctl argument is a userspace address, and dereferencing it directly in kernel code faults: the address may be unmapped, and with SMAP the CPU blocks direct kernel access to user memory anyway. Use copy_from_user, copy_to_user, or get_user/put_user, which validate the address and return -EFAULT instead of crashing.

Why are some Call Trace lines prefixed with a question mark?
On kernels using the ORC unwinder, a ? marks an address the unwinder found on the stack but could not confirm belongs to the live call chain. These are usually leftovers from an earlier call path and should be ignored; lines without a ? are the confirmed frames.

What is the difference between a general protection fault and a page fault in a dump?
A page fault (#PF) is an access to a canonical address whose page is not present or not permitted, such as NULL. A general protection fault is raised for a non-canonical address, which usually means a wild or corrupted pointer rather than a simple NULL.

Where the series goes next

Reading the dump is half the work. Knowing the bug class from the faulting address and trusting only the confirmed frames already narrows most investigations, but the dump still points at where execution crashed, not at the defect. The next article turns these fields into an exact source line, recovers the frames the compiler hid through inlining, and works backward from the crash site to the code that actually went wrong — the point where an oops stops being a wall of hex and becomes a short, direct path to the fix. Building this fluency is one of the debugging skills we practise on real hardware in the TECH VEDA Linux Device Drivers course.

Further reading

• Bug hunting — The Linux Kernel documentation
• The ORC unwinder — The Linux Kernel documentation
• Tainted kernels — The Linux Kernel documentation
• include/linux/poison.h (torvalds/linux)