Capturing a Kernel Crash Dump with kdump on ARM64

When a panic happens on a board on your desk, you read dmesg. When it happens in the field and the register dump scrolls off a console nobody was watching, you need the memory image instead. That is what a kernel crash dump is for. Our earlier tutorial on decoding a kernel oops ended by recommending panic_on_oops with kdump; this post is the part that was left out.

What you need

• An ARM64 target you can reboot and whose kernel command line you control (a board with U-Boot, or QEMU with -append).
• A kernel tree you can rebuild, plus the matching vmlinux with debug information.
• kexec-tools on the target, and roughly 128 MB of RAM you can set aside.
• Root on the target. Every command below is run as root.

QEMU -M virt is enough to learn the mechanics. Boot it with -kernel rather than UEFI firmware, because a UEFI boot leaves no /sys/firmware/fdt for kexec to reuse, and give it at least -m 2048. It will not reproduce the failures described at the end of this post, which are all properties of real boards.

What kdump does when the kernel panics

kdump is built on kexec, which boots one kernel directly from another without going back to firmware. The variant used here is kexec -p, the panic path: it preloads a second kernel, the capture kernel, into memory the first kernel reserved at boot. When panic(), die() or the SysRq crash handler fires, the kernel calls crash_kexec(), saves CPU registers as ELF notes, and jumps to that image. The address of the ELF header describing the old memory reaches the second kernel through the elfcorehdr= parameter, and that memory then appears at /proc/vmcore.

Kernel configuration for a kernel crash dump

The system kernel needs the kexec syscall; the capture kernel needs the dump-reading side. On ARM64 the same relocatable Image can serve as both, so in practice you enable all of it in one config. A separate, smaller capture-kernel build is only worth the trouble if you need it to fit a smaller reservation:

CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y

CONFIG_KEXEC (or CONFIG_KEXEC_FILE, the file-descriptor-based variant that CONFIG_KEXEC_SIG builds signature checking on) selects CONFIG_KEXEC_CORE. CONFIG_CRASH_DUMP selects CONFIG_VMCORE_INFO and CONFIG_CRASH_RESERVE, and enables CONFIG_PROC_VMCORE by default. One point causes confusion: CONFIG_DEBUG_INFO has no menu entry of its own any more — it is selected by whichever entry you pick in the “Debug information” choice under Kernel hacking, and that choice appears only when CONFIG_DEBUG_KERNEL is set. Debug information is not needed to capture a dump, only to read it. Confirm what you built:

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E 'CONFIG_(KEXEC|CRASH_DUMP|PROC_VMCORE)'
CONFIG_KEXEC_CORE=y
CONFIG_KEXEC=y
# CONFIG_KEXEC_FILE is not set
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y

If /proc/config.gz does not exist, the running kernel was built without CONFIG_IKCONFIG_PROC; check the build tree’s .config instead. One ARM64 trap: CONFIG_KEXEC depends on ARCH_SUPPORTS_KEXEC, which on this architecture is def_bool PM_SLEEP_SMP. On a uniprocessor board, or one built without suspend support, the “Enable kexec system call” prompt does not appear at all. CONFIG_KEXEC_FILE has no such dependency and is the way out.

Reserving memory with crashkernel=

The reservation happens at boot and cannot be done later, so it goes on the kernel command line. That timing is the point: at the moment of a panic the kernel cannot be trusted to allocate anything, so the region has to be set aside while the machine is still healthy. The simplest form gives a size and lets the kernel choose the address:

crashkernel=128M

You can pin the address with crashkernel=128M@0xb0000000. On ARM64 an explicitly given start address must be aligned to 2 MiB (0x200000), and the reservation is refused if the region is already in use. A range form, crashkernel=512M-2G:64M,2G-:128M, scales the size with the RAM present, which helps when one image ships to several board variants. For a first setup, use the plain form.

Reboot with the parameter added and confirm the reservation happened. The kernel prints it, and the region appears in /proc/iomem under the name Crash kernel:

raghu@techveda.org:~$ dmesg | grep -i crashkernel
[    0.000000] crashkernel reserved: 0x00000000b0000000 - 0x00000000b8000000 (128 MB)
raghu@techveda.org:~$ grep -i 'crash kernel' /proc/iomem
  b0000000-b7ffffff : Crash kernel

If nothing is printed, the reservation failed. The kernel says so in the same log, either with crashkernel reservation failed - memory is in use. for a fixed address, or a line beginning cannot allocate crashkernel when no suitable region was found.

Loading the capture kernel with kexec -p

Now preload the capture kernel into that reserved region. Check first that an uncompressed Image is actually present on the target: ARM64 kexec cannot load Image.gz, zImage, uImage or a FIT image, and most embedded root filesystems keep no kernel under /boot at all, so copy arch/arm64/boot/Image from your build tree onto the board before running the command below.

raghu@techveda.org:~$ kexec -p /boot/Image --initrd=/boot/initramfs-kdump.img --append="1 nr_cpus=1 reset_devices panic=10 console=ttyAMA0,115200"

Build that initramfs before you load the capture kernel. It needs the block or network driver for wherever you intend to write, plus a minimal user space containing makedumpfile and vmcore-dmesg. On most embedded images the quickest route is to reuse the initramfs the board already boots with, after confirming those two binaries are in it. If the storage driver is built into the kernel rather than a module, you can drop --initrd and give the capture kernel root=/dev/mmcblk0p2 rootwait rw instead, so it mounts the real root filesystem and runs your normal init.

Each argument here has a specific purpose. 1 boots to single-user mode, nr_cpus=1 keeps the capture kernel to one CPU because it has very little memory, and reset_devices asks drivers to reset their hardware rather than assume the state the crashed kernel left behind. panic=10 matters more than it looks: if the capture kernel itself panics, the board reboots after ten seconds instead of hanging until someone visits it. Keep the console setting, or a failure in the capture kernel is silent. No device tree is needed on the usual path: ARM64 kexec-tools reuses the running one from /sys/firmware/fdt unless you pass --dtb.

Verify the image is loaded before you trust it:

raghu@techveda.org:~$ cat /sys/kernel/kexec_crash_loaded
1
raghu@techveda.org:~$ cat /sys/kernel/kexec_crash_size
134217728

A 0 from the first file means no capture kernel is loaded, and a panic will simply hang or reboot. In current mainline these attributes were regrouped under /sys/kernel/kexec/ as crash_loaded and crash_size, with the old names kept as compatibility symlinks, so the paths above work on old and new kernels alike.

kexec_crash_size is writable: writing a smaller value shrinks the reservation and returns the rest to the system, but only while no capture kernel is loaded. To unload, run kexec -p -u.

Triggering a panic and saving the dump

Test the path deliberately, before you need it. SysRq gives you a crash on demand. Writing 1 first enables all SysRq functions, because many distributions ship a restricted mask that does not include the crash trigger:

raghu@techveda.org:~$ echo 1 > /proc/sys/kernel/sysrq
raghu@techveda.org:~$ echo c > /proc/sysrq-trigger

The console should show the panic, then a second kernel banner. When it settles, /proc/vmcore is the crashed kernel’s memory. Filter it rather than copying it whole — a full copy is the size of your RAM, which is rarely what you want on a device with eMMC:

raghu@techveda.org:~$ makedumpfile -c --message-level 1 -d 31 /proc/vmcore /var/crash/vmcore
raghu@techveda.org:~$ vmcore-dmesg /proc/vmcore > /var/crash/dmesg.txt

-d 31 is the dump level: it discards zero, cache, user and free pages and keeps kernel data, which covers most driver bugs. -c compresses each page with zlib; -l (LZO) and -z (zstd) are usually smaller and faster, but they have to be enabled when makedumpfile is built, and the stock Yocto build has only zlib. --message-level 1 keeps the progress output quiet. vmcore-dmesg, also from kexec-tools, extracts just the kernel log ring buffer — on a constrained target that one small file is often enough to identify the fault.

Reading the kernel crash dump

Reboot into the normal kernel first, then analyse offline on your workstation. The crash utility is the usual tool, and it needs the exact vmlinux that produced the dump:

raghu@techveda.org:~$ crash vmlinux /var/crash/vmcore
      KERNEL: vmlinux
    DUMPFILE: /var/crash/vmcore  [PARTIAL DUMP]
        CPUS: 4
       ... (date, uptime, release, machine and memory lines omitted)
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
crash> bt
crash> log
crash> ps

[PARTIAL DUMP] is expected: it means the file was filtered by makedumpfile -d rather than copied whole. bt gives the backtrace of the panicking task, log the kernel ring buffer, and ps the task list at the moment of the crash.

Plain gdb can inspect registers and memory too, but only on an uncompressed ELF dump — it cannot open the kdump-compressed file that makedumpfile -c produces. Expand it first with makedumpfile -R, and expect gdb to understand far less about kernel structures than crash does.

Where kdump fails on embedded boards

• The reservation never happened. The bootloader replaced your command line, or the fixed address overlapped a carveout for a display or DSP. Check dmesg, not the bootloader environment.
• kexec loading is blocked. If the kernel.kexec_load_disabled sysctl has been set to 1, kexec -p fails; that sysctl cannot be set back to 0, so only a reboot clears it. A locked-down kernel refuses the legacy kexec_load() path as well, leaving kexec_file_load() with a signed image as the sanctioned route.
• The capture kernel boots but cannot write anywhere. Its initramfs must contain the storage or network driver and a userspace able to run makedumpfile. This is the most common reason a setup that appears to work produces no file.
• The reservation is too small. If the capture kernel dies during boot with allocation failures, increase it rather than guessing.
• The reservation cannot be placed in low memory. A plain crashkernel=128M is placed below 4 GB, which fails on boards whose low memory is already full of reserved-memory carveouts. Use crashkernel=128M,high crashkernel=64M,low so the bulk goes above 4 GB and only the DMA-capable part stays low.
• Not every oops produces a dump. An oops reaches crash_kexec() only when kexec_should_crash() agrees: in interrupt context, in the idle task or init, or when panic_on_oops is set. A plain oops in ordinary process context kills the task and leaves no dump. Set panic_on_oops=1, and optionally panic_on_warn=1, to cover those.
• The watchdog reboots the board mid-dump. The SoC or PMIC watchdog keeps counting while the capture kernel boots and makedumpfile runs, and nothing in that path services it. Either disable it on the capture kernel’s command line, or accept vmcore-dmesg only, which finishes in seconds.
• Devices left running by the crashed kernel disturb the capture kernel. In-flight DMA and IOMMU contexts survive the panic, so you can see SMMU stream-ID faults or a hang as soon as the capture kernel touches that controller. Keep reset_devices even when it looks unnecessary, and blacklist the offending driver on the capture kernel’s command line.

Building kdump into a Yocto image

OpenEmbedded-Core carries the recipes, so nothing has to be written by hand. At the time of writing it packages kexec-tools 2.0.32, splits it into kexec, kdump and vmcore-dmesg, and runtime-depends on makedumpfile, which is also in OE-core. That dependency hangs off the kexec-tools meta-package rather than off kdump, so list everything you want explicitly:

IMAGE_INSTALL:append = " kexec kdump vmcore-dmesg makedumpfile"

The kdump package installs /etc/sysconfig/kdump.conf and either a sysvinit script or a kdump.service unit, depending on your distro features. Set KDUMP_KIMAGE to the capture kernel and KDUMP_CMDLINE to the arguments above, and add crashkernel=128M to the machine’s command line.

Three things about that script are worth knowing before you rely on it. It never passes an initrd — there is no KDUMP_INITRD variable — so the capture kernel has to reach your real root filesystem on its own. Its save path runs when /proc/vmcore is non-empty, which is only true inside the capture kernel, so kdump and makedumpfile must be present in the capture environment and not merely in the normal image. And the shipped kdump.conf sets MAKEDUMPFILE_ARGS to --dump-dmesg, so out of the box you get a kernel-log extract rather than the filtered vmcore this post produced. Change that variable if you want the full dump.

Key takeaways

• A kernel crash dump needs three things in place before the crash: a crashkernel= reservation, a capture kernel loaded with kexec -p, and somewhere for it to write.
• Verify the reservation in dmesg and /proc/iomem, and the load in /sys/kernel/kexec_crash_loaded. Do not assume either worked.
• Test with echo c > /proc/sysrq-trigger while you can still watch the console.
• Use makedumpfile -d 31 to filter, and vmcore-dmesg when you only need the log. The result is kdump-compressed, so read it with crash, not gdb.
• On ARM64 kexec loads only an uncompressed Image, an explicit crashkernel base must be 2 MiB aligned, and the capture kernel’s initramfs must carry the driver it needs to save the file.

Frequently asked questions

Why does kdump need memory reserved at boot instead of allocating it when the crash happens?
At the moment of a panic the kernel cannot be trusted to allocate anything. The reserved region is set aside at boot and never used by the running kernel, so the capture kernel sitting there is intact when it is needed.

Do I need to build a separate capture kernel?
Not on ARM64. The architecture supports relocatable kernels, so the same Image can act as both. A separate, smaller build is only worth doing if you want to fit a smaller reservation.

How much memory should I give to crashkernel on an embedded board?
128 MB is a reasonable starting point. If the capture kernel fails to boot with allocation errors, raise it. You can shrink an existing reservation at runtime by writing a smaller value to /sys/kernel/kexec_crash_size, but only while no capture kernel is loaded.

Why did my board reboot on panic without producing a dump?
Either no capture kernel was loaded, which /sys/kernel/kexec_crash_loaded will tell you, or it booted but its initramfs had no driver for the storage you asked it to write to. A hardware watchdog can also reboot the board before the dump finishes. Keep a console configured in the capture kernel’s command line so you can see which happened.

Further reading

• Documentation for Kdump — The kexec-based Crash Dumping Solution (kernel.org)
• kernel/crash_reserve.c — crashkernel parsing and reservation (torvalds/linux)
• kernel/kexec_core.c — the kexec sysfs attributes (torvalds/linux)
• kexec-tools recipe and kdump init script (OpenEmbedded-Core)
• The crash utility documentation
• Reading a Kernel Oops, Part 2: Decode to the Source Line and Reason Backward