Skip to main content

TECH VEDA

Corporate on-site training - Submit proposal Pick your modulesSharpen your kernel skills: deep dives, drivers, Yocto, CVEs, careers — updated daily. Read the blog →
Debug Stories

RK3399 Kernel Panic: A Call Trace With No Driver in It

An RK3399 kernel panic on a Rock960 board running v6.0-rc6 showed only timer code in its call trace. Follow the bisection that traced it to one commit.

RK3399 Kernel Panic: A Call Trace With No Driver in It

A Rock960 board stopped booting on a Linux 6.0 release candidate. The RK3399 kernel panic came a few seconds after start-up, and its call trace held timer code, interrupt code and the idle loop, but no driver at all. Finding the change responsible took two full bisections.

This RK3399 kernel panic was reported to the Linux kernel mailing list by an engineer testing a new release candidate. The board was a 96boards Rock960, built on the Rockchip RK3399, a six-core arm64 system-on-chip used in many single-board computers and industrial products. The kernel was v6.0-rc6. What follows is reconstructed from the public thread, the commits and the driver source.

The tool at the centre of the story is git bisect. Given a known-good and a known-bad kernel, it picks the commit halfway between them; you build it, test it and mark it good or bad, and git halves the range again. About fourteen tests cover ten thousand commits. Git only knows what you tell it, so if a test result describes a different failure, git will still name a commit, and it will be the wrong one.

The report from a Rock960 board

The reporter’s first message carried a bisection result and an RK3399 kernel panic log, and asked the list for help.

there is a kernel panic on a rock960 board: […] Any clue?

That bisection had landed on 14facbc1871a, “Merge 5.19-rc6 into char-misc-next”. A merge commit joins two lines of development and usually adds no code. Naming one as first bad means each parent tested good and only the combination tested bad: possibly a real interaction, but also the usual sign of an inconsistent test.

The attached log is a kernel oops, the report printed when kernel code hits a fault it cannot handle:

[    2.377386] Unable to handle kernel NULL pointer dereference at
virtual address 0000000000000005
[    2.377390] Mem abort info:
[    2.377392]   ESR = 0x0000000096000004
[    2.379529]   EC = 0x25: DABT (current EL), IL = 32 bits
[    2.380964]   FSC = 0x04: level 0 translation fault
...
[    2.383740] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 5.19.0-rc6+ #728
[    2.384311] Hardware name: 96boards Rock960 (DT)
[    2.385323] pc : pinctrl_lookup_state+0x20/0xc0
[    2.385729] lr : rockchip_i2s_probe+0x1a8/0x54c
...
[    2.392051] x2 : 0000000000000000 x1 : ffff800009e1e0e8 x0 :
ffffffffffffffed
[    2.392677] Call trace:
[    2.392892]  pinctrl_lookup_state+0x20/0xc0
[    2.393261]  rockchip_i2s_probe+0x1a8/0x54c
[    2.393629]  platform_probe+0x68/0xe0
...
[    2.399887] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b

On arm64, ESR is the Exception Syndrome Register, which the CPU fills in to say why it trapped. EC = 0x25: DABT is the exception class, a data abort: a load or store failed. FSC is the fault status code; a translation fault means no page-table mapping existed for the address. pc is the faulting instruction and lr, the link register, is the caller. PID: 1 Comm: swapper/0 shows the fault hit the first kernel thread during driver initialisation; that thread becomes init, so the kernel ends with Attempted to kill init!.

x0, the first function argument on arm64, holds 0xffffffffffffffed, which is -19, or -ENODEV. Kernel functions that return pointers often encode an error inside the pointer with ERR_PTR(), and callers must test it with IS_ERR() before use. Here an error pointer reached pinctrl_lookup_state(), and reading a field five bytes in gave the address 0x5. The RK3399’s I2S audio driver was the obvious suspect.

A suspect in the audio driver

Another developer replied within fifteen minutes with a change to sound/soc/rockchip/rockchip_i2s.c that widened the pointer check to IS_ERR_OR_NULL(), and noted that this code had needed several commits for one fix. That was correct. A version of the driver’s new bit-clock pin switching reached a release candidate without checking for a failed devm_pinctrl_get(), and that exact NULL pointer dereference had already been fixed separately, in commit 26b9f2fa7b1c.

That matters for the bisection. The I2S crash was real but existed only on some intermediate commits, not in the RK3399 kernel panic seen on the release candidate. At the v6.0-rc6 tag the I2S probe code already guards the pointer with !IS_ERR(), and the log’s kernel version is 5.19.0-rc6+, not 6.0-rc6. The reporter compared it with what the release candidate itself printed and saw the problem with the first result.

Hmm, actually, I think it is not the same kernel panic I’m observing on v6.0-rc6. I’ve been puzzled by different kernel panics from different issues when git bisecting.

The RK3399 kernel panic on the release candidate

The oops posted next, from a 6.0.0-rc1-based build, looks quite different:

[    3.726846] Unable to handle kernel execute from non-executable
memory at virtual address ffff80000a516530
[    3.728308]   ESR = 0x000000008600000f
[    3.728667]   EC = 0x21: IABT (current EL), IL = 32 bits
[    3.730223]   FSC = 0x0f: level 3 permission fault
...
[    3.732918] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.0.0-rc1+ #730
[    3.734635] pc : extcon_dev_list+0x0/0x10
[    3.735056] lr : call_timer_fn.constprop.0+0x24/0x80
...
[    3.742865] x2 : 0000000000000200 x1 : ffff80000a516530 x0 :
ffff0000007e7310
[    3.742865] Call trace:
[    3.743107]  extcon_dev_list+0x0/0x10
[    3.743477]  __run_timers.part.0+0x1f0/0x234
[    3.743908]  run_timer_softirq+0x3c/0x7c
...
[    3.746484]  arch_cpu_idle+0x18/0x2c
[    3.746843]  default_idle_call+0x34/0x70
[    3.747243]  do_idle+0x240/0x2bc
...
[    3.749911] Kernel panic - not syncing: Oops: Fatal exception in
interrupt

This time the exception class is IABT, an instruction abort: the CPU could not fetch an instruction. A permission fault, rather than a translation fault, means the page was mapped, but as data that must not be executed. extcon_dev_list is not a function. It is a list head in the data section of extcon, the framework that reports the state of external connectors such as USB ports. The kernel had jumped into a variable.

PID: 0 Comm: swapper/1 is the idle task of CPU 1, sleeping in arch_cpu_idle(). An interrupt woke it, and on the way out the kernel ran pending softirqs, deferred interrupt work that runs outside any process. run_timer_softirq handles expired kernel timers. A kernel timer is a struct timer_list holding an expiry time and a function pointer; __run_timers() reads the pointer and call_timer_fn() calls it. In interrupt context there is no process to kill, hence Fatal exception in interrupt.

So a timer expired and its stored function was not a function. The registers agree: call_timer_fn() receives the timer and its function pointer in x0 and x1, and x1 equals the faulting pc. No driver appears in the trace because the code that armed a timer has long returned by the time it runs. The trace shows who called the broken pointer, not who left it there.

The second bisection

A crash with no culprit in its trace is where bisection helps most, because it needs no theory. The reporter started again, chasing the RK3399 kernel panic the release candidate actually showed, and a day later posted the result.

after a painful git bisect, the kernel panic happens with: commit 8dc60f8da22fdbaa1fafcfb5ff6d24bc9eff56aa (refs/bisect/bad) […] Reverting it fixes the issue

The commit was “phy: rockchip-inno-usb2: Sync initial otg state”, merged for 6.0. This time the result named one ordinary commit, not a merge, and reverting it removed the RK3399 kernel panic: two signs of a trustworthy bisection. The oops in that message had the same shape, a timer softirq calling a bad pointer on an idle CPU, but this time the pointer was zero:

[    3.566365] Unable to handle kernel NULL pointer
dereference at virtual address 0000000000000000
[    3.567783]   EC = 0x21: IABT (current EL), IL = 32 bits
[    3.573354] pc : 0x0
[    3.573567] lr : call_timer_fn.constprop.0+0x24/0x80
...
[    3.580290] x2 : 0000000000000200 x1 : 0000000000000000 x0 :
ffff000001025310
[    3.581178]  0x0
[    3.581359]  __run_timers.part.0+0x1f0/0x234
[    3.581759]  run_timer_softirq+0x3c/0x7c

A function pointer that is sometimes zero and sometimes the address of an unrelated variable is a hint in itself. Code does not change its own function pointers at random; freed memory that someone else has reused does.

The RK3399 has an OTG USB port, which can act as host or device depending on the connector’s ID pin, and the PHY driver reports that state through extcon. Previously the driver learned the state only when an ID interrupt fired, so a board booted with a cable attached started in the wrong role. The commit reads the ID state once at initialisation:

		ret = devm_extcon_register_notifier(rphy->dev, rphy->edev,
					EXTCON_USB_HOST, &rport->event_nb);
		if (ret)
			dev_err(rphy->dev, "register USB HOST notifier failed\n");

+		if (!of_property_read_bool(rphy->dev->of_node, "extcon")) {
+			/* do initial sync of usb state */
+			ret = property_enabled(rphy->grf, &rport->port_cfg->utmi_id);
+			extcon_set_state_sync(rphy->edev, EXTCON_USB_HOST, !ret);
+		}
 	}

 out:
 	return ret;

The intent is correct and the change was reviewed. Nothing in it touches a timer.

Root cause

The thread did not spell out the full chain behind the RK3399 kernel panic; the fix commit says only that the OTG PHY aborts halfway through setup. The sequence below is our reading of drivers/phy/rockchip/phy-rockchip-inno-usb2.c, drivers/base/dd.c and drivers/extcon/extcon.c at v6.0-rc6, and each step can be checked there.

The new code stores the ID pin reading in ret, the same variable the function returns as its status. property_enabled() returns a boolean. When the ID pin reads high, ret becomes 1, execution falls through to out:, and rockchip_usb2phy_otg_port_init() returns 1. In the kernel, zero means success and anything else means failure, so probe takes its error path and returns 1. The driver core’s really_probe() prints a probe-failed warning and calls device_unbind_cleanup(), which runs devres_release_all().

Devres, the device-managed resource layer behind every devm_* call, records what probe allocates and registers, and undoes it in reverse order when the driver detaches or probe fails. The driver’s state structure, rphy, came from devm_kzalloc() and holds the per-port structures, so it is freed. Devres undoes only what it was told about.

It was not told about a delayed work item. A struct delayed_work is a work item paired with a timer_list; scheduling it arms the timer, and on expiry the timer queues the work. The OTG port’s otg_sm_work lives inside rphy. Just before returning, the new code called extcon_set_state_sync(), which calls extcon_sync(), which runs the notifier chain whether or not the state changed. The driver’s own notifier, registered a few lines earlier, is rockchip_otg_event(), and it does one thing:

	schedule_delayed_work(&rport->otg_sm_work, OTG_SCHEDULE_DELAY);

OTG_SCHEDULE_DELAY is 2 * HZ, two seconds. So the driver arms a two-second timer inside its own state, reports failure, the driver core frees that state, and the timer stays queued with a pointer into freed memory. Two seconds later the timer softirq reads a function pointer from memory now owned by another allocation and calls it: zero in one build, the address of extcon_dev_list in another. Both panics came 3.5 to 3.8 seconds after boot, which fits a timer armed during probing two seconds earlier.

This is a use-after-free: the timer core reads a structure the driver core has already freed. It also explains why only some boards failed. The bug needs an OTG port configured for OTG or peripheral mode, no external extcon property in the device tree, and the ID pin reading high at boot.

The fix and confirmation

The commit’s author already had the answer.

A fix has been submitted and accepted, it is currently sitting in -next

Two other users had reported the problem a few weeks earlier, and the author had already posted “phy: rockchip-inno-usb2: Return zero after otg sync”. It gives the ID reading its own variable and skips the sync if the notifier could not be registered:

-	int ret;
+	int ret, id;
...
-		if (ret)
+		if (ret) {
 			dev_err(rphy->dev, "register USB HOST notifier failed\n");
+			goto out;
+		}

 		if (!of_property_read_bool(rphy->dev->of_node, "extcon")) {
 			/* do initial sync of usb state */
-			ret = property_enabled(rphy->grf, &rport->port_cfg->utmi_id);
-			extcon_set_state_sync(rphy->edev, EXTCON_USB_HOST, !ret);
+			id = property_enabled(rphy->grf, &rport->port_cfg->utmi_id);
+			extcon_set_state_sync(rphy->edev, EXTCON_USB_HOST, !id);
 		}

With ret left at zero, probe succeeds, rphy stays allocated, and the timer later runs the real otg_sm_work. Both earlier reporters gave Tested-by tags, another developer reviewed it, and it became commit f340ed8664a5. The kernel’s regression tracker had logged the report in regzbot, the regression-tracking bot, against the merge commit, and updated it to name 8dc60f8da22f as the cause and f340ed8664a5 as the fix. The fix had been queued for the next merge window rather than the current release, and that had a cost.

This should have been applied for v6.0-rc, not -next. We now have a v6.0 release that doesn’t boot properly on RK3399.

We checked the tags. The fix is absent from v6.0 and present from v6.1-rc1; in the 6.0 stable series it first appears in v6.0.3. A product team that moved an affected board to 6.0, 6.0.1 or 6.0.2 would have met this RK3399 kernel panic. Notice what the fix does not change: the driver still does not cancel otg_sm_work if probe fails after the timer is armed. Writing probe error paths that release everything they started, including timers and work, is a core part of what we teach in Linux device driver development.

How the maintainers reached the answer

  • The reporter posted the full oops for the RK3399 kernel panic, not a summary, so others could read the registers and see the ERR_PTR in x0 within minutes.
  • When the proposed I2S fix did not match, the reporter compared the oops from the release candidate with the oops from the first bisection and saw they were different crashes. That comparison is what saved the investigation.
  • The second bisection named one ordinary commit, and the reporter confirmed it by reverting that commit before reporting it.
  • The commit author already had a fix queued after two earlier reports. Searching the lists for a culprit’s hash or title is often the fastest last step.

A method you can reuse

These steps follow the RK3399 kernel panic investigation and apply to any regression that crashes.

Step 1. Capture the complete oops, including the register dump, one file per crash.

raghu@techveda.org:~$ dmesg | sed -n '/Unable to handle kernel/,/end trace/p' > oops-rc6.txt

If the board resets first, save the serial console output instead.

Step 2. Decode the trace to source lines with the vmlinux from the same build.

raghu@techveda.org:~$ ./linux/scripts/decode_stacktrace.sh linux/vmlinux < oops-rc6.txt
raghu@techveda.org:~$ ./linux/scripts/faddr2line linux/vmlinux call_timer_fn.constprop.0+0x24/0x80

Step 3. Write down the crash signature: exception class, fault code, the lr function and the last trace frames. Here it was IABT, lr in call_timer_fn, trace through run_timer_softirq. A different signature means a different bug.

Step 4. Bisect against that signature. Mark a commit bad only when the crash matches; skip any other failure.

raghu@techveda.org:~$ cd linux
raghu@techveda.org:~$ git bisect start v6.0-rc6 v5.19
raghu@techveda.org:~$ git bisect bad
raghu@techveda.org:~$ git bisect good
raghu@techveda.org:~$ git bisect skip
raghu@techveda.org:~$ git bisect log > bisect-rk3399.log

With git bisect run ./step.sh, the script exits 0 for good, 1 for a matching crash and 125 for anything else, which git treats as skip.

Step 5. Distrust a merge commit as the answer. Re-check the signature at both parents, or bisect the main line first with --first-parent and then inside the branch it names.

raghu@techveda.org:~$ git bisect start --first-parent v6.0-rc6 v5.19

Step 6. Confirm with a revert on top of the failing kernel. If the revert boots, the result is real.

raghu@techveda.org:~$ git bisect reset
raghu@techveda.org:~$ git checkout -b test-revert v6.0-rc6
raghu@techveda.org:~$ git revert --no-edit 8dc60f8da22f

Step 7. Look for an existing fix through its Fixes: tag, and check where it landed.

raghu@techveda.org:~$ git log --oneline --grep='Fixes: 8dc60f8da22f' origin/master
raghu@techveda.org:~$ git tag --contains f340ed8664a5 | head

For stable releases, run git tag --contains in a clone of the stable tree.

Best practice from this case

Keep a status variable for status only. If a function returns ret, do not reuse ret for a register value, a count or a boolean. The RK3399 kernel panic came from one such line, which looked harmless in a diff that showed only the new lines.

Anything probe starts must be stopped on every exit path. Devres does not cancel a timer or work item unless you register that. Use devm_delayed_work_autocancel() from include/linux/devm-helpers.h, which initialises the work and arranges cancel_delayed_work_sync() on detach or probe failure, or cancel explicitly in the error path. For any probe function, ask: if the last call fails, what is still running?

A bisection is only as good as the test at each step. Compare the crash signature every time, skip unrelated failures, and confirm the answer with a revert before reporting it.

Try it yourself

This exercise reproduces the core mistake behind the RK3399 kernel panic, a structure freed while its delayed work is still armed, and shows the kernel’s object debugging catching it at the moment of the free. Run it only in a disposable virtual machine or test board, never on a production system: without the debug options the module is likely to crash the kernel.

First check whether your test kernel has debug objects for timers enabled:

raghu@techveda.org:~$ grep -E 'CONFIG_DEBUG_OBJECTS(_FREE|_TIMERS|_WORK)?=' /boot/config-$(uname -r)

If they are missing, enable CONFIG_DEBUG_OBJECTS, CONFIG_DEBUG_OBJECTS_FREE and CONFIG_DEBUG_OBJECTS_TIMERS under Kernel hacking and rebuild. Then build this module as armed.c:

#include <linux/module.h>
#include <linux/slab.h>
#include <linux/workqueue.h>

struct demo {
	int value;
	struct delayed_work dwork;
};

static void demo_work(struct work_struct *work)
{
	pr_info("armed: work ran\n");
}

static int __init armed_init(void)
{
	struct demo *d = kzalloc(sizeof(*d), GFP_KERNEL);

	if (!d)
		return -ENOMEM;
	INIT_DELAYED_WORK(&d->dwork, demo_work);
	schedule_delayed_work(&d->dwork, 2 * HZ);
	kfree(d);	/* the mistake: the timer inside d is still armed */
	return 0;
}

static void __exit armed_exit(void)
{
}

module_init(armed_init);
module_exit(armed_exit);
MODULE_LICENSE("GPL");
raghu@techveda.org:~$ echo 'obj-m += armed.o' > Makefile
raghu@techveda.org:~$ make -C /lib/modules/$(uname -r)/build M=$PWD modules
raghu@techveda.org:~$ sudo insmod armed.ko
raghu@techveda.org:~$ sudo dmesg | grep -A 3 ODEBUG

With the options enabled, the free is reported at once as a warning beginning ODEBUG: free active that names the object type timer_list, with a stack trace through armed_init. A debug build of the affected kernel would be expected to report the RK3399 kernel panic’s cause the same way, at probe time. Now add cancel_delayed_work_sync(&d->dwork); before kfree(d), rebuild, reboot, load it again and confirm the warning is gone. As a second exercise, write a step.sh for git bisect run that greps a saved console log for your crash signature and exits 0, 1 or 125.

Was this worth your time?

Frequently asked questions

What caused the RK3399 kernel panic on Linux v6.0-rc6?
Commit 8dc60f8da22f in the Rockchip inno-usb2 PHY driver stored the OTG ID pin reading in the return variable. When the pin read high, probe returned 1 and failed, devres freed the driver state, and a delayed work timer armed inside it fired two seconds later through a stale function pointer.

Why did the call trace show no driver?
The crash happened when the timer softirq called the function stored in an expired timer. The driver that armed the timer had returned long before, so only the timer code and the idle loop appear in the trace.

Why did the first bisection point at a merge commit?
Some intermediate commits crashed for a different reason, an I2S audio driver NULL pointer dereference fixed separately. Treating both crashes as one failure led the bisection to a merge commit instead of the commit behind the release-candidate crash.

Which kernels contain the fix?
The fix, commit f340ed8664a5, is in v6.1-rc1 and later, and in the 6.0 stable series from v6.0.3. Linux 6.0, 6.0.1 and 6.0.2 do not have it.

How can I investigate a similar timer softirq crash on my own system?
Capture the full oops and decode it with scripts/decode_stacktrace.sh; an lr in call_timer_fn means a timer called a bad pointer. Enable CONFIG_DEBUG_OBJECTS_TIMERS and CONFIG_DEBUG_OBJECTS_FREE on a test build to catch the free of an active timer, and bisect against the crash signature if it is a regression.

Further reading

RB
Raghu Bharadwaj

Founder, TECH VEDA — 20+ years teaching the Linux kernel, device drivers and embedded systems.

Follow on LinkedIn

Get new posts by email

Kernel, embedded Linux and AI-era engineering — a few sharp reads a month. No spam.

We email occasionally and never share your address.