Debug Story: A Circular Locking Dependency in a Driver

This is a debug story about a bug that never produced a hang, a crash, or a wrong result during testing. The board ran fine. The only sign of trouble was a message in dmesg that named a circular locking dependency. That message came from lockdep, the kernel’s runtime locking validator, and it was warning that two mutexes in a sensor driver could be taken in opposite orders on two different code paths. On the test bench the two paths never collided at the wrong moment, so nothing locked up. In the field, with more devices and more load, the same two paths would eventually meet and the board would freeze with no output at all. This post walks through how the warning was read, how the two lock orders were confirmed, and how the driver was fixed.

The symptom: a warning with no hang

The driver was a small platform driver, tv_sensor, that managed several sensor devices on an i.MX8M board. During a routine test that changed a sampling rate through sysfs while a background rescan was running, this appeared in the kernel log:

raghu@techveda.org:~$ dmesg | grep -A 30 "circular locking"
======================================================
WARNING: possible circular locking dependency detected
6.6.30 #1 Not tainted
------------------------------------------------------
kworker/2:1/74 is trying to acquire lock:
ffff0000031a4a70 (&sdev->lock){+.+.}, at: sensor_rescan_work+0x84/0x160 [tv_sensor]

but task is already holding lock:
ffff0000031b0120 (sensor_core_lock){+.+.}, at: sensor_rescan_work+0x30/0x160 [tv_sensor]

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (sensor_core_lock){+.+.}:
       __mutex_lock+0x9c/0x568
       mutex_lock_nested+0x24/0x30
       sensor_update_aggregate+0x2c/0xb0 [tv_sensor]
       sampling_rate_store+0x74/0xd0 [tv_sensor]
       dev_attr_store+0x20/0x3c
       sysfs_kf_write+0x50/0x74
       kernfs_fop_write_iter+0x120/0x1b0
       vfs_write+0x2b0/0x390
       ksys_write+0x70/0x108

-> #0 (&sdev->lock){+.+.}:
       __lock_acquire+0x119c/0x1e10
       lock_acquire+0xc8/0x2f0
       __mutex_lock+0x9c/0x568
       mutex_lock_nested+0x24/0x30
       sensor_rescan_work+0x84/0x160 [tv_sensor]
       process_one_work+0x1f8/0x458
       worker_thread+0x50/0x3d0
       kthread+0x110/0x114
       ret_from_fork+0x10/0x20

Directly below it, lockdep printed the part that makes the problem concrete:

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(sensor_core_lock);
                               lock(&sdev->lock);
                               lock(sensor_core_lock);
  lock(&sdev->lock);

 *** DEADLOCK ***

What the driver was doing

The driver used two mutexes. A global mutex, sensor_core_lock, protected a linked list of registered devices. Each device also had its own mutex, sdev->lock, protecting that device’s fields. Two code paths touched both locks.

The first path was the sysfs write handler that set a new sampling rate. It took the per-device lock, changed the field, and then recomputed a cross-device aggregate that needed the global lock:

static ssize_t sampling_rate_store(struct device *dev,
                struct device_attribute *attr,
                const char *buf, size_t len)
{
        struct tv_sensor *sdev = dev_get_drvdata(dev);
        unsigned int rate;
        int ret;

        ret = kstrtouint(buf, 0, &rate);
        if (ret)
                return ret;

        mutex_lock(&sdev->lock);          /* device lock first  */
        sdev->sampling_rate = rate;
        sensor_update_aggregate();         /* takes sensor_core_lock */
        mutex_unlock(&sdev->lock);

        return len;
}

The helper it called took the global lock:

static void sensor_update_aggregate(void)
{
        struct tv_sensor *s;

        mutex_lock(&sensor_core_lock);     /* global lock second */
        max_rate = 0;
        list_for_each_entry(s, &sensor_list, node)
                if (s->sampling_rate > max_rate)
                        max_rate = s->sampling_rate;
        mutex_unlock(&sensor_core_lock);
}

So this path acquired the device lock first and the global lock second. The second path was the background rescan run from a workqueue. It walked the list under the global lock and touched each device under the device lock, in the opposite order:

static void sensor_rescan_work(struct work_struct *work)
{
        struct tv_sensor *s;

        mutex_lock(&sensor_core_lock);     /* global lock first  */
        list_for_each_entry(s, &sensor_list, node) {
                mutex_lock(&s->lock);      /* device lock second */
                s->flags &= ~TV_SENSOR_STALE;
                mutex_unlock(&s->lock);
        }
        mutex_unlock(&sensor_core_lock);
}

One path takes device then global. The other takes global then device. That is the classic AB-BA lock inversion.

Reading the circular locking dependency report

The header names the running task and the lock it wants: kworker/2:1 is trying to acquire &sdev->lock, and it is already holding sensor_core_lock. That matches sensor_rescan_work exactly, which holds the global lock and then reaches for a device lock.

The two entries in the dependency chain are the two lock orders lockdep has observed. Entry #1 shows how sensor_core_lock was acquired while a device lock was held: the stack runs through sampling_rate_store into sensor_update_aggregate. Entry #0 shows the current acquisition of &sdev->lock while the global lock is held, from sensor_rescan_work. Together they close a cycle.

The characters in braces are lockdep’s usage bits. For these mutexes the report shows {+.+.}, meaning each lock was only ever acquired with interrupts enabled and never from an interrupt context, which is normal for a mutex. The Possible unsafe locking scenario block then spells out the inversion in the clearest possible form: CPU0 holds sensor_core_lock and wants &sdev->lock, while CPU1 holds &sdev->lock and wants sensor_core_lock. If both reach their second acquisition at the same time, neither can proceed.

The important point is that lockdep did not need the deadlock to actually happen. It records each single-path lock order the first time that path runs, then reports the moment a second path uses the reverse order. That is why the warning appeared even though the board never hung during the test.

Confirming the two lock orders

This checking only runs when the kernel is built with lockdep enabled, which is not the default for a production BSP. The relevant option is CONFIG_PROVE_LOCKING, which is found under Kernel hacking, Lock Debugging in menuconfig and pulls in the rest of the lockdep machinery. Confirm it is set on the running kernel:

raghu@techveda.org:~$ zcat /proc/config.gz | grep PROVE_LOCKING
CONFIG_PROVE_LOCKING=y

With that in place, the two lock chains the validator has learned can be inspected at runtime:

raghu@techveda.org:~$ cat /proc/lockdep_stats
 lock-classes:                          412 [max: 8191]
 direct dependencies:                  3218 [max: 32768]
 dependency chains:                    1174 [max: 65536]
 ...

The reproduction was simple once both paths were known: trigger a sysfs write on one device while the rescan work is scheduled.

raghu@techveda.org:~$ echo 200 > /sys/bus/platform/devices/tv_sensor.0/sampling_rate

lockdep reported the cycle on the first run and stayed quiet after that, because it only reports each new violating chain once. Reading the two stacks side by side confirmed that sampling_rate_store and sensor_rescan_work disagreed on the order of the two locks.

The fix: remove the nesting, and fix the order

There are two independent problems, and the cleanest fix removes the need for the nested locks entirely. The sysfs handler does not actually need to hold the device lock while it recomputes the aggregate. It only needs the device lock to update the device field, and the global lock to recompute the aggregate. Splitting them removes the AB-BA pattern:

        mutex_lock(&sdev->lock);
        WRITE_ONCE(sdev->sampling_rate, rate);
        mutex_unlock(&sdev->lock);

        sensor_update_aggregate();         /* takes only sensor_core_lock */

Because the aggregate now reads each device’s sampling_rate without the device lock, that field is written with WRITE_ONCE() and read with READ_ONCE() in the loop, which is sufficient for a single aligned word.

The second problem is the general rule the driver had never stated: when two locks can be held together, every path must take them in the same order. The standing rule here is global lock before device lock, matching what sensor_rescan_work already did. Any future path that must hold both at once now follows that order, and lockdep enforces it. After the change, the same reproduction ran clean and no further warning appeared.

This kind of locking discipline is one of the core skills we teach in the Linux Device Drivers course, because lock ordering bugs are common in real driver code and rarely show up in casual testing.

Key takeaways

• A circular locking dependency warning is lockdep telling you two locks were seen taken in opposite orders. It fires even when no deadlock has happened yet.
• Read the header for the held lock and the lock being acquired, then read entries #1 and #0 as the two conflicting orders. The Possible unsafe locking scenario block states the inversion directly.
• Build at least one test kernel with CONFIG_PROVE_LOCKING=y. The validator proves ordering from single paths, so bugs surface long before they cause a field hang.
• Fix lock inversion either by removing the nested locking or by enforcing one consistent lock order everywhere. Removing the nesting is usually the simpler and safer change.

Further reading

• Runtime locking correctness validator (lockdep-design) — kernel.org documentation.
• Lock types and their rules — kernel.org documentation.
• Generic Mutex Subsystem — kernel.org documentation.