Runtime PM: Why Your Device Never Suspends, and How to Find Out

A device that will not enter runtime PM suspend is one of the most common power problems in embedded work, and also one of the most tractable. The symptom is familiar: the board draws more idle current than the datasheet suggests, the clocks and the regulator are correct, and the driver’s ->runtime_suspend() callback is simply never called. The kernel does not decide this by guesswork. A single function decides it, that function has seven possible outcomes, and it will tell you which one it picked.

This is a companion to Deferred Probe in the Linux Kernel, which covers a driver whose probe() runs late. This one covers a driver that probed correctly and then never powers down.

One function decides

Every runtime suspend attempt funnels through rpm_suspend() in drivers/base/power/runtime.c. Before it goes anywhere near your callback, it calls rpm_check_suspend_allowed(). So does rpm_idle(). Here is that function, in full:

static int rpm_check_suspend_allowed(struct device *dev)
{
	int retval = 0;

	if (dev->power.runtime_error)
		retval = -EINVAL;
	else if (dev->power.disable_depth > 0)
		retval = -EACCES;
	else if (atomic_read(&dev->power.usage_count))
		retval = -EAGAIN;
	else if (!dev->power.ignore_children && atomic_read(&dev->power.child_count))
		retval = -EBUSY;
	/* Pending resume requests take precedence over suspends. */
	else if ((dev->power.deferred_resume &&
	    dev->power.runtime_status == RPM_SUSPENDING) ||
	    (dev->power.request_pending && dev->power.request == RPM_REQ_RESUME))
		retval = -EAGAIN;
	else if (__dev_pm_qos_resume_latency(dev) == 0)
		retval = -EPERM;
	else if (dev->power.runtime_status == RPM_SUSPENDED)
		retval = 1;

	return retval;
}

Two properties matter for debugging. It reads only fields of struct dev_pm_info — the power member of struct device, defined in include/linux/pm.h — so it asks the driver and the hardware nothing. And it is an if/else-if ladder, so the first matching condition wins. If two things are wrong at once, you are only told about the earlier one. Fix that, run again, and the next reason appears.

What runtime PM checks before it suspends a device

Reading the ladder from the top, these are the seven outcomes and what each one means:

• -EINVAL (22) — power.runtime_error is set. A previous callback returned a fatal error, and the PM core has stopped working on this device until the status is set directly.
• -EACCES (13) — power.disable_depth is above zero. Runtime PM is disabled. This field starts at 1, so runtime PM is disabled for every device until something calls pm_runtime_enable().
• -EAGAIN (11) — power.usage_count is not zero. Somebody holds a reference. This is the common one.
• -EBUSY (16) — a child is still active and power.ignore_children is not set. A parent cannot suspend under an active child.
• -EAGAIN again — a resume is pending or deferred. Resume requests take precedence over suspends.
• -EPERM (1) — the PM QoS resume latency limit is 0, which is how user space says it cannot tolerate any resume latency at all.
• 1 — the device is already suspended. Not an error.

Keep those errno numbers in mind. They appear as negative values in the trace output below.

The same fields, readable from sysfs

Every field in the ladder is exposed under /sys/devices/.../power/. Some of them require CONFIG_PM_ADVANCED_DEBUG, which itself depends on CONFIG_PM_DEBUG — if the files below are missing, that is why, and it is worth turning on in a development defconfig.

• runtime_status — "active", "suspended", "suspending", "resuming", "error", or "unsupported". A value of "error" is the -EINVAL case.
• runtime_usage — power.usage_count. Non-zero is the -EAGAIN case. Requires CONFIG_PM_ADVANCED_DEBUG.
• runtime_active_kids — power.child_count. Non-zero is the -EBUSY case. Requires CONFIG_PM_ADVANCED_DEBUG.
• runtime_enabled — "enabled", "disabled", "forbidden", or a combination of the last two. "disabled" is the -EACCES case. Requires CONFIG_PM_ADVANCED_DEBUG.
• pm_qos_resume_latency_us — a value of 0 here is the -EPERM case.
• control — "auto" or "on". Writing "on" forbids runtime power management from user space, and shows up as "forbidden" in runtime_enabled.
• runtime_active_time and runtime_suspended_time — milliseconds in each state. These two answer the question “is it suspending at all, or just rarely?”

Locate the device and read the set. Using find avoids needing the exact bus path, which differs across SoC variants:

raghu@techveda.org:~$ D=$(find /sys/devices -name '30a20000.i2c' | head -1)
raghu@techveda.org:~$ grep . "$D"/power/runtime_status "$D"/power/runtime_usage \
    "$D"/power/runtime_enabled "$D"/power/control
/sys/devices/.../30a20000.i2c/power/runtime_status:active
/sys/devices/.../30a20000.i2c/power/runtime_usage:1
/sys/devices/.../30a20000.i2c/power/runtime_enabled:enabled
/sys/devices/.../30a20000.i2c/power/control:auto

A runtime_usage of 1 on a device nobody is using is the whole answer. Runtime PM is enabled, user space has not forbidden it, and the core refuses to suspend only because the counter says the device is in use.

The tracepoints print the fields and the verdict

Sysfs gives you the state now. The tracepoints in include/trace/events/rpm.h give you the state at each decision, which is what you want when the counter is leaking rather than merely stuck.

The rpm_suspend, rpm_resume and rpm_idle tracepoints share one event class, which records the same fields the ladder reads: usage count, disable depth, runtime_auto, request pending, irq safe, and child count. Its print format is:

TP_printk("%s flags-%x cnt-%-2d dep-%-2d auto-%-1d p-%-1d"
	" irq-%-1d child-%d", ...)

So cnt is usage_count, dep is disable_depth, and child is child_count. A fourth tracepoint, rpm_return_int, records the verdict along with the call site:

TP_printk("%pS:%s ret=%d", (void *)__entry->ip, __get_str(name),
	__entry->ret)

Between them, the two formats give you the inputs and the output of the same decision. Turn them on:

raghu@techveda.org:~$ cd /sys/kernel/tracing
raghu@techveda.org:~$ echo 1 | sudo tee events/rpm/enable
raghu@techveda.org:~$ sudo cat trace_pipe

On a device that will not suspend, the pair reads like this:

rpm_idle: 30a20000.i2c flags-0 cnt-1 dep-0 auto-1 p-0 irq-0 child-0
rpm_return_int: rpm_idle+0x54/0x1a0:30a20000.i2c ret=-11

Read it straight off the line. cnt-1: the usage count is 1. dep-0: runtime PM is enabled. child-0: no active children. auto-1: user space has not forbidden it. And ret=-11 is -EAGAIN, which in that ladder can only mean the usage count. The trace and the sysfs read agree.

Narrow the noise to one device with the event filter:

raghu@techveda.org:~$ echo 'name == "30a20000.i2c"' | sudo tee events/rpm/filter
raghu@techveda.org:~$ sudo cat trace_pipe

The usual cause is a usage count nobody dropped

Once you know it is the usage count, there are only a few ways it gets stuck, and the first is the one the kernel documentation itself warns about. pm_runtime_get_sync() increments the counter and then resumes the device — but it does not drop the counter when the resume fails. The documentation is explicit that it does not, and recommends pm_runtime_resume_and_get() instead, especially when the return value is checked.

This is the leak, and it is easy to write:

	ret = pm_runtime_get_sync(dev);
	if (ret < 0)
		return ret;		/* counter is still incremented */

Every failed transfer raises the counter by one, permanently. The device is fine, the error is handled, and it never suspends again. Either use the helper that cleans up after itself:

	ret = pm_runtime_resume_and_get(dev);
	if (ret < 0)
		return ret;		/* counter already dropped */

or drop it yourself with pm_runtime_put_noidle() on the error path. One other cause is worth ruling out: a negative autosuspend_delay_ms prevents runtime suspend entirely, which is documented behaviour of that attribute and not a bug.

The shape of the counter tells you which it is. A counter stuck at 1 is one missing put. A counter that climbs is a put missing from an error path that keeps being taken. Read runtime_usage twice, a few operations apart, and you know where to look.

Reasoning about the driver model at this level of detail is what our Linux Device Drivers program is built around.

Key takeaways

• One function, rpm_check_suspend_allowed(), decides every runtime suspend. It reads only struct dev_pm_info fields, never the hardware. It is an if/else-if ladder, so the first matching condition wins and fixing one cause can reveal the next.
• Each return code maps to one field: -EAGAIN to usage_count, -EACCES to disable_depth, -EBUSY to child_count, -EPERM to the PM QoS resume latency, -EINVAL to runtime_error.
• Every one of those fields is readable from /sys/devices/.../power/, though several need CONFIG_PM_ADVANCED_DEBUG.
• The rpm tracepoints print the inputs (cnt, dep, child) and rpm_return_int prints the verdict, so you never have to guess which branch was taken.
• The most common single cause is pm_runtime_get_sync() on an error path, which does not drop the counter. Use pm_runtime_resume_and_get().

Frequently asked questions

Why is runtime_usage missing from my device’s power directory?
That attribute is only created when CONFIG_PM_ADVANCED_DEBUG is enabled, which in turn depends on CONFIG_PM_DEBUG. The same applies to runtime_enabled and runtime_active_kids. Enable it in your development defconfig.

What is the difference between pm_runtime_get_sync() and pm_runtime_resume_and_get()?
Both increment the usage counter and resume the device. pm_runtime_get_sync() does not drop the counter if the resume fails, so an error path that returns early leaks a reference and the device never suspends again. pm_runtime_resume_and_get() drops the counter on failure, which is why the kernel documentation recommends it when you check the return value.

My device shows runtime_status of “suspended” but the hardware is clearly still powered. Is that a bug?
Not necessarily. The initial runtime PM status of every device is “suspended” regardless of the real hardware state. A driver whose device is actually active at probe time must call pm_runtime_set_active() before pm_runtime_enable() to correct it.

Further reading

• drivers/base/power/runtime.c — rpm_check_suspend_allowed(), rpm_suspend(), rpm_idle() and rpm_resume().
• Runtime Power Management Framework for I/O Devices — the field-by-field reference for struct dev_pm_info and every helper function.
• ABI: sysfs-devices-power — the authoritative description of each power/ attribute.
• include/trace/events/rpm.h — the rpm_internal event class and rpm_return_int.
• kernel/power/Kconfig — PM_ADVANCED_DEBUG and its dependency on PM_DEBUG.
• Deferred Probe in the Linux Kernel — the companion piece on drivers that probe late.