What PREEMPT_RT Changes: Preemptible Softirqs (Part 5)

This is Part 5 of the series on what PREEMPT_RT changes underneath code that looks portable. Part 3 covered local_lock_t and migrate_disable(), and Part 4 covered forced interrupt threading, which moves your handler into a kernel thread. This part follows that thread into the moment it re-enables bottom halves. Preemptible softirqs make the rest of the real-time kernel possible, and they are also the change most likely to invalidate an assumption buried in a driver you already shipped.

Where softirq work runs on a stock kernel

Start with the code you are probably running today. Without PREEMPT_RT, local_bh_disable() is an addition to the preemption counter:

/* include/linux/bottom_half.h, v7.3 */
static __always_inline void __local_bh_disable_ip(unsigned long ip, unsigned int cnt)
{
	preempt_count_add(cnt);
	barrier();
}

static inline void local_bh_disable(void)
{
	__local_bh_disable_ip(_THIS_IP_, SOFTIRQ_DISABLE_OFFSET);
}

Because that counter is the preemption counter, a bottom-half disabled section on a stock kernel is also a non-preemptible section, and so are the handlers. handle_softirqs() is reached from invoke_softirq() at interrupt exit, and whichever task was interrupted is charged for that time and cannot be scheduled out of it.

The kernel bounds this with a budget, and the shape of the bound is the part worth reading carefully:

/* kernel/softirq.c, v7.3 */
#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
#define MAX_SOFTIRQ_RESTART 10

The budget is checked once per full pass over the pending bitmask, after the loop has called every pending vector’s action(). A single handler is never interrupted by it. The real bound is therefore two milliseconds plus the duration of the longest single pass, set by whichever handler is slowest — on an embedded gateway, usually NET_RX under load. Hence the familiar symptom: a board that measures clean worst-case latency on an idle bench develops jitter as soon as traffic arrives, the jitter grows with the traffic, and no tunable bounds it. The design is simply that softirq work outranks every task on the CPU, including a SCHED_FIFO task at priority 80.

What preemptible softirqs replace that with

PREEMPT_RT removes the premise. If softirq work is to be preemptible by a real-time task, it cannot be accounted for in the preemption counter, so RT keeps the accounting somewhere else:

/* kernel/softirq.c, v7.3, CONFIG_PREEMPT_RT */
struct softirq_ctrl {
	local_lock_t	lock;
	int		cnt;
};

static DEFINE_PER_CPU(struct softirq_ctrl, softirq_ctrl) = {
	.lock	= INIT_LOCAL_LOCK(softirq_ctrl.lock),
};

There are two counters, not one, and the comment above this structure says why each exists. The per-task counter in task_struct::softirq_disable_cnt is what softirq_count(), in_softirq() and in_serving_softirqs() read, because those counts are only meaningful while the task that owns the section is running. The per-CPU counter prevents pointless wakeups of ksoftirqd when that task is preempted or blocks. include/linux/preempt.h switches the macro accordingly, so softirq_count() reads current->softirq_disable_cnt on RT and preempt_count() everywhere else.

Two things follow. local_bh_disable() becomes a real synchronisation call rather than a counter bump. And, more surprising the first time you trace it, the handlers no longer run at interrupt exit at all:

/* kernel/softirq.c, v7.3, CONFIG_PREEMPT_RT */
static inline bool should_wake_ksoftirqd(void)
{
	return !this_cpu_read(softirq_ctrl.cnt);
}

static inline void invoke_softirq(void)
{
	if (should_wake_ksoftirqd())
		wakeup_softirqd();
}

On a stock kernel invoke_softirq() calls __do_softirq() inline. On RT it only ever wakes a thread. Preemptible softirqs therefore run in one of two places: in ksoftirqd/N, or inside __local_bh_enable_ip() in whichever task happened to re-enable bottom halves.

Why preemptible softirqs need an explicit RCU read lock

Here is the entry path, with the debug branches removed:

/* kernel/softirq.c, v7.3, CONFIG_PREEMPT_RT */
void __local_bh_disable_ip(unsigned long ip, unsigned int cnt)
{
	WARN_ON_ONCE(in_hardirq());
	lock_map_acquire_read(&bh_lock_map);

	/* First entry of a task into a BH disabled section? */
	if (!current->softirq_disable_cnt) {
		if (preemptible()) {
			if (IS_ENABLED(CONFIG_PREEMPT_RT_NEEDS_BH_LOCK))
				local_lock(&softirq_ctrl.lock);
			else
				migrate_disable();

			/* Required to meet the RCU bottomhalf requirements. */
			rcu_read_lock();
		}
	}
	/* ... per-CPU and per-task counter updates ... */
}

The rcu_read_lock() is the detail most people miss, and it is a direct consequence of preemptible softirqs. On a stock kernel, disabling bottom halves disables preemption, and a non-preemptible region is already an RCU read-side critical section by construction, so code that relies on that is correct without saying so. On RT the region is preemptible, that implication is gone, and the kernel has to take the read lock explicitly. Any driver that calls local_bh_disable() and then dereferences an RCU-protected pointer is relying on this line, whether its author knew it or not.

The same logic appears in handle_softirqs(), which reports an RCU quiescent state through rcu_softirq_qs() only when !IS_ENABLED(CONFIG_PREEMPT_RT), because on RT the softirq is a task context whose context switches RCU already sees.

Where preemptible softirqs actually run: one NAPI poll, two kernels

On a stock kernel the device raises its interrupt, the handler runs in hard interrupt context and calls napi_schedule(), which raises NET_RX. The handler returns, irq_exit() calls invoke_softirq(), and __do_softirq() runs the NAPI poll immediately, with preemption disabled, charged to whichever task was running. Your real-time audio thread waits.

On PREEMPT_RT the handler runs in its IRQ thread at that thread’s SCHED_FIFO priority. The forced-threading wrapper calls local_bh_disable() around your handler, so the NET_RX raise lands inside a bottom-half disabled section. When the wrapper calls local_bh_enable(), this happens:

/* kernel/softirq.c, v7.3, CONFIG_PREEMPT_RT, abridged */
void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)
{
	bool preempt_on = preemptible();

	/* ... if this call does not re-enable softirqs, or nothing is
	   pending, fall through to the counter update and return ... */

	if (!preempt_on) {
		wakeup_softirqd();
		goto out;
	}

	cnt = SOFTIRQ_OFFSET;
	__local_bh_enable(cnt, false);
	__do_softirq();
out:
	__local_bh_enable(cnt, preempt_on);
}

The NAPI poll runs right there, in the IRQ thread, at the IRQ thread’s real-time priority, and preemptibly. Note what the count is set to: SOFTIRQ_OFFSET, not the SOFTIRQ_DISABLE_OFFSET used on entry, which makes in_serving_softirq() true while leaving the section preemptible. That single assignment is preemptible softirqs in one line.

So receive processing now inherits the priority of the thread that enabled bottom halves, which is a knob you never had before. But when the raise happens outside a bottom-half section — a timer, another CPU’s call, any path with no IRQ thread involved — the work goes to ksoftirqd/N, whose upstream registration has no .setup callback and so stays on the default time-sharing policy. Your real-time task outranks it. That is the point, and it is also the trap: a busy SCHED_FIFO task can starve receive processing, RCU callbacks and tasklets indefinitely, and the symptom will look like a networking fault rather than a scheduling decision you made.

Linux 6.18 dropped the lock that made bottom-half sections exclusive

Until 6.18 the branch shown above had no branch. local_lock(&softirq_ctrl.lock) was unconditional, so exactly one task per CPU could be inside a bottom-half disabled section at a time. Since local_lock_t on RT is a per-CPU sleeping lock with priority inheritance, a second task entering such a section boosted the holder and waited.

Commit 3253cb49cbad (“softirq: Allow to drop the softirq-BKL lock on PREEMPT_RT”, Sebastian Andrzej Siewior, 4 September 2025, signed off by Thomas Gleixner) removed that. Its message states the cost plainly: with the lock in place, “a forced threaded interrupt cannot preempt another forced threaded interrupt. Instead it will PI-boost the other handler and wait for its completion.” On a board with several threaded handlers at different priorities, that is a priority inversion hiding inside the mechanism meant to deliver real time.

The lock was removable because the reason for it had been engineered away. Code that genuinely needed exclusive access to a per-CPU resource was converted to local_lock_nested_bh(), which expands to a lockdep annotation with no runtime effect on a stock kernel and to a real spin_lock() on RT, where local_lock_t is typedef’d to spinlock_t and therefore to an rtmutex. Documentation/locking/locktypes.rst states the trade directly: on PREEMPT_RT local_lock_t behaves as a real lock and serialises access to the data structure, “which allows removal of serialization via local_bh_disable()”. Narrow locks around the data replaced one wide lock around the context.

The new switch is CONFIG_PREEMPT_RT_NEEDS_BH_LOCK. Its help text restores the old synchronised behaviour “for testing reasons”, to be selected if you suspect an error with preemptible softirqs: a bisection aid, not a tuning knob. It carries no default line, so on a fresh configuration it is off and you get preemptible softirqs with no mutual exclusion between bottom-half sections.

The races that dropping the lock exposed

Within six months two of them surfaced in the BPF map code. Jiayuan Chen fixed them on 25 February 2026 in 869c63d5975d (cpumap) and 1872e75375c4 (devmap), and the commit messages are the clearest statement of the new hazard anywhere:

On PREEMPT_RT kernels, the per-CPU xdp_bulk_queue (bq) can be accessed
concurrently by multiple preemptible tasks on the same CPU.

The original code assumes bq_enqueue() and __cpu_map_flush() run
atomically with respect to each other on the same CPU, relying on
local_bh_disable() to prevent preemption.

The cpumap failure runs like this. Task A is in bq_flush_to_queue(), has reset bq->count to zero and dropped the producer lock, and is preempted. Task B, on the same CPU, enqueues until the queue is full, flushes it, and calls __list_del_clearprev() on bq->flush_node, setting prev to NULL. Task A resumes and dereferences that NULL. The devmap version is worse: bq_xmit_all() snapshots cnt = bq->count before iterating, a preempting task transmits and frees the same frames, and the first task walks stale pointers.

The fix in both cases was a local_lock_t inside the queue structure, taken with local_lock_nested_bh() in the enqueue and the flush path. That is the pattern to copy. If your code holds a per-CPU buffer, index, pointer list or pinned timer and relies on local_bh_disable() to keep a second task out, it is correct on a stock kernel, was correct on RT before 6.18, and is wrong on RT from 6.18 onward. The audit question is narrow: for every DEFINE_PER_CPU your driver touches inside a bottom-half section, is there a lock on the data, or only the section?

Which kernels behave which way

All four rows were read from kernel/softirq.c, kernel/Kconfig.preempt and the architecture Kconfig files at the tags named, not inferred from release dates.

Two readings matter. The first is that 6.12 is the oldest longterm series in which CONFIG_PREEMPT_RT can be selected on arm64 or x86 from the mainline or stable tree at all: it depends on ARCH_SUPPORTS_RT, and neither architecture selects that symbol at 6.6 or earlier. The RT softirq code sits in those older trees because other architectures could enable it, which is not the same as your board being able to. CONFIG_PREEMPT_RT_NEEDS_BH_LOCK likewise first appears at 6.18, so on a 6.12 real-time kernel there is nothing to configure.

The second is the backport position, which is what will catch people. The two BPF fixes are in mainline from 7.0 and were backported into 6.18.y: kernel/bpf/cpumap.c and kernel/bpf/devmap.c both carry the local_lock at 6.18.60 and carry none at the 6.18 release tag. A 6.18.y kernel can therefore sit on either side of those fixes depending on the point release, while being on the non-exclusive side for everything else. Pin the point release, not the series.

How to check this on your own board

Three checks, in the order worth running them. First, establish what kernel you are on, since preemptible softirqs only exist if RT is enabled:

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E 'CONFIG_PREEMPT_RT|CONFIG_PREEMPT_RT_NEEDS_BH_LOCK'
raghu@techveda.org:~$ grep -E 'CONFIG_PREEMPT_RT|CONFIG_PREEMPT_RT_NEEDS_BH_LOCK' /boot/config-$(uname -r)

Use whichever of the two your board provides. CONFIG_PREEMPT_RT=y without CONFIG_PREEMPT_RT_NEEDS_BH_LOCK=y, on 6.18 or later, is the configuration this post is about. If the second symbol is absent entirely, check your version against the table above before concluding anything.

Second, look at who the softirq work can be charged to:

raghu@techveda.org:~$ ps -eo pid,class,rtprio,comm | grep -E 'ksoftirqd|ktimers|irq/'

ksoftirqd/N appears with the default time-sharing class and no real-time priority, which is what the registration above predicts. On 6.18 and later you will also see ktimers/N threads at SCHED_FIFO priority 1, because ktimerd_setup() calls sched_set_fifo_low(); that is a separate mechanism for timer softirqs. The irq/N-name threads are your forced-threaded handlers, and their priorities are the ones charged for the work they raise.

Third, see the context for yourself rather than trusting the argument:

raghu@techveda.org:~$ cd /sys/kernel/debug/tracing
raghu@techveda.org:~$ sudo sh -c 'echo 1 > events/irq/softirq_entry/enable'
raghu@techveda.org:~$ sudo head -40 trace_pipe
raghu@techveda.org:~$ sudo sh -c 'echo 0 > events/irq/softirq_entry/enable'

Read the task name and the context flags on each line, not the event name. On a stock kernel the softirq_entry events are attributed to whatever was interrupted, often the idle task, with the hard-interrupt flag set. On a real-time kernel they are attributed to a named thread — an irq/N thread, ksoftirqd/N, or an ordinary application that called local_bh_enable() — with the soft-interrupt flag set instead. The last command is not optional on a device you care about.

Our learning tracks cover the kernel and driver layers where these decisions are made.

Limits, costs and what was not verified

Preemptible softirqs move latency rather than removing it. The work still has to run, and on RT it runs under the scheduler, so you can now starve it: a SCHED_FIFO task that does not sleep holds off ksoftirqd for as long as it runs, and RCU and NET_RX are among the things waiting. CPU isolation and priority discipline are the other half of this mechanism. There are also no latency or throughput figures anywhere in this post, because I measured none.

The audit implied by the 6.18 change is bounded by what was fixed upstream, not by what might still be wrong: I verified the two BPF map paths and the mechanism, and did not sweep every subsystem for other code that assumed local_bh_disable() gave it exclusive per-CPU access. Treat the pattern in those two commits as the review checklist for your own out-of-tree drivers. The version statements here describe the mainline and stable trees at the tags listed below; vendor BSPs carry their own backport decisions, and an out-of-tree RT patch set on a 6.6 kernel may behave like none of the four rows. Check the source on the tree you ship, not the series number.

Key takeaways

• On a stock kernel, local_bh_disable() adds SOFTIRQ_DISABLE_OFFSET to the preemption counter, so both the section and the handlers are non-preemptible and outrank every task on the CPU.
• On RT the accounting moves to task_struct::softirq_disable_cnt and a per-CPU softirq_ctrl, which is what makes preemptible softirqs possible, and local_bh_disable() takes rcu_read_lock() explicitly because a preemptible section is no longer an RCU read-side critical section.
• invoke_softirq() on RT only wakes a thread: preemptible softirqs run either in ksoftirqd/N at the default policy, or inline in __local_bh_enable_ip() at the priority of whichever task re-enabled bottom halves.
• From 6.18, local_bh_disable() on RT takes only migrate_disable() unless CONFIG_PREEMPT_RT_NEEDS_BH_LOCK is set, so two bottom-half sections on one CPU can run concurrently. That switch is a debugging aid, not a tuning knob.
• Per-CPU data that relied on the old exclusivity needs local_lock_nested_bh(). The cpumap and devmap fixes are the worked example; they landed in 7.0 and in 6.18.y point releases, so pin the point release.

Frequently asked questions

Which task is charged for network receive processing on a real-time kernel?
Usually the IRQ thread that raised the softirq, because the forced-threading wrapper runs your handler inside a bottom-half disabled section and the work is executed when that section ends. If the raise happens outside such a section, the work goes to ksoftirqd/N instead, which runs at the default scheduling policy.

Should I enable CONFIG_PREEMPT_RT_NEEDS_BH_LOCK on a product kernel?
No. Its help text describes it as a way to restore the old synchronised behaviour for testing, to be selected if you suspect an error with preemptible softirqs. Use it to bisect a suspected regression, not as a default.

How do I know whether my per-CPU data is affected by the 6.18 change?
For each DEFINE_PER_CPU variable your code touches inside a local_bh_disable() section, ask whether anything other than that section keeps a second task out. If the answer is nothing, add a local_lock_t to the data and take it with local_lock_nested_bh(), as the cpumap and devmap fixes did.

Further reading

• kernel/softirq.c at v7.3 — the CONFIG_PREEMPT_RT block, handle_softirqs() and the ksoftirqd and ktimerd thread registrations
• Commit 3253cb49cbad — softirq: Allow to drop the softirq-BKL lock on PREEMPT_RT
• Commit 869c63d5975d — bpf: Fix race in cpumap on PREEMPT_RT
• Commit 1872e75375c4 — bpf: Fix race in devmap on PREEMPT_RT
• kernel/Kconfig.preempt at v7.3 — PREEMPT_RT and PREEMPT_RT_NEEDS_BH_LOCK
• Documentation: Lock types and their rules — local_lock_t, local_lock_nested_bh() and spinlock_t under PREEMPT_RT
• include/linux/local_lock_internal.h at v7.3 — the two definitions of local_lock_nested_bh()
• include/linux/preempt.h at v7.3 — SOFTIRQ_DISABLE_OFFSET and the RT definition of softirq_count()
• kernel.org releases — the current mainline and longterm series