What PREEMPT_RT Changes: Forced Interrupt Threading (Part 4)

Part 3 of this series, local_lock and migrate_disable, ended on a hardware boundary: data also touched from an IRQF_NO_THREAD handler or an NMI cannot use a local_lock, because those contexts cannot block. That sentence assumed you knew which of your handlers still run in hard interrupt context on an RT kernel. Almost none of them do. This part traces forced interrupt threading in Linux 7.3: what the kernel does to a handler you wrote for hard interrupt context, why the exclusion list contains exactly the flags it does, and the one-line priority change that separates a working driver from one that makes no progress. Everything below was read from kernel/irq/manage.c, include/linux/interrupt.h and kernel/sched/syscalls.c at the v7.3 tag.

A failure that only happens on PREEMPT_RT

Start with a real one, because it shows the whole mechanism from the outside. The PCIe Advanced Error Reporting driver registers two handlers: a primary, aer_irq(), which clears the Root Error Status register, and a thread function, aer_isr(), which walks the hierarchy to find which device reported the error. That split is deliberate. The primary must clear the status register promptly, because if a second correctable error arrives before it does, the hardware sets the Multiple ERR_COR Received bit, and once that bit is set the Error Source Identification register can no longer be trusted. The thread then has to scan every device instead of going straight to the one that failed.

On an ordinary kernel this works because the primary handler runs in hard interrupt context. It can interrupt the thread at any point, clear the register, and return. The priority relationship is not written down anywhere in the driver; it is a property of the context the handler runs in.

On PREEMPT_RT both halves are threads. Before Linux 7.0 both threads were created at the same SCHED_FIFO priority, so the primary could no longer preempt the secondary. Crystal Wood reported the consequence on an Arm system: the scan in the thread provoked further errors, the Multiple ERR_COR bit stayed set because nothing was clearing the status register, and the driver stopped making progress. The upstream report describes a loop with no exit, on a kernel configuration that differs from the working one only in how interrupts are delivered.

Nothing in the driver was wrong. What changed was an assumption the driver never stated, and that is the useful way to read the rest of this article: forced interrupt threading preserves the code but not the context, and anything the code inferred from the context has to be re-established some other way.

What forced interrupt threading does to your handler

The whole transformation is one function, called from __setup_irq() when the interrupt is requested:

static int irq_setup_forced_threading(struct irqaction *new)
{
	if (!force_irqthreads())
		return 0;
	if (new->flags & (IRQF_NO_THREAD | IRQF_PERCPU | IRQF_ONESHOT))
		return 0;

	/*
	 * No further action required for interrupts which are requested as
	 * threaded interrupts already
	 */
	if (new->handler == irq_default_primary_handler)
		return 0;

	new->flags |= IRQF_ONESHOT;

Three things have already happened. The function decided whether forced threading applies at all, it checked the exclusions, and it set IRQF_ONESHOT on your action whether you asked for it or not. That last line deserves more attention than it usually gets: IRQF_ONESHOT means the interrupt line is not re-enabled when the primary handler returns. It stays masked until the thread has finished. So forced interrupt threading does not merely move your code, it changes when the line reopens, and on a shared line it serialises every handler on it behind the slowest thread.

The rest of the function does the shuffle:

	/*
	 * Handle the case where we have a real primary handler and a
	 * thread handler. We force thread them as well by creating a
	 * secondary action.
	 */
	if (new->handler && new->thread_fn) {
		/* Allocate the secondary action */
		new->secondary = kzalloc_obj(struct irqaction);
		if (!new->secondary)
			return -ENOMEM;
		new->secondary->handler = irq_forced_secondary_handler;
		new->secondary->thread_fn = new->thread_fn;
		new->secondary->dev_id = new->dev_id;
		new->secondary->irq = new->irq;
		new->secondary->name = new->name;
	}
	/* Deal with the primary handler */
	set_bit(IRQTF_FORCED_THREAD, &new->thread_flags);
	new->thread_fn = new->handler;
	new->handler = irq_default_primary_handler;
	return 0;
}

Read the last three lines first, because they are the core of it. Your handler is moved into the thread_fn slot, and the handler slot is filled with a stub whose entire body is return IRQ_WAKE_THREAD;. From the interrupt core’s point of view your driver now looks exactly like one that called request_threaded_irq() with a NULL primary handler. The IRQTF_FORCED_THREAD bit is the only record that this was not what you wrote.

If you registered both halves, the block above runs too, and a second irqaction is allocated to carry your original thread function. Its handler field is set to irq_forced_secondary_handler, which is not a handler at all:

static irqreturn_t irq_forced_secondary_handler(int irq, void *dev_id)
{
	WARN(1, "Secondary action handler called for irq %d\n", irq);
	return IRQ_NONE;
}

It exists to be recognised, not to be run. If it ever executes, something has gone wrong and the WARN says so. Its real job is to be a marker, and two sections below is where that matters.

Why the exclusion list contains exactly those flags

Three flags switch forced threading off for an action, and each is a different kind of impossibility rather than a preference.

• IRQF_NO_THREAD is the explicit opt-out, and the only one a driver author chooses deliberately. It is also how timer interrupts escape, because IRQF_TIMER is a composite: #define IRQF_TIMER (__IRQF_TIMER | IRQF_NO_SUSPEND | IRQF_NO_THREAD). A driver that marks its interrupt as a timer has opted out of threading without writing the flag.
• IRQF_PERCPU cannot be threaded because the identity of the CPU is part of the interrupt’s meaning. A thread can be migrated; the per-CPU interrupt it serviced cannot follow it.
• IRQF_ONESHOT already set means the driver asked for the masked-until-the-thread-completes behaviour itself, so there is nothing to force.

There is a fourth exit, and it explains the stub. If new->handler is already irq_default_primary_handler, the function returns immediately, because the driver passed NULL as its primary handler and the core substituted the stub. That action is threaded by the driver’s own choice and needs no help. The check is an identity comparison against a function pointer, which is why a stub is used rather than a NULL handler: the core needs something it can compare against.

Whether this machinery is compiled in at all is not configurable from a menu. CONFIG_IRQ_FORCED_THREADING in kernel/irq/Kconfig is a bare bool with no prompt, selected by the architecture. At v7.3, arm64, arm and x86 all select it, so on any board this blog is about the machinery is present and the only question is whether it is active.

The two threads, and the priority that keeps them ordered

A forced-threaded driver that registered both halves gets two kernel threads, and they are named differently:

	if (!secondary) {
		t = kthread_create(irq_thread, new, "irq/%d-%s", irq,
				   new->name);
	} else {
		t = kthread_create(irq_thread, new, "irq/%d-s-%s", irq,
				   new->name);
	}

So irq/39-aerdrv runs what used to be your hard-IRQ handler, and irq/39-s-aerdrv runs what used to be your thread function. The -s- infix is the only clue in process listings that a secondary action exists. They are woken in order: the primary thread returning IRQ_WAKE_THREAD causes irq_wake_secondary() to wake the second one.

Waking in order is not the same as running in order, and that gap is where the AER loop came from. The repair is four lines at the top of irq_thread():

	if (action->handler == irq_forced_secondary_handler)
		sched_set_fifo_secondary(current);
	else
		sched_set_fifo(current);

This is what the poison handler is for. It is the marker that tells a thread it is the secondary one. The two priorities it selects between differ by one:

void sched_set_fifo(struct task_struct *p)
{
	struct sched_param sp = { .sched_priority = MAX_RT_PRIO / 2 };
	...
}

void sched_set_fifo_secondary(struct task_struct *p)
{
	struct sched_param sp = { .sched_priority = MAX_RT_PRIO / 2 - 1 };
	...
}

MAX_RT_PRIO is 100, so the forced primary thread runs at SCHED_FIFO 50 and the secondary at 49. The comment above sched_set_fifo_secondary() states the intent plainly: a lower priority “so that the primary handler can preempt it, thereby emulating the behavior of a non-PREEMPT_RT system where the primary handler runs in hard interrupt context”.

That is the design in one sentence. Forced threading removed a guarantee that came free with hard interrupt context, and the kernel rebuilds it out of a scheduling priority. It is a weaker reconstruction, because a priority can be changed at runtime and hard interrupt context could not, but it restores the ordering the driver depended on.

The change is recent. sched_set_fifo_secondary() arrived in mainline 7.0, from commit 51d0656959bc by Lukas Wunner dated 27 October 2025, reported by Crystal Wood and reviewed by Sebastian Andrzej Siewior. It is absent at the v6.18 and v6.12 tags, where irq_thread() calls sched_set_fifo(current) unconditionally. On those kernels, and on every older longterm series, both forced threads run at 50 and the ordering is not restored. If you maintain a driver with a primary and a thread handler on a 6.12 or 6.6 BSP running PREEMPT_RT, that is a live difference rather than a historical note.

What the thread runs instead of hard interrupt context

A handler written for hard interrupt context relies on more than priority. It assumes interrupts are disabled and that bottom halves cannot run underneath it. The kernel reconstructs part of that too, with a wrapper selected by the IRQTF_FORCED_THREAD bit set earlier:

static irqreturn_t irq_forced_thread_fn(struct irq_desc *desc, struct irqaction *action)
{
	irqreturn_t ret;

	local_bh_disable();
	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
		local_irq_disable();
	ret = irq_thread_fn(desc, action);
	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
		local_irq_enable();
	local_bh_enable();
	return ret;
}

On a non-RT kernel booted with threadirqs, your handler still runs with interrupts disabled, so its environment is close to what it expected. On PREEMPT_RT the two conditionals compile away and interrupts stay enabled. Bottom halves are still disabled, and that is all you get. A handler that relied on interrupt disabling to protect per-CPU data is now running without that protection, which is exactly the case Part 3 covered: the repair is a named per-CPU lock, not an attempt to restore the old environment.

Forced threading also reaches past the handlers themselves. In include/linux/interrupt.h, raise_timer_softirq() and local_timers_pending() both branch on force_irqthreads(), routing timer softirqs to raise_ktimers_thread() rather than raising them directly. So the setting does not only change where your handler runs; it changes how timer softirq work is dispatched, on an RT kernel and on a threadirqs kernel alike.

IRQF_NO_THREAD is the opt-out, and it is expensive

On RT there is one way to keep a handler in hard interrupt context, and it is IRQF_NO_THREAD. Before reaching for it, note what the handler then gives up. It cannot sleep, so it cannot take any RT-converted lock: spin_lock() is an rtmutex on RT, and a local_lock is a per-CPU rtmutex. A hard-IRQ handler is limited to raw_spinlock_t, and any data it shares with threaded code has to be reachable with a raw spinlock from both sides. Part 3’s local_trylock exists for precisely this split, and on RT it fails immediately in hard interrupt context by design.

So the flag is not a performance switch. It converts a latency problem into a locking constraint that propagates to every other user of the same data. The cases where it is right are the ones where the hardware leaves no choice: a line that must be acknowledged inside a window shorter than a thread wake-up, or a handler that is already raw-spinlock-only. Everything else is better served by leaving the handler threaded and setting its priority deliberately.

How to check this on your own board

First establish whether forced interrupt threading is active. On an RT kernel it always is, and the boot parameter is irrelevant there:

raghu@techveda.org:~$ uname -v
#1 SMP PREEMPT_RT Mon Oct 19 09:12:44 IST 2026
raghu@techveda.org:~$ grep -E 'CONFIG_PREEMPT_RT=|CONFIG_IRQ_FORCED_THREADING=' /boot/config-$(uname -r)
CONFIG_PREEMPT_RT=y
CONFIG_IRQ_FORCED_THREADING=y

Then list the interrupt threads with their scheduling policy and priority. This is the check that shows the split directly:

raghu@techveda.org:~$ ps -eo pid,class,rtprio,comm | grep '^ *[0-9]* FF' | grep irq/

A driver with both halves shows two rows whose rtprio values differ by one, and the secondary carries the -s- infix. On a kernel older than 7.0 the two values are identical, which is the condition to look for if a threaded driver is not making progress.

To find which handlers escaped threading, compare the interrupts your board reports against the threads that exist. An interrupt with no matching irq/N- thread is either per-CPU or marked IRQF_NO_THREAD:

raghu@techveda.org:~$ cat /proc/interrupts
raghu@techveda.org:~$ ls /proc/irq/

Changing a thread’s priority is the one adjustment available without rebuilding, and it applies to the running thread only:

raghu@techveda.org:~$ sudo chrt -f -p 60 $(pgrep -f 'irq/39-aerdrv')

Raise one of a pair and you break the ordering the kernel just restored, so change both or neither. Deciding which interrupts deserve which priority against the application threads around them is the design work we spend most time on in our Linux systems engineering training.

Limits, and what was not verified

The code is quoted from the v7.3 tag as it stands. The version boundary for sched_set_fifo_secondary() was established by reading kernel/sched/syscalls.c at v6.12, v6.18, v7.0, v7.1, v7.2 and v7.3, and irq_thread() at v6.18, so the claim that older longterm kernels run both threads at 50 rests on source rather than inference.

The AER failure is reported from the upstream commit message and the report it links, not reproduced here. The driver’s internal logic is summarised from that description rather than from reading drivers/pci/pcie/aer.c, so treat the account of which register matters as the reporter’s, not as a code reading.

No command output was captured on hardware. The listings above give the commands and a plausible shape for what they return; the interrupt number, the driver name and the build string are illustrative. Nothing here was timed, and the claim that forced threading changes when a shared line reopens follows from IRQF_ONESHOT semantics rather than from a measurement.

Key takeaways

• On PREEMPT_RT, force_irqthreads() is a compile-time true and the threadirqs parameter is not registered at all, so forced interrupt threading is a property of the kernel rather than a boot-time choice.
• Your handler is moved to the thread_fn slot and replaced by a stub returning IRQ_WAKE_THREAD. IRQF_ONESHOT is set at the same time, so the line stays masked until the thread finishes.
• A driver registering both a handler and a thread function gets two threads, irq/N-name and irq/N-s-name, woken in that order.
• Since mainline 7.0 the secondary runs at SCHED_FIFO 49 against the primary’s 50, to rebuild the preemption the hard-IRQ handler used to get for free. On 6.18 and older both are 50, and a real driver stopped making progress because of it.
• IRQF_NO_THREAD keeps a handler in hard interrupt context, at the price of raw spinlocks only and no local_lock on any data it touches.

Frequently asked questions

Does the threadirqs boot parameter do anything on a PREEMPT_RT kernel?
No. The parameter is registered inside a block guarded by !defined(CONFIG_PREEMPT_RT), so on an RT kernel it is not recognised at all. force_irqthreads() is already defined as true there.

Why does my driver suddenly have two interrupt threads?
Because it registered both a primary handler and a thread function, and forced threading moved the primary into a thread of its own while allocating a secondary action for the original thread function. The second one is named with an -s- infix.

Which interrupts are not threaded on PREEMPT_RT?
Those whose action carries IRQF_NO_THREAD, IRQF_PERCPU or an already-set IRQF_ONESHOT. Timer interrupts are covered because IRQF_TIMER is defined to include IRQF_NO_THREAD.

Can I keep a handler in hard interrupt context on RT?
Yes, with IRQF_NO_THREAD, but the handler then cannot sleep and so cannot take a spin_lock() or a local_lock, both of which are rtmutexes on RT. It is limited to raw_spinlock_t, and so is every other user of the data it touches.

Further reading

• genirq/manage: Reduce priority of forced secondary interrupt handler (commit 51d0656959bc) — the AER report and the priority fix.
• kernel/irq/manage.c — irq_setup_forced_threading(), irq_thread(), irq_forced_thread_fn(), setup_irq_thread() and the threadirqs parameter.
• include/linux/interrupt.h — the IRQF_ flag definitions, force_irqthreads() and the timer softirq routing.
• kernel/sched/syscalls.c — sched_set_fifo() and sched_set_fifo_secondary().
• Linux generic IRQ handling — the core API documentation for the interrupt subsystem.
• The kernel’s command-line parameters — the threadirqs entry.
• Lock types and their rules — which lock types are legal in hard interrupt context on PREEMPT_RT.