What PREEMPT_RT Changes: The ktimers Thread (Part 6)

This series has read PREEMPT_RT one mechanism at a time: sleeping spinlocks, priority inheritance, local_lock and migrate_disable(), forced interrupt threading, and then preemptible softirqs in Part 5. Part 5 left one thread unexamined on purpose. Once softirq processing is under scheduler control, the question of which thread runs which softirq, and at what priority, decides whether a timer expires on time. For timers, the answer on a current kernel is the ktimers thread, one instance per CPU, and this part reads it.

Where timer work went before the ktimers thread

Start with the code that decides a timer has expired. Every tick, the timer interrupt eventually calls run_local_timers() in kernel/time/timer.c. At v6.12 that function ends like this:

		if (time_after_eq(jiffies, READ_ONCE(base->next_expiry)) ||
		    (i == BASE_DEF && tmigr_requires_handle_remote())) {
			raise_softirq(TIMER_SOFTIRQ);
			return;
		}

The hrtimer side does the equivalent. In hrtimer_run_queues() and hrtimer_interrupt(), once the soft-expiry deadline has passed, the kernel sets cpu_base->softirq_activated = 1 and raises HRTIMER_SOFTIRQ.

Raising a softirq only sets a bit in a per-CPU mask. Running it is a separate decision, taken in __irq_exit_rcu() on the way out of the interrupt, and on PREEMPT_RT at v6.12 it is short:

#ifdef CONFIG_PREEMPT_RT
static inline void invoke_softirq(void)
{
	if (should_wake_ksoftirqd())
		wakeup_softirqd();
}

There is no in-place execution on the interrupt stack. The work is handed to ksoftirqd, and ksoftirqd is registered with no .setup callback at all:

static struct smp_hotplug_thread softirq_threads = {
	.store			= &ksoftirqd,
	.thread_should_run	= ksoftirqd_should_run,
	.thread_fn		= run_ksoftirqd,
	.thread_comm		= "ksoftirqd/%u",
};

No .setup means no call to any sched_set_* helper, so ksoftirqd/N keeps the policy it was created with: SCHED_OTHER. That is the problem in one sentence. The kernel has just learned, in hard interrupt context, that a timer is due, and it then asks the fair scheduler to run a normal-priority thread so the timer can actually expire.

The failure mode is gradual rather than sudden. On an idle board the delay is invisible: ksoftirqd is the only runnable task, so the timer expires within microseconds of the tick. Add a logging daemon, a compression pass or a video pipeline — anything CPU-bound at SCHED_OTHER — and ksoftirqd waits for a slice of the fair scheduler like every other normal task. The more such load the board carries, the longer the wait. The commit message states it plainly: because ksoftirqd runs as a SCHED_OTHER task it competes with other tasks for CPU, which can delay timers on a heavily loaded system.

What the ktimers thread is

The change is commit 49a17639508c, written by Sebastian Andrzej Siewior on 6 November 2024 and committed by Thomas Gleixner the next day. It landed in mainline Linux 6.13. Its commit message describes the change as splitting TIMER_SOFTIRQ and HRTIMER_SOFTIRQ processing into a dedicated timers thread, running at the lowest SCHED_FIFO priority so that timer work is handled before SCHED_OTHER tasks get the CPU.

In the source it is one more per-CPU smpboot thread, sitting next to ksoftirqd in the same file:

static struct smp_hotplug_thread timer_thread = {
	.store			= &ktimerd,
	.setup			= ktimerd_setup,
	.thread_should_run	= ktimerd_should_run,
	.thread_fn		= run_ktimerd,
	.thread_comm		= "ktimers/%u",
};

Two differences from the ksoftirqd struct matter. The thread name is ktimers/%u, so a four-core board shows ktimers/0 through ktimers/3 in a process listing. And there is a .setup callback, which exists for one purpose:

static void ktimerd_setup(unsigned int cpu)
{
	/* Above SCHED_NORMAL to handle timers before regular tasks. */
	sched_set_fifo_low(current);
}

Registration is conditional, and this is the first fact that surprises people:

#ifdef CONFIG_IRQ_FORCED_THREADING
	if (force_irqthreads())
		BUG_ON(smpboot_register_percpu_thread(&timer_thread));
#endif

The guard is CONFIG_IRQ_FORCED_THREADING, not CONFIG_PREEMPT_RT, and the whole block sits under that symbol. On PREEMPT_RT, force_irqthreads() is the constant (true), so the thread always exists. Without PREEMPT_RT it is a static branch that the threadirqs boot parameter turns on, so a plain kernel booted with threadirqs gets ktimers threads too. As Part 4 showed, threadirqs is not even recognised on an RT kernel, because there it would be redundant. On arm64 the option is always available: arch/arm64/Kconfig contains select IRQ_FORCED_THREADING.

How the ktimers thread works

The mechanism has three pieces: a second pending mask, a raise function that only writes to it, and a wakeup on the way out of the interrupt. The mask is a plain per-CPU word, declared beside the thread pointer:

DEFINE_PER_CPU(struct task_struct *, ktimerd);
DEFINE_PER_CPU(unsigned long, pending_timer_softirq);

The raise function does nothing but set a bit in it, plus the tracepoint every softirq raise emits:

void raise_ktimers_thread(unsigned int nr)
{
	trace_softirq_raise(nr);
	__this_cpu_or(pending_timer_softirq, BIT(nr));
}

Timer code does not call that directly. It calls a wrapper in include/linux/interrupt.h which picks the destination:

static inline void raise_timer_softirq(unsigned int nr)
{
	lockdep_assert_in_irq();
	if (force_irqthreads())
		raise_ktimers_thread(nr);
	else
		__raise_softirq_irqoff(nr);
}

So the line in run_local_timers() that read raise_softirq(TIMER_SOFTIRQ) at v6.12 reads raise_timer_softirq(TIMER_SOFTIRQ) at v6.18, and both hrtimer raises go through the same wrapper. The lockdep_assert_in_irq() is not decoration: it records that this path is legal only from interrupt context, which is what allows the cheap non-atomic __this_cpu_or() on the mask.

Waking the thread is a separate step in __irq_exit_rcu(), beside the existing softirq check:

	if (IS_ENABLED(CONFIG_IRQ_FORCED_THREADING) && force_irqthreads() &&
	    local_timers_pending_force_th() && !(in_nmi() | in_hardirq()))
		wake_timersd();

The thread body merges its private mask back into the main one and then runs the standard softirq loop:

static void run_ktimerd(unsigned int cpu)
{
	unsigned int timer_si;

	ksoftirqd_run_begin();

	timer_si = local_timers_pending_force_th();
	__this_cpu_write(pending_timer_softirq, 0);
	or_softirq_pending(timer_si);

	__do_softirq();

	ksoftirqd_run_end();
}

Why a second mask at all, when the thread ORs its contents into the first one anyway? The commit message answers it: the separate per-CPU variable keeps timer softirqs from being picked up by ksoftirqd and by other threaded interrupt handlers. Part 5 showed that on PREEMPT_RT a threaded handler runs its own softirq work inline in the thread that re-enabled bottom halves. If timer bits sat in the shared mask, any of those threads could consume them at its own priority. Putting the bits where only ktimers/N looks makes the destination deterministic.

Why the priority is 1 and not 50

This is the decision in the patch that looks wrong at first reading. The kernel gives threaded interrupt handlers priority 50; the ktimers thread gets 1, the lowest real-time priority there is. Both numbers come from kernel/sched/syscalls.c, where the comments explain the reasoning better than any summary:

/*
 * SCHED_FIFO is a broken scheduler model; that is, it is fundamentally
 * incapable of resource management, which is the one thing an OS really should
 * be doing.
 *
 * ...
 *
 * For this reason 'all' FIFO tasks the kernel creates are basically at:
 *
 *   MAX_RT_PRIO / 2
 *
 * The administrator _MUST_ configure the system, the kernel simply doesn't
 * know enough information to make a sensible choice.
 */
void sched_set_fifo(struct task_struct *p)
{
	struct sched_param sp = { .sched_priority = MAX_RT_PRIO / 2 };
	WARN_ON_ONCE(sched_setscheduler_nocheck(p, SCHED_FIFO, &sp) != 0);
}

/*
 * For when you don't much care about FIFO, but want to be above SCHED_NORMAL.
 */
void sched_set_fifo_low(struct task_struct *p)
{
	struct sched_param sp = { .sched_priority = 1 };
	WARN_ON_ONCE(sched_setscheduler_nocheck(p, SCHED_FIFO, &sp) != 0);
}

Read the second comment literally and the choice stops looking odd. The goal was never to make timer processing urgent. It was to get timer processing out of the fair scheduler, where its delay depends on how much other normal work exists. Priority 1 achieves that and nothing more: ahead of every SCHED_OTHER task on the CPU, behind every real-time thread an application has created, because any sensible application priority is above 1.

That ordering suits a product. If your control loop runs at SCHED_FIFO 80, it should run first and the timer bookkeeping in the gap that follows. At priority 50 the thread would sit among application threads and threaded handlers, and every integrator would have to reason about it. At priority 1 it needs no configuration to be correct.

The same jiffy on 6.12 and on 6.18

One tick through both kernels shows that the difference is a single step.

Nothing else moved. The expiry functions, __do_softirq() and the handlers registered with open_softirq() are the same. Only one step changed hands, and it was the step exposed to SCHED_OTHER competition.

One limit on this comparison: which timers reach the softirq at all, rather than expiring in hard interrupt context, is decided by the hrtimer mode flags set where the timer is created. This article reads the dispatch path, not that survey, so it tells you where softirq timer work ends up and not which of your own timers are affected.

What changed in the NOHZ idle check

Splitting the mask had a consequence elsewhere. The tickless idle code needs to know whether timer softirq work is pending before it stops the tick. At v6.12 it asked the shared mask directly:

static inline bool local_timer_softirq_pending(void)
{
	return local_softirq_pending() & BIT(TIMER_SOFTIRQ);
}

With timer bits living elsewhere, that question would have returned the wrong answer without any warning. So a second accessor was added in the header, which asks whichever mask is in use:

static inline unsigned int local_timers_pending(void)
{
	if (force_irqthreads())
		return local_timers_pending_force_th();
	else
		return local_softirq_pending();
}

and kernel/time/tick-sched.c at v6.18 calls that instead. Note what did not change in the same file. The idle-entry warning that engineers see in boot logs still reads the shared mask:

static bool report_idle_softirq(void)
{
	static int ratelimit;
	unsigned int pending = local_softirq_pending();

	if (likely(!pending))
		return false;
	...
	if (ratelimit < 10) {
		pr_warn("NOHZ tick-stop error: local softirq work is pending, handler #%02x!!!\n",
			pending);
		ratelimit++;
	}

So on a kernel with the new timer thread, that diagnostic reports on the shared mask and not on pending timer work. If you have used the handler number in that message to reason about timers, the mapping is no longer complete on 6.13 and later.

Which kernels have the ktimers thread

Here the change becomes a product question. Each version below was checked by fetching the files at that tag and looking for the symbols, not inferred from the merge date.

If your product ships 6.12, which is the longterm series a great many embedded teams moved to and will stay on, you do not have the ktimers thread and you will not get it from a stable update. Timer softirq work on that kernel is still handled by a SCHED_OTHER thread. Moving to 6.18 is what changes it, and this is one concrete item for the list of reasons to make that move.

What it costs, and where it does not help

The thread drains more than timers. run_ktimerd() ORs its private mask into the shared pending mask and then calls __do_softirq(), which processes everything the shared mask holds. If network or RCU work happened to be pending on that CPU, it runs in ktimers/N at SCHED_FIFO 1 rather than in ksoftirqd. That is what the code does; whether it matters on your board is a measurement question this article does not answer.

Priority 1 is still a priority you can starve. Any application thread at SCHED_FIFO 2 or above preempts the ktimers thread. A thread that busy-loops at priority 50 without ever blocking keeps timer processing off that CPU for as long as it runs. The change removes the fair scheduler from the path; it does not create a reservation. This is the trap for an experienced engineer: having read that timers are now real-time, they stop suspecting the timer path, while the starvation case still exists.

It costs one thread per CPU. Registration uses BUG_ON(), so a failure to create the thread at boot is fatal rather than degraded.

It does not change interrupt latency. The timer interrupt still runs where it ran before, and the decision that a timer is due is still made in hard interrupt context. What changed is who performs the expiry work afterwards.

Two notes on scope. No latency numbers appear above: nothing was measured on hardware, and every claim is a reading of source at a named tag or of the commit message. The backport check covers 6.12.115 only, so on 6.6 or 6.1 run the check below rather than assume.

How to check your own board

Every check below works on a running board. The sample outputs show the format to expect; they are illustrative, not a capture from one specific board. First, confirm what kind of kernel you are on, because the ktimers thread exists only where forced threading is active:

raghu@techveda.org:~$ uname -r
6.18.60-rt8-mydistro
raghu@techveda.org:~$ uname -v | grep -o 'PREEMPT_RT'
PREEMPT_RT

Next, check the option the whole block depends on:

raghu@techveda.org:~$ zgrep IRQ_FORCED_THREADING /proc/config.gz
CONFIG_IRQ_FORCED_THREADING=y

If /proc/config.gz is absent, the same symbol is in the /boot/config-$(uname -r) file most build systems install. Now look for the threads, with their policy and priority:

raghu@techveda.org:~$ ps -eo comm,class,rtprio | grep -E 'ktimers|ksoftirqd'
ksoftirqd/0     TS      -
ktimers/0       FF      1
ksoftirqd/1     TS      -
ktimers/1       FF      1

Those rows, trimmed here to the first two CPUs, answer the question directly. TS is SCHED_OTHER with no real-time priority; FF is SCHED_FIFO, at the priority ktimerd_setup() set. On a 6.12 kernel the ktimers rows are absent. chrt confirms the same for one thread:

raghu@techveda.org:~$ chrt -p $(pgrep -x 'ktimers/0')
pid 24's current scheduling policy: SCHED_FIFO
pid 24's current scheduling priority: 1

Timer softirq activity is still counted per CPU in /proc/softirqs, whichever thread performed the work:

raghu@techveda.org:~$ grep -E 'TIMER|HRTIMER' /proc/softirqs
      TIMER:     184233     181904     182551     181677
    HRTIMER:         41         38         44         39

A useful exercise: put a CPU-bound SCHED_OTHER load on one core, pin a thread that sleeps on a timer to the same core, and compare the wakeup delay on 6.12 and on 6.18. Reading the scheduling class of a kernel thread as a first-line diagnostic is part of our Linux systems engineering training.

Key takeaways

• From Linux 6.13, timer and hrtimer softirq work under forced interrupt threading runs in a per-CPU ktimers/N thread at SCHED_FIFO priority 1, not in ksoftirqd.
• The ktimers thread has its own pending mask, pending_timer_softirq, so no other thread can consume timer work at the wrong priority.
• Priority 1 is deliberate: the aim was to leave the fair scheduler, not to outrank application real-time threads.
• The guard is CONFIG_IRQ_FORCED_THREADING, so a non-RT kernel booted with threadirqs gets the thread as well.
• 6.18 is the oldest longterm series with it; 6.12.115 does not have it, so a 6.12 product will not receive it through stable updates.

Frequently asked questions

What is the ktimers thread in Linux?
It is a per-CPU kernel thread, named ktimers/0, ktimers/1 and so on, that processes timer and hrtimer softirq work. It was added in Linux 6.13 and runs at SCHED_FIFO priority 1, ahead of all normal tasks on its CPU.

Does my 6.12 kernel have the ktimers thread?
No. The code is absent from kernel/softirq.c at both v6.12 and v6.12.115, and was not backported into that longterm series. On 6.12, timer softirq work is handled by ksoftirqd, which runs at SCHED_OTHER.

Why does the ktimers thread run at priority 1 instead of 50?
Because the goal was to move timer processing out of the fair scheduler, not to make it outrank application threads. Priority 1 puts it above every SCHED_OTHER task and below any application real-time thread, which needs no configuration to be correct.

Do I need PREEMPT_RT to get the ktimers thread?
No. The code is guarded by CONFIG_IRQ_FORCED_THREADING and registered when force_irqthreads() is true. That is always true on PREEMPT_RT, and also true on a non-RT kernel booted with threadirqs.

Can an application thread still delay timer processing?
Yes. Any thread at SCHED_FIFO priority 2 or above preempts the ktimers thread, so a real-time thread that never blocks can keep timer work off that CPU. The change removes the fair scheduler from the path; it grants no share of the CPU.

Further reading

• Commit 49a17639508c — softirq: Use a dedicated thread for timer wakeups on PREEMPT_RT
• Patch series cover letter on LWN
• Real-time theory of operation
• kernel/softirq.c at v6.18
• include/linux/interrupt.h at v6.18
• kernel/time/tick-sched.c at v6.18
• Active kernel releases