Real-Time Linux with PREEMPT_RT — Part 3: SCHED_FIFO Priorities and CPU Isolation

This is the third part of our hands-on series on real-time Linux. In Part 2 you built a PREEMPT_RT kernel from mainline source, booted it, and confirmed that the maximum cyclictest latency dropped from thousands of microseconds to a bounded worst case. The kernel alone is not the complete solution. In this part you will place your real-time task correctly on that kernel: choose a SCHED_FIFO priority with intent, set it with chrt, allow a non-root user to use real-time priorities, configure CPU isolation so one core serves your critical task with as little kernel interference as possible, and then verify the result with cyclictest and rtla timerlat.

What you need

• The PREEMPT_RT kernel you built and verified in Part 2, confirmed by cat /sys/kernel/realtime returning 1.
• The rt-tests package for cyclictest, and stress-ng for load generation, both used in the earlier parts.
• The rtla package for the verification step (sudo apt install rtla on Debian and Ubuntu).
• A machine with at least 4 CPUs. The examples isolate CPU 3 on a 4-CPU system; adjust the CPU numbers for your hardware.
• Root access through sudo.

How SCHED_FIFO priorities work

Linux schedules ordinary tasks under the SCHED_OTHER policy, where the nice value influences how CPU time is shared. Real-time tasks use the SCHED_FIFO or SCHED_RR policies instead, with a static priority between 1 (low) and 99 (high). Any runnable SCHED_FIFO task always preempts every SCHED_OTHER task, regardless of nice values. Between real-time tasks, the higher priority runs first, and a SCHED_FIFO task keeps the CPU until it blocks, yields, or is preempted by a higher priority — there is no time slicing at the same priority.

One PREEMPT_RT detail matters for priority planning. On a PREEMPT_RT kernel, hardware interrupt handlers run as kernel threads scheduled under SCHED_FIFO, and their default priority is 50. You can see them with:

raghu@techveda.org:~$ ps -eo pid,cls,rtprio,comm | grep irq/
   45  FF     50 irq/27-eth0
   52  FF     50 irq/30-i2c_designware
   61  FF     50 irq/31-xhci_hcd

This gives you a real decision to make. A task at priority 80 preempts interrupt processing for the network and storage devices; a task at priority 40 does not. Place your application above 50 only if it must not wait for device interrupt handling, and keep it below 99 so system management threads and an emergency shell can still run above it. Our cyclictest runs use priority 80, which follows this reasoning.

Set real-time priorities with chrt

The chrt tool from util-linux starts a program under a real-time policy or changes a running task. Start a program under SCHED_FIFO at priority 80:

raghu@techveda.org:~$ sudo chrt -f 80 ./rt_app

Inspect a running task by PID:

raghu@techveda.org:~$ chrt -p 1234
pid 1234's current scheduling policy: SCHED_FIFO
pid 1234's current scheduling priority: 80

Change the priority of a running task:

raghu@techveda.org:~$ sudo chrt -f -p 70 1234

By default only root can set a real-time priority. For an unprivileged user, the kernel enforces the RLIMIT_RTPRIO resource limit, which defines the highest real-time priority that user may set. Check it with:

raghu@techveda.org:~$ ulimit -r
0

A value of 0 means no real-time priority is allowed. To let a user set priorities up to 90, add one line to /etc/security/limits.conf and log in again:

raghu    -    rtprio    90

After a fresh login, ulimit -r reports 90 and chrt -f 80 ./rt_app works without sudo. This is the correct setup for deployed systems, where running the application as root only to obtain a priority is not acceptable.

Real-time throttling protects the system

A SCHED_FIFO task that never blocks would keep its CPU forever and lock out everything else. The kernel guards against this with two sysctl values, introduced in Part 2 and relevant again here:

raghu@techveda.org:~$ cat /proc/sys/kernel/sched_rt_period_us
1000000
raghu@techveda.org:~$ cat /proc/sys/kernel/sched_rt_runtime_us
950000

Together they mean real-time tasks may consume at most 950,000 microseconds of every 1,000,000-microsecond period; 5% of CPU time is reserved for non-real-time tasks. A busy real-time loop therefore pauses briefly every second. Writing -1 to sched_rt_runtime_us removes the reservation, but then a runaway real-time task can make the machine unreachable, so change it only on systems where the real-time application is trusted and tested. Tasks that sleep between activations, such as cyclictest, stay far below the limit and are not affected.

Configure CPU isolation on the kernel command line

Priority decides which task gets the CPU; CPU isolation decides what else runs on that CPU at all. The kernel documentation describes three cooperating mechanisms, and we set all three for CPU 3 on a 4-CPU system. isolcpus=domain,managed_irq,3 removes CPU 3 from scheduler load balancing, so no task lands there unless explicitly pinned, and asks the kernel to keep managed device IRQs away from it. nohz_full=3 stops the periodic scheduler tick on CPU 3 while it runs a single task. irqaffinity=0-2 sets the default interrupt affinity so device interrupts are delivered to the housekeeping CPUs 0 to 2.

Edit the GRUB configuration and append the parameters:

raghu@techveda.org:~$ sudo nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=domain,managed_irq,3 nohz_full=3 irqaffinity=0-2"
raghu@techveda.org:~$ sudo update-grub
raghu@techveda.org:~$ sudo reboot

After the reboot, confirm that the parameters took effect:

raghu@techveda.org:~$ cat /proc/cmdline
... isolcpus=domain,managed_irq,3 nohz_full=3 irqaffinity=0-2
raghu@techveda.org:~$ cat /sys/devices/system/cpu/isolated
3
raghu@techveda.org:~$ cat /sys/devices/system/cpu/nohz_full
3
raghu@techveda.org:~$ cat /proc/irq/default_smp_affinity
7

The value 7 is the CPU mask 0111 in binary, covering CPUs 0 to 2. Also note the trade-off stated plainly in the kernel documentation: boot-time isolcpus cannot be changed at runtime. The runtime alternative is a cgroup v2 cpuset partition, where writing isolated to cpuset.cpus.partition isolates CPUs on a live system; the boot parameters remain the simpler and more common choice on dedicated embedded devices.

Check and move individual IRQs

irqaffinity= sets only the default affinity. An interrupt that received an explicit affinity earlier, from a driver or a tool, can still point at the isolated CPU. The per-CPU counters in /proc/interrupts show whether anything still fires on CPU 3: read the file twice a few seconds apart and compare the CPU3 column. Any interrupt whose count keeps rising there needs attention. Each interrupt has its affinity exposed under /proc/irq/, and you can move one by hand:

raghu@techveda.org:~$ cat /proc/irq/27/smp_affinity_list
0-3
raghu@techveda.org:~$ echo 0-2 | sudo tee /proc/irq/27/smp_affinity_list
0-2

Two limits apply. Per-CPU interrupts, such as each CPU’s local timer, are per-CPU by design and cannot be moved. And a write to smp_affinity_list fails with an input/output error for interrupts whose controller or driver does not allow the change. Both cases are normal; the goal is to move every device interrupt that can be moved, not to reach a count of zero.

Measure the effect with cyclictest

Now combine everything. Pin one cyclictest measurement thread at priority 80 to the isolated CPU 3, and keep the load generator on the housekeeping CPUs so it competes with the measurement only through shared hardware, not through the scheduler:

raghu@techveda.org:~$ taskset -c 0-2 stress-ng --cpu 3 --io 2 --vm 1 --vm-bytes 256M --timeout 600s &
raghu@techveda.org:~$ sudo cyclictest --mlockall --priority=80 --interval=200 --affinity=3 --threads=1

Let it run for several minutes, as in the earlier parts. The numbers below are illustrative only; absolute values depend on your hardware and firmware. The pattern to expect looks like this:

PREEMPT_RT kernel, no isolation (Part 2 result, under load):
T: 0 (1044)  P:80  I:200  C: 921550  Min:  2  Avg:  3  Max:  72

PREEMPT_RT kernel, isolated CPU 3 (this part, same load):
T: 0 (1201)  P:80  I:200  C: 2954021  Min:  1  Avg:  2  Max:  34

The average barely changes, and that is expected. The gain is again in the worst case: with device interrupts, the scheduler tick, and unrelated tasks moved off CPU 3, the sources of occasional long delays are removed, so the maximum drops and repeat runs become more consistent. For ordinary applications, start the pinned task the same way: sudo taskset -c 3 chrt -f 80 ./rt_app. Scheduling policy is covered in depth in our Linux Systems Engineering training.

Verify the quiet CPU with rtla timerlat

cyclictest reports the size of the worst case; the kernel’s own rtla tool reports where the delay comes from. Its timerlat mode runs a measurement thread on each selected CPU and separates the latency into two columns: the delay of the timer interrupt itself (IRQ) and the additional delay until the measurement thread actually ran (Thread). The tool requires the timerlat tracer in the kernel; our build inherited the distribution configuration, so confirm and run it:

raghu@techveda.org:~$ grep TIMERLAT_TRACER /boot/config-$(uname -r)
CONFIG_TIMERLAT_TRACER=y
raghu@techveda.org:~$ sudo rtla timerlat top -c 3 -d 60s
                                   Timer Latency
  0 00:01:00   |          IRQ Timer Latency (us)        |         Thread Timer Latency (us)
CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max
  3 #59988     |        1         0         1         4 |        6         2         5        12

On a well-isolated core, both maxima stay in the low microseconds. The measurement threads run at SCHED_FIFO priority 95 by default, above our application priority of 80, which is intentional for a measurement tool. The feature that makes rtla more than a second opinion is the automatic trace mode: with sudo rtla timerlat top -c 3 -a 40, tracing stops the moment the thread latency exceeds 40 microseconds, and the tool prints which task or interrupt blocked the measurement thread, including a kernel stack trace, and saves the raw trace to timerlat_trace.txt. That turns “the maximum is too high” into “this specific code path caused it” in one step. The related rtla osnoise mode measures how much CPU time the kernel takes away from a busy workload on the same cores; we will use the underlying tracing machinery in more depth in Part 5.

Hardware limits: idle states, frequency scaling, and SMIs

Software isolation cannot remove delays that come from the hardware and firmware, and three of them set the floor for your numbers. First, deep CPU idle states: a core in a deep C-state needs time to wake up, which appears as latency on the first event after an idle period. Note that cyclictest holds /dev/cpu_dma_latency at 0 by default, which keeps the CPU out of deep idle states during the run — your application does not get this behaviour unless it opens that file itself, so application latency can look worse than cyclictest latency for this reason alone.

Idle depth can be limited with boot parameters such as processor.max_cstate or intel_idle.max_cstate on x86. Second, frequency scaling: a core that has clocked down needs time to ramp up, so latency-sensitive systems commonly pin the governor with sudo cpupower frequency-set -g performance. Third, System Management Interrupts on x86: firmware-level interrupts the kernel cannot see or block.

The kernel documentation’s isolation checklist recommends checking BIOS options for low-latency operation, and rtla hwnoise measures noise that arrives with interrupts disabled, which exposes exactly this class of disturbance. On processors with SMT, also add nosmt to the kernel command line, since the sibling hardware thread disturbs the isolated core below the operating system’s control.

Key takeaways

• SCHED_FIFO priorities run from 1 to 99, and any SCHED_FIFO task preempts all SCHED_OTHER tasks; at equal priority there is no time slicing.
• On a PREEMPT_RT kernel, interrupt handlers run as SCHED_FIFO threads at default priority 50; placing your task above or below 50 decides whether it waits for device interrupt handling.
• Use chrt -f to set priorities, and grant unprivileged users real-time priority through the rtprio entry in /etc/security/limits.conf rather than running as root.
• The kernel reserves 5% of CPU time for non-real-time tasks through sched_rt_runtime_us; disable it only with a clear reason.
• Isolate a core with isolcpus=domain,managed_irq,N, nohz_full=N, and irqaffinity pointing at the housekeeping CPUs, then pin the real-time task there with taskset or --affinity.
• irqaffinity= sets only the default: watch the isolated CPU’s column in /proc/interrupts and move remaining interrupts through /proc/irq/N/smp_affinity_list.
• Use rtla timerlat to split latency into IRQ and thread components, and its -a mode to capture a stack trace of whatever blocked the measurement.
• Deep C-states, frequency scaling, and SMIs set a hardware floor under your latency; cyclictest masks the C-state effect by holding /dev/cpu_dma_latency at 0.
• Isolation improves the maximum latency and run-to-run consistency, not the average.

What’s next in this series

The series continues with Part 4 next Sunday. We will keep the topic a surprise this time — the tuned system you built in this part will be the starting point.

Frequently asked questions

What SCHED_FIFO priority should my application use on a PREEMPT_RT kernel?
Priorities run from 1 to 99, and interrupt handler threads default to priority 50. Set your task above 50 only if it must not wait for device interrupt handling, and keep it below 99 so system management threads can still preempt it.

Can a non-root user run a SCHED_FIFO task?
Yes. Add an rtprio entry for that user in /etc/security/limits.conf, for example allowing priorities up to 90, and after a fresh login the user can start real-time tasks with chrt without sudo.

What does isolcpus=domain actually do?
It removes the listed CPUs from scheduler load balancing, so no task runs there unless it is explicitly pinned with taskset or a CPU affinity call. It is a boot-time setting; the runtime alternative is a cgroup v2 cpuset isolated partition.

Why does a busy real-time task pause briefly every second?
By default the kernel limits real-time tasks to 950,000 microseconds of every 1,000,000-microsecond period, reserving 5% of CPU time for non-real-time tasks. Writing -1 to /proc/sys/kernel/sched_rt_runtime_us removes the limit, at the risk of a runaway task locking up the system.

Further reading

• sched(7) — overview of CPU scheduling, Linux man-pages
• CPU Isolation — The Linux Kernel documentation
• rtla: The realtime Linux analysis tool — The Linux Kernel documentation
• Real-Time group scheduling — The Linux Kernel documentation
• cyclictest — Linux Foundation real-time documentation