Real-Time Linux with PREEMPT_RT — Part 2: Building a PREEMPT_RT Kernel and Comparing Latency

This is the second part of our hands-on series on real-time Linux. In Part 1 we explained what PREEMPT_RT changes inside the kernel and measured a latency baseline on a standard kernel using cyclictest. In this part you will build a PREEMPT_RT kernel from mainline source, install and boot it, confirm that the real-time configuration is active, and then re-run the exact same cyclictest command to compare the maximum latency against that baseline. Building a PREEMPT_RT kernel is the step that turns the theory from Part 1 into a system you can measure and trust.

What you need

• The same x86_64 Linux machine you used in Part 1, with root access through sudo. The steps also apply to arm64 with the usual cross-compilation changes.
• The baseline cyclictest numbers you recorded in Part 1. Keep them; the whole point of this part is the comparison.
• Roughly 20 GB of free disk space and time for a full kernel build.
• The rt-tests package installed in Part 1, which provides cyclictest.

We use mainline Linux 6.12, the first release where the core PREEMPT_RT support was merged into the tree, so no external patch is strictly required on x86_64, arm64, or RISC-V. For older kernels the PREEMPT_RT patch is mandatory, and even on 6.12 and later the maintained patch adds further architectures and improvements.

Install the build dependencies

On Debian or Ubuntu, install the tools needed to configure and compile the kernel:

raghu@techveda.org:~$ sudo apt update
raghu@techveda.org:~$ sudo apt install build-essential libncurses-dev bison flex libssl-dev libelf-dev bc dwarves

The dwarves package provides pahole, which recent kernels use to generate BTF type information during the build. The libncurses-dev package is needed for the menuconfig interface.

Get the kernel source

Download a stable 6.12 tarball from kernel.org and unpack it:

raghu@techveda.org:~$ cd ~
raghu@techveda.org:~$ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.12.tar.xz
raghu@techveda.org:~$ tar xf linux-6.12.tar.xz
raghu@techveda.org:~$ cd linux-6.12

A good starting configuration is the one your running kernel already uses. Copy it and let the build system fill in any new options with their defaults:

raghu@techveda.org:~$ cp /boot/config-$(uname -r) .config
raghu@techveda.org:~$ make olddefconfig

Clear the Ubuntu signing keys before building

This is a common stopping point. On Ubuntu, the distribution config points the kernel’s module-signing options at certificate files that ship only inside Canonical’s own kernel source package. When you build from a plain kernel.org tree those files are not present, and the build stops with an error like this:

make[1]: *** No rule to make target 'debian/canonical-certs.pem', needed by 'certs/x509_certificate_list'.  Stop.

A second, similar error mentioning debian/canonical-revoked-certs.pem can appear for the revocation list. The two responsible options are CONFIG_SYSTEM_TRUSTED_KEYS and CONFIG_SYSTEM_REVOCATION_KEYS, and both hold a file path as a string. Set them to an empty string so the build no longer looks for the missing certificates:

raghu@techveda.org:~$ scripts/config --set-str SYSTEM_TRUSTED_KEYS ""
raghu@techveda.org:~$ scripts/config --set-str SYSTEM_REVOCATION_KEYS ""

This leaves the lines as CONFIG_SYSTEM_TRUSTED_KEYS="" and CONFIG_SYSTEM_REVOCATION_KEYS="" in .config. It removes the certificate requirement for a local development build without turning off module support. You can confirm the change:

raghu@techveda.org:~$ grep -E "SYSTEM_TRUSTED_KEYS|SYSTEM_REVOCATION_KEYS" .config
CONFIG_SYSTEM_TRUSTED_KEYS=""
CONFIG_SYSTEM_REVOCATION_KEYS=""

Enable the real-time configuration

The single most important setting is CONFIG_PREEMPT_RT, shown in the menu as “Fully Preemptible Kernel (Real-Time)”. Open the configuration interface:

raghu@techveda.org:~$ make menuconfig

Navigate to “General setup”, then “Preemption Model”, and select “Fully Preemptible Kernel (Real-Time)”. If that option is not visible, first enable “Configure standard kernel features (expert users)” under “General setup”, which sets CONFIG_EXPERT, then return to the Preemption Model menu. Save the configuration and exit.

It is worth setting a local version string so the new kernel is easy to identify in the boot menu and in uname. Under “General setup”, set “Local version” to -rt, or do it from the command line:

raghu@techveda.org:~$ scripts/config --set-str LOCALVERSION "-rt"

A few debugging options have a large effect on latency and should be off for a real-time kernel. The Linux Foundation guide names DEBUG_PREEMPT, DEBUG_LOCKDEP, DEBUG_OBJECTS, and SLUB_DEBUG. Distribution configs often enable some of these. Disable them and let the build system resolve any dependencies:

raghu@techveda.org:~$ scripts/config --disable DEBUG_PREEMPT
raghu@techveda.org:~$ scripts/config --disable DEBUG_LOCKDEP
raghu@techveda.org:~$ scripts/config --disable DEBUG_OBJECTS
raghu@techveda.org:~$ scripts/config --disable SLUB_DEBUG
raghu@techveda.org:~$ make olddefconfig

Confirm that the real-time option is actually set before you spend time compiling:

raghu@techveda.org:~$ grep PREEMPT_RT .config
CONFIG_PREEMPT_RT=y

Build and install the PREEMPT_RT kernel

Compile the kernel and modules in parallel, using one job per CPU:

raghu@techveda.org:~$ make -j$(nproc)

This step can take from several minutes to over an hour depending on the machine. When it finishes, install the modules, then the kernel image:

raghu@techveda.org:~$ sudo make modules_install
raghu@techveda.org:~$ sudo make install

On Debian and Ubuntu, make install copies the kernel image to /boot, generates an initramfs, and triggers a boot loader update through the kernel install hooks. If your system does not update the GRUB menu automatically, do it explicitly:

raghu@techveda.org:~$ sudo update-grub

Boot the new kernel and confirm PREEMPT_RT is active

Reboot and select the new kernel. On most desktops it appears under “Advanced options for Ubuntu” (or your distribution’s equivalent) in the GRUB menu. After it boots, verify two things. First, that the running kernel is the one you built and that its version string contains PREEMPT_RT:

raghu@techveda.org:~$ uname -a
Linux box 6.12.0-rt #1 SMP PREEMPT_RT ... x86_64 GNU/Linux

Second, that the kernel exposes the real-time flag in sysfs:

raghu@techveda.org:~$ cat /sys/kernel/realtime
1

A value of 1 confirms that a PREEMPT_RT kernel is running. If the file does not exist, or uname does not show PREEMPT_RT, you have booted the wrong kernel; reboot and check the GRUB selection.

Re-run cyclictest and compare latency

Now run the same command from Part 1, with no changes, so the comparison is fair:

raghu@techveda.org:~$ sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0

As before, let it run for at least several minutes, and run it under realistic load rather than on an idle system. In a second terminal, generate load with stress-ng:

raghu@techveda.org:~$ stress-ng --cpu $(nproc) --io 4 --vm 2 --vm-bytes 256M --timeout 600s

The difference you are looking for is in the maximum latency, not the average. The following numbers are illustrative only; absolute values depend heavily on your hardware, firmware, and load. A typical pattern under load looks like this:

Standard kernel (Part 1 baseline, under load):
T: 0 (1021)  P:80  I:200  C: 920431  Min:  2  Avg:  9  Max:  3187

PREEMPT_RT kernel (this part, same load):
T: 0 (1044)  P:80  I:200  C: 921550  Min:  2  Avg:  3  Max:  72

The averages may be similar, but the maximum on the standard kernel can reach thousands of microseconds under load, while the PREEMPT_RT kernel keeps the maximum low and, more importantly, consistent. That bounded worst case is the property real-time systems depend on. Do not treat any single run as final: worst-case events are rare, so run for longer when you need confidence. For reference values on a wide range of hardware, the Open Source Automation Development Lab (OSADL) publishes continuous cyclictest latency plots.

One configuration detail affects measurement. The kernel reserves a slice of each second for non real-time tasks through /proc/sys/kernel/sched_rt_runtime_us, which defaults to 950000 microseconds out of a 1000000 microsecond period. This protects the system from a runaway real-time task. For controlled latency measurement you can read its current value, and only change it if you understand the risk of starving other tasks:

raghu@techveda.org:~$ cat /proc/sys/kernel/sched_rt_runtime_us
950000

Key takeaways

• The core PREEMPT_RT support is in the mainline kernel from version 6.12, so on x86_64, arm64, and RISC-V you can build a real-time kernel without an external patch.
• On Ubuntu, the build stops with a “No rule to make target ‘debian/canonical-certs.pem'” error; clear CONFIG_SYSTEM_TRUSTED_KEYS and CONFIG_SYSTEM_REVOCATION_KEYS to an empty string before building.
• The decisive setting is CONFIG_PREEMPT_RT (“Fully Preemptible Kernel (Real-Time)”) under General setup, Preemption Model; enable CONFIG_EXPERT if it is hidden.
• Disable high-impact debug options such as DEBUG_PREEMPT and SLUB_DEBUG for a real-time kernel.
• After booting, confirm the kernel with uname -a (the string PREEMPT_RT) and cat /sys/kernel/realtime (value 1).
• Compare the maximum latency from the same cyclictest command under load; the gain from PREEMPT_RT is a lower and more consistent worst case, not a lower average.

What’s next in this series

In Part 3 we will move from building the kernel to tuning the system: real-time priorities, the SCHED_FIFO scheduling class, and isolating CPU cores so a critical task runs with as little interference as possible.

Further reading

• Getting Started with PREEMPT_RT Guide — Real-Time Linux (Linux Foundation)
• Building and installing the Linux kernel — The Linux Kernel documentation
• cyclictest — Linux Foundation real-time documentation
• OSADL real-time QA farm latency plots