Real-Time Linux with PREEMPT_RT — Part 4: Writing RT-Safe User-Space Code

In Part 1 of this series we measured latency with cyclictest, in Part 2 we built and booted a PREEMPT_RT kernel and compared the numbers, and in Part 3 we assigned SCHED_FIFO priorities and isolated CPUs. All of that work bounds the latency the kernel adds. It does nothing about the latency your own program adds. This part is about writing RT-safe user-space code: the specific things a periodic real-time thread must do at startup, and the specific things it must never do inside its loop.

What you need

• A board or virtual machine running the PREEMPT_RT kernel built in Part 2 (PREEMPT_RT has been in mainline since Linux 6.12, so a stock distribution RT kernel is fine too).
• GCC and the standard glibc headers on the target, or a cross toolchain.
• Root, or a user with the CAP_IPC_LOCK capability and an rtprio limit high enough to run at SCHED_FIFO priority 80. Both can be granted in /etc/security/limits.conf.

The three things that break a real-time thread in user space

A thread that is scheduled correctly can still miss its deadline for reasons that have nothing to do with the scheduler:

1. Page faults. The first write to a freshly allocated page, or the first execution of a code page that was never touched, traps into the kernel and can take hundreds of microseconds. Memory pages can also be dropped under memory pressure even on a system with no swap, because read-only pages such as program text can always be re-read from disk.
2. Dynamic memory allocation. malloc() and free() may call into the kernel (brk, mmap, munmap) and take a lock inside the allocator. Neither has a bounded execution time.
3. Priority inversion. A high-priority thread blocks on a mutex held by a low-priority thread, and the low-priority thread is itself preempted by a medium-priority thread that does not care about the mutex at all. The high-priority thread now waits for the medium-priority thread. With SCHED_FIFO this can last indefinitely.

Lock memory before the real-time loop starts

mlockall(MCL_CURRENT | MCL_FUTURE) pins the entire virtual address space of the process — code, globals, heap and every thread stack — into physical memory, and keeps future mappings pinned as well. The Linux Foundation real-time documentation is explicit that the locking call itself triggers the page faults needed to bring those pages in, so a separate stack pre-fault routine is not required as long as the RT threads are created after the mlockall() call.

if (mlockall(MCL_CURRENT | MCL_FUTURE) == -1) {
        perror("mlockall");
        return 1;
}

Two consequences follow from that. First, every RT thread must be created at startup, before the periodic loop begins, so that its stack is faulted in while the process is still an ordinary non-RT process. Second, the stack size should be set explicitly with pthread_attr_setstacksize(). The default is 8 MB per thread on glibc, and locking many 8 MB stacks wastes physical memory on a small board.

Heap memory should be allocated once, before the loop, and then reused. Two glibc tuning calls make the allocator behave predictably: mallopt(M_TRIM_THRESHOLD, -1) stops glibc from returning freed heap memory to the kernel, and mallopt(M_MMAP_MAX, 0) stops it from servicing large allocations with mmap(), which would be unmapped again on free() and would fault on the next use.

Set the scheduling policy on the thread attributes

A common mistake is to create the thread and then raise its priority from inside the thread. The thread then runs at the default policy for a short window, which is a source of jitter at startup. Set the policy and priority on the pthread_attr_t before pthread_create(), and — this is the part people forget — call pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED). The default is PTHREAD_INHERIT_SCHED, which makes the new thread inherit the creator’s scheduling parameters and silently ignore the policy and priority you just set.

Priority inheritance with PTHREAD_PRIO_INHERIT

POSIX defines three mutex protocols: PTHREAD_PRIO_NONE (the default), PTHREAD_PRIO_INHERIT and PTHREAD_PRIO_PROTECT. With PTHREAD_PRIO_INHERIT, when a thread blocks on a mutex that another thread owns, the owner runs at the higher of its own priority and the priority of the highest-priority waiter, for as long as it holds the mutex. That bounds the inversion to the length of the critical section. The effect also propagates recursively if the owner is itself blocked on another PI mutex.

The protocol is a property of the mutex attributes object, so it must be set before the mutex is initialised:

pthread_mutexattr_t mattr;

pthread_mutexattr_init(&mattr);
pthread_mutexattr_setprotocol(&mattr, PTHREAD_PRIO_INHERIT);
pthread_mutex_init(&data_lock, &mattr);

Note that a mutex created with the static initialiser PTHREAD_MUTEX_INITIALIZER uses PTHREAD_PRIO_NONE and gives you no protection at all. On Linux, a PI mutex is backed by the priority-inheritance futex operations in the kernel, which is why the protection works across the whole system and not only inside glibc.

The complete RT-safe user-space code

The periodic loop below follows the structure used in the Linux Foundation cyclic-task documentation: compute the absolute time of the next period, do the work, then sleep until that absolute time with clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, ...). Using an absolute deadline on a monotonic clock is what stops the period from drifting, and CLOCK_MONOTONIC cannot be stepped backwards by an administrator or by NTP, unlike CLOCK_REALTIME, which plain nanosleep() uses.

/* rt_task.c — periodic SCHED_FIFO thread, RT-safe startup */
#define _GNU_SOURCE
#include <pthread.h>
#include <sched.h>
#include <stdio.h>
#include <string.h>
#include <limits.h>
#include <malloc.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <time.h>

#define PERIOD_NS       1000000L        /* 1 ms */
#define RT_PRIORITY     80
#define ITERATIONS      10000

struct period_info {
        struct timespec next_period;
        long period_ns;
};

static pthread_mutex_t data_lock;
static long shared_value;

static void inc_period(struct period_info *pinfo)
{
        pinfo->next_period.tv_nsec += pinfo->period_ns;
        while (pinfo->next_period.tv_nsec >= 1000000000L) {
                pinfo->next_period.tv_nsec -= 1000000000L;
                pinfo->next_period.tv_sec++;
        }
}

static void periodic_task_init(struct period_info *pinfo)
{
        pinfo->period_ns = PERIOD_NS;
        clock_gettime(CLOCK_MONOTONIC, &pinfo->next_period);
}

static void do_rt_task(void)
{
        /* Touch only memory that is already allocated and locked. */
        pthread_mutex_lock(&data_lock);
        shared_value++;
        pthread_mutex_unlock(&data_lock);
}

static void wait_rest_of_period(struct period_info *pinfo)
{
        inc_period(pinfo);
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME,
                        &pinfo->next_period, NULL);
}

static void *rt_thread(void *arg)
{
        struct period_info pinfo;
        struct rusage before, after;
        int i;

        (void)arg;
        getrusage(RUSAGE_SELF, &before);
        periodic_task_init(&pinfo);

        for (i = 0; i < ITERATIONS; i++) {
                do_rt_task();
                wait_rest_of_period(&pinfo);
        }

        getrusage(RUSAGE_SELF, &after);
        /* printf() outside the loop only, never inside it. */
        printf("minor faults during loop: %ld\n",
               after.ru_minflt - before.ru_minflt);
        printf("major faults during loop: %ld\n",
               after.ru_majflt - before.ru_majflt);
        printf("iterations completed:     %d\n", i);
        return NULL;
}

int main(void)
{
        pthread_t thread;
        pthread_attr_t attr;
        pthread_mutexattr_t mattr;
        struct sched_param param;
        int ret;

        /* 1. Lock every page, current and future, before anything else. */
        if (mlockall(MCL_CURRENT | MCL_FUTURE) == -1) {
                perror("mlockall");
                return 1;
        }

        /* 2. Keep glibc from giving heap memory back to the kernel. */
        mallopt(M_TRIM_THRESHOLD, -1);
        mallopt(M_MMAP_MAX, 0);

        /* 3. A mutex that inherits the waiter's priority. */
        pthread_mutexattr_init(&mattr);
        pthread_mutexattr_setprotocol(&mattr, PTHREAD_PRIO_INHERIT);
        pthread_mutex_init(&data_lock, &mattr);

        /* 4. SCHED_FIFO, explicit, with a bounded stack. */
        pthread_attr_init(&attr);
        pthread_attr_setstacksize(&attr, PTHREAD_STACK_MIN + 64 * 1024);
        pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
        param.sched_priority = RT_PRIORITY;
        pthread_attr_setschedparam(&attr, &param);
        pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);

        ret = pthread_create(&thread, &attr, rt_thread, NULL);
        if (ret) {
                fprintf(stderr, "pthread_create: %s\n", strerror(ret));
                return 1;
        }

        pthread_join(thread, NULL);
        pthread_attr_destroy(&attr);
        pthread_mutexattr_destroy(&mattr);
        pthread_mutex_destroy(&data_lock);
        return 0;
}

Build it, run it, and check that it is really RT-safe

raghu@techveda.org:~$ gcc -O2 -Wall -o rt_task rt_task.c
raghu@techveda.org:~$ sudo ./rt_task
minor faults during loop: 0
major faults during loop: 0
iterations completed:     10000

Zero minor and zero major faults inside the loop is the result you want. getrusage() counts faults for the whole process, so any non-zero number here means something in the loop touched memory that was not locked, or allocated memory. Remove the mlockall() call, rebuild, and the minor fault count becomes non-zero — that difference is the entire point of the call.

Confirm the thread’s policy and priority while it runs. The main thread stays SCHED_OTHER (shown as TS); only the worker thread is FF at priority 80:

raghu@techveda.org:~$ ps -eLo pid,tid,class,rtprio,comm | grep rt_task
   2914   2914 TS       -   rt_task
   2914   2915 FF      80   rt_task

If the second line shows TS and no priority, pthread_attr_setinheritsched() was not called with PTHREAD_EXPLICIT_SCHED. Check the locked memory as well:

raghu@techveda.org:~$ grep VmLck /proc/2914/status
VmLck:      3204 kB

What must stay out of the periodic loop

• malloc(), free(), new, delete, and any container that resizes itself.
• printf() and file I/O. Both take locks and can block on the device. Log into a pre-allocated ring buffer and let a low-priority thread print it.
• Ordinary mutexes, and any library that hides its own locking. If an RT thread must share data with a non-RT thread, the mutex has to be a PI mutex.
• Any first-touch of memory: a freshly mmaped buffer, a newly loaded shared library, a lazily resolved PLT entry. Linking with -Wl,-z,now resolves symbols at load time and removes the last one.

One more system-level point that catches people after the code is correct: the kernel’s real-time throttling still applies. /proc/sys/kernel/sched_rt_runtime_us defaults to 950000 against a period of 1000000, so RT tasks are allowed at most 95% of each second per CPU. A runaway SCHED_FIFO loop is throttled rather than locking up the machine, which is helpful during development and worth knowing about before you blame the scheduler.

This is the level of detail we work through with real hardware in the Linux systems engineering track at TECH VEDA, where the application side and the kernel side are debugged together rather than separately.

Key takeaways

• An RT kernel bounds kernel latency. Page faults, allocation and priority inversion in your own code are yours to remove.
• Call mlockall(MCL_CURRENT | MCL_FUTURE) first, then create every RT thread. The lock call faults the pages in, so no separate pre-fault loop is needed.
• Set SCHED_FIFO policy, priority and stack size on the thread attributes, and set PTHREAD_EXPLICIT_SCHED, or the settings are ignored.
• Shared mutexes must use PTHREAD_PRIO_INHERIT. The static initialiser does not.
• Time the loop with clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, ...) against an absolute deadline so the period does not drift.
• Verify, do not assume: zero ru_minflt growth in the loop, and FF with the expected priority in ps -eLo class,rtprio.

What’s next in this series

Part 5 is the last one: tracing the source of a latency spike with ftrace, using the wakeup_rt and irqsoff tracers to find out which piece of code delayed the real-time thread.

Frequently asked questions

Do I still need to pre-fault the stack if I call mlockall()?
No. The Linux Foundation real-time documentation states that the memory-locking call itself triggers the page faults required to bring the pages into physical memory, so threads created after mlockall(MCL_CURRENT | MCL_FUTURE) have their stacks already resident. Create all RT threads at startup, after the lock call.

Why does my thread run at the default priority even though I set sched_priority?
Because pthread_attr_setinheritsched() was not set to PTHREAD_EXPLICIT_SCHED. The default is PTHREAD_INHERIT_SCHED, which makes the new thread take the creator’s policy and priority and discard the ones you configured in the attributes object.

Is a normal pthread mutex safe inside a real-time thread?
Not if a lower-priority thread can hold it. Without PTHREAD_PRIO_INHERIT, the owner keeps its own low priority and can be preempted by unrelated medium-priority threads while the RT thread waits, which is unbounded priority inversion.

Why clock_nanosleep instead of nanosleep or usleep?
nanosleep() sleeps for a relative interval on CLOCK_REALTIME, which can be stepped by an administrator or by time synchronisation. clock_nanosleep() with CLOCK_MONOTONIC and TIMER_ABSTIME sleeps until an absolute point on a clock that never jumps, so the period does not drift and cannot be disturbed by a clock change.

Further reading

• Memory for Real-time Applications — Linux Foundation real-time wiki
• HOWTO build a basic cyclic application — Linux Foundation real-time wiki
• pthread_mutexattr_setprotocol(3p) — man7.org
• clock_nanosleep(2) — man7.org
• mlock(2) / mlockall(2) — man7.org
• sched(7) — scheduling policies and RT throttling
• Understanding Linux real-time with PREEMPT_RT — Bootlin training slides