Skip to main content

TECH VEDA

Corporate on-site training - Submit proposal Pick your modulesSharpen your kernel skills: deep dives, drivers, Yocto, CVEs, careers — updated daily. Read the blog →
Kernel Features

The Ring Buffer Overwrites Early Boot Events: Can a Trace Event Filter Start Before Userspace?

In Linux 7.4-rc1 and later, trace_event= accepts a :filter: keyword. How the kernel applies a trace event filter at boot, and its limits.

The Ring Buffer Overwrites Early Boot Events: Can a Trace Event Filter Start Before Userspace?

In Linux 7.4-rc1 and later, the trace_event= boot parameter has a new :filter: keyword. A trace event filter can now be written on the kernel command line. The ring buffer then records only the matching events during boot. Linux 7.4 is not released yet. No stable or longterm kernel has the keyword, so teams that use a vendor BSP based on a longterm kernel cannot use it. The keyword is useful to engineers who trace initcalls, interrupts or scheduling during boot on a board with a small ring buffer.

Engineers who debug the boot of an embedded board often enable trace events with the trace_event= boot parameter. Before 7.4, an event enabled with this parameter had no trace event filter. The kernel recorded every occurrence of the event. This article explains the new :filter: keyword, how the kernel parses and applies it, and when the filter becomes active during boot.

What people do today without it

The kernel writes one record into the ring buffer each time an enabled trace event occurs. During boot, events such as sched:sched_switch or irq:irq_handler_entry occur many thousands of times. The tracing documentation states the default behaviour when the buffer is full: the oldest events are discarded and overwritten. The oldest events are the early boot events, which the engineer needs.

The first method is to make the buffer larger with trace_buf_size=, which uses more memory on every CPU. The second method is to write the trace event filter from userspace, in the filter file of the event. That filter does not exist while the kernel boots.

The third method is boot-time tracing through bootconfig. A bootconfig file can set a filter for an event:

ftrace.event {
        task.task_newtask {
                filter = "pid < 128"
                enable
        }
}

The kernel must be built with CONFIG_BOOTTIME_TRACING, which selects CONFIG_BOOT_CONFIG. The bootconfig file must be attached to the initrd or built into the kernel image. The 5.10 and 5.15 series support only the initrd method. Our earlier article on bootconfig and boot parameters describes that work. The function trace_boot_init() is registered with core_initcall_sync(). The comment at that line states that tracing starts from the beginning of postcore_initcall.

What was added

The kernel parameter documentation at v7.4-rc2 now contains this text for trace_event=:

To apply a filter, use the :filter: keyword:

trace_event=...:filter:<filter>

The event tracing documentation contains the full form:

<system>:<event>:mod:<module>:filter:<filter>

The :filter: keyword must be at the end of the event specification. The commit that added the keyword contains this example. It needs a kernel built with CONFIG_FTRACE_SYSCALLS:

trace_event=syscalls:sys_enter_clock_nanosleep:filter:which_clock==0

The text after :filter: is an ordinary trace event filter, with the same syntax as the filter file in tracefs. The keyword has no separate configuration symbol. The code is in kernel/trace/trace_events.c, which is built when CONFIG_EVENT_TRACING is enabled.

How it works

The boot parameter handler setup_trace_event() does not parse the string. It appends the string to a buffer of COMMAND_LINE_SIZE bytes. Later, __early_set_events() splits that buffer at each comma and passes each event specification to ftrace_set_clr_event(). This function now has a fourth argument, handle_filter. The parser ftrace_parse_event_string() searches for the text :filter: only when that argument is true:

/* event filters can be appended with :filter:<filter> */
if (handle_filter) {
        *filter = strstr(buf, ":filter:");
        if (*filter) {
                **filter = '\0';
                *filter += 8;
        }
}

Only the boot path passes true. The set_event file in tracefs and the bootconfig event list pass false. The cover letter of version 2 lists this limit as a change from version 1.

The parser searches for :filter: before it searches for :mod:. Everything after the first :filter: becomes the filter text. If :mod: is written after :filter:, it becomes part of the filter text and the filter is invalid.

For each event that matches, __ftrace_set_clr_event_nolock() first calls apply_event_filter(). It enables the event only if the filter was accepted:

if (filter)
        ret = apply_event_filter(file, filter);

if (!filter || !ret)
        ret = ftrace_event_enable_disable(file, set);

An event with a rejected trace event filter stays disabled, so it is not enabled without a filter. When the function returns an error, the boot path prints Failed to enable trace event: followed by the event specification.

With :mod:, the event can belong to a module that is not loaded yet. In that case, the kernel stores the filter text with the cached module request and applies both when the module loads. The filter is not checked at boot in this case. If it is rejected when the module loads, the event stays disabled and the kernel prints no failure message. The error text is then in the filter file of the event.

When the trace event filter becomes active

The kernel processes the trace_event= list twice. The first pass is in trace_event_init(), which start_kernel() calls through trace_init(), before any initcall runs. The second pass is in event_trace_enable_again(), an early_initcall. A comment in the source explains the reason. System call events cannot be enabled on the first pass, because it runs before process 1 starts.

The order of calls inside trace_event_init() affects filters:

event_trace_memsetup();
init_ftrace_syscalls();
event_trace_enable();
event_trace_init_fields();

event_trace_enable() defines the fields of each event and then applies the filters. event_trace_init_fields() runs after it and defines the fields that are shared by all events. These include comm, cpu and common_pid.

From this order, the following should happen. A filter that uses only the fields of the event itself is accepted on the first pass and is active before the first initcall. A filter that uses a shared field is rejected on the first pass. The kernel prints the failure message and the event stays disabled. On the second pass, the shared fields exist, so the filter is accepted and the event is enabled at early_initcall time. One exception follows from the search order. The kernel searches the fields of the event itself first. An event field with the same name as a shared field is therefore used on the first pass.

This description is based on reading the source at v7.4-rc2. We did not confirm it on a board. For driver probing the difference is small, because both passes run before the core and device initcalls.

Try it

First, check that the kernel source has the keyword. Run this command in the kernel source directory. In the v7.4-rc2 source the count is 2. In v7.3.4 it is 0:

raghu@techveda.org:~$ grep -c ':filter:' kernel/trace/trace_events.c
2

On a running kernel, check that the version is 7.4-rc1 or later and that event tracing is built in. CONFIG_EVENT_TRACING has no menu entry. The tracer options select it through CONFIG_TRACING. The second command needs /proc/config.gz:

root@beagleplay:~# uname -r
root@beagleplay:~# zcat /proc/config.gz | grep EVENT_TRACING

As an example, record only the initcalls that return a non-zero value. The initcall_finish event has two fields, func and ret. Add this to the kernel command line in the bootloader:

trace_event=initcall:initcall_finish:filter:ret!=0

This trace event filter uses a field of the event itself, so it should be active before the first initcall. After boot, read the result on the board. If tracefs is not mounted, run mount -t tracefs nodev /sys/kernel/tracing first:

root@beagleplay:~# cat /sys/kernel/tracing/events/initcall/initcall_finish/enable
root@beagleplay:~# cat /sys/kernel/tracing/events/initcall/initcall_finish/filter
root@beagleplay:~# cat /sys/kernel/tracing/trace
root@beagleplay:~# dmesg | grep 'Failed to enable trace event'

The enable file contains 1 if the event is enabled. The filter file contains the active filter, or the error text if the kernel rejected the filter. The trace file contains the recorded events. A non-zero return value is not always a fault, so read the trace as a list of initcalls to review.

The last command should print nothing for this example. For a filter on a shared field, it should print one failure message from the first pass, even when the second pass accepts the filter.

If the board stops before userspace, add tp_printk to the kernel command line. The kernel then also sends the events to printk. That code checks the filter, so only the matching events are printed.

The example is written without spaces. The kernel command line is split at spaces, so a filter with spaces needs double quotes around the whole parameter value. Write a string value inside the filter with single quotes, for example comm=='init'.

Which kernels have it

The commit that added the keyword is 084100a1b579, “tracing: Parse filter from boot event string”. It was merged into mainline with the tracing pull request for 7.4, and the first tag that contains it is v7.4-rc1. On 15 November 2026, kernel.org lists 7.4-rc2 as mainline.

We compared kernel/trace/trace_events.c at these stable and longterm tags: v7.3.4, v6.18.63, v6.12.117, v6.6.162, v6.1.192, v5.15.225 and v5.10.274. None of them contains the keyword. A team that uses a vendor BSP based on any of these series should not add :filter: to the kernel command line. There, the text is read as part of an event name, and the kernel prints the failure message. Use the bootconfig method to set a trace event filter at boot on those kernels.

Limits

  • The event list is split at commas before the filter is read. A filter that contains a comma, for example CPUS{0,2}, is split at the comma.
  • One filter text applies to every event that the event specification matches. With a wildcard, the events that have the named field are enabled. An event without the field stays disabled, and the kernel prints the failure message for the whole event specification.
  • The whole trace_event= list must fit in COMMAND_LINE_SIZE. The size is 2048 bytes on arm64 and RISC-V and 1024 bytes on 32-bit Arm.
  • The code is not specific to an architecture and does not depend on PREEMPT_RT.
  • The trace_instance= parameter enables its events through the same boot path. Its documentation does not mention filters, and we did not test that combination.

What was checked and what could not be verified

We read the series, its cover letter, and the source and documentation at v7.4-rc2.

We did not boot a 7.4-rc kernel with the keyword. For that reason, only the grep command has captured output. We ran it on the downloaded source files. We did not read the review of version 1, and the cover letter contains no reason for the command line limit.

Key takeaways

  • trace_event=<system>:<event>:filter:<filter> sets a trace event filter from the kernel command line in Linux 7.4-rc1 and later.
  • The kernel enables the event only if it accepts the filter. Otherwise, the event stays disabled.
  • From the source, a filter on event fields should be active before the first initcall. A filter on a shared field such as comm should become active at early_initcall time.
  • The filter cannot contain a comma, and no stable or longterm kernel has the keyword.
Was this worth your time?

Frequently asked questions

Which kernel version supports a trace event filter on the kernel command line?
The :filter: keyword is in Linux 7.4-rc1 and later. The stable 7.3 series and the longterm series from 5.10 to 6.18 do not have it.

What happens if the trace event filter is invalid?
The kernel does not enable the event. It prints Failed to enable trace event: followed by the event specification. For a :mod: entry whose module loads later, no failure message is printed.

How can I see the filtered events if the board stops before userspace?
Add tp_printk to the kernel command line. The kernel then also sends the matching events to printk.

References

RB
Raghu Bharadwaj

Founder, TECH VEDA — 20+ years teaching the Linux kernel, device drivers and embedded systems.

Follow on LinkedIn

Get new posts by email

Kernel, embedded Linux and AI-era engineering — a few sharp reads a month. No spam.

We email occasionally and never share your address.