Embedded Linux Observability: What to Instrument on a Device You Cannot Reach

Devices fail in the field in ways a lab cannot reproduce: a scheduling stall that only appears when a radio wakes, a memory pressure event that needs twenty-six hours of uptime, an interrupt storm that correlates with a customer site’s temperature. When those devices are ten thousand units in ten thousand homes, the log file on any one of them is not an observability strategy. It is at most an evidence source.

The gap is not that logging is bad. It is that the interesting failures happen inside the kernel or between the kernel and the hardware, and a log statement placed at build time cannot reach there. That is the gap eBPF was built to close. When a customer reports that the device stutters for two seconds every few minutes, the logs almost never contain the stutter. They contain what happened on either side of it.

Why classic logging cannot answer the question

A shipped device usually has three logging surfaces: the kernel ring buffer, a journal or equivalent in user space, and one or more application logs. All three share the same limits. They capture what a developer thought to record at build time, they cost flash writes and RAM continuously, and they do not answer scheduling or interrupt-level questions at all.

That is the case for eBPF: instrumentation you can attach to a running kernel, change without a rebuild, and remove when you are finished. It is also the case for treating the output as data rather than as English sentences, and for pointing the instrumentation at power and scheduling behaviour, where the expensive regressions hide.

Layer 1: eBPF, and whether your kernel can run it

eBPF runs sandboxed programs inside the kernel without loading a module or patching the source. For a fleet, three properties matter more than the feature list.

Programs are verified before they run. The verifier rejects anything it cannot prove terminates and stays in bounds, so a malformed instrumentation script cannot brick a device. The budget is generous but finite: BPF_COMPLEXITY_LIMIT_INSNS in include/linux/bpf.h is one million processed instructions, and a program may use 512 bytes of stack, MAX_BPF_STACK in include/linux/filter.h. Privilege changes the ceiling sharply. In kernel/bpf/syscall.c the load path compares the instruction count against BPF_COMPLEXITY_LIMIT_INSNS when the caller is capable and against BPF_MAXINSNS, which is 4096, when it is not.

Programs are compiled to native code. That is what makes tracing affordable on a device with no headroom, and it is where the first surprise lives.

Programs exchange data with user space through maps rather than by calling arbitrary kernel functions, which gives a stable contract that survives kernel upgrades better than a hand-written module does.

The JIT default is not what the documentation says

Documentation/admin-guide/sysctl/net.rst lists the values for bpf_jit_enable and states that 0, disable the JIT, is the default. On arm64 that is not true, and the trace through the source shows why.

kernel/bpf/core.c initialises the variable from a config symbol rather than from a constant:

int bpf_jit_enable __read_mostly = IS_BUILTIN(CONFIG_BPF_JIT_DEFAULT_ON);

and kernel/bpf/Kconfig derives that symbol from an architecture opt-in:

config BPF_JIT_DEFAULT_ON
        def_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON
        depends on HAVE_EBPF_JIT && BPF_JIT

arm64 selects ARCH_WANT_DEFAULT_BPF_JIT; 32-bit ARM does not. So on an arm64 gateway the JIT is on from boot, and on a 32-bit board it is off until something writes to the sysctl, with the interpreter carrying the cost in the meantime. The documentation’s “default value” is the generic case, not your case.

There is a second architecture gate. arch/arm64/Kconfig selects HAVE_EBPF_JIT unconditionally, but arch/arm/Kconfig selects it only if !CPU_ENDIAN_BE32. A big-endian 32-bit ARM product has no eBPF JIT available at all.

The check to run before you plan anything

Whether any of this is available is decided by the kernel configuration, and CONFIG_BPF_SYSCALL is default n in kernel/bpf/Kconfig. On a vendor BSP it is often unset. Check the board before designing around it:

root@beaglebone:~# zcat /proc/config.gz | grep -E "BPF_SYSCALL|BPF_JIT|BPF_EVENTS|DEBUG_INFO_BTF"
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_EVENTS=y
# CONFIG_DEBUG_INFO_BTF is not set
root@beaglebone:~# cat /proc/sys/net/core/bpf_jit_enable
0

CONFIG_BPF_EVENTS is the one that decides whether you can attach eBPF programs to kprobes, uprobes and tracepoints, which is most of what observability means. It is not directly selectable. In kernel/trace/Kconfig it depends on BPF_SYSCALL and on (KPROBE_EVENTS || UPROBE_EVENTS) && PERF_EVENTS. Either probe type satisfies it, but PERF_EVENTS is not optional.

The missing DEBUG_INFO_BTF in that output is the line that will cost you a week. BTF is the type information the kernel exposes about itself, and libbpf relies on the running kernel’s BTF to relocate a program’s field offsets so one binary works across kernel versions. The kernel publishes it at /sys/kernel/btf/vmlinux. Without it, portable programs built the modern way will not load, and you are back to compiling against each kernel you ship. In lib/Kconfig.debug the option depends on BPF_SYSCALL and on PAHOLE_VERSION >= 122, which the help text states as pahole v1.22 or later. It also depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED, and that second dependency is the one that catches people: a BSP that sets DEBUG_INFO_REDUCED to keep the build output small makes BTF unavailable, and nothing in the resulting configuration explains why. It is not free — generating and carrying that type information adds to the image — but the kernel documentation does not quantify the cost, so measure it on your own build rather than trusting a number from anywhere.

What to instrument, in priority order

1. Scheduling latency: wakeup-to-run delay for the critical thread, run queue length per CPU, and preemption disabled longer than a threshold.
2. Block and I/O latency from submission to completion, split by device, so a slow eMMC becomes a distribution rather than an anecdote.
3. Interrupt storms and softirq time, split by IRQ line, so one misbehaving peripheral driver is visible.
4. Memory pressure events and OOM kills, including which process was killed.
5. Power transitions: idle state residency and frequency residency, because a device that never sleeps deeply enough fails thermal and battery goals silently.

Layer 2: structured events, and why the ring buffer matters

The output of instrumentation should not be human-readable strings. It should be structured events with a schema version, a timestamp, a device identifier, an event type and typed fields. Every free-form string is a parser regression waiting to happen, every timestamp format inconsistency is a query bug, and human phrasing locks the pipeline to one product team’s vocabulary. On a constrained link, structured binary events also compress far better than log lines, and a schema break becomes a compile-time failure instead of a silent parsing regression on the backend.

The transport inside the kernel matters more than it looks on a small device. The older perf buffer allocates per CPU, and the BPF ring buffer was introduced to fix two specific consequences of that. The kernel documentation gives the motivation directly: more efficient memory utilisation by sharing the ring buffer across CPUs, and preserving the ordering of events that happen sequentially in time even across multiple CPUs. On a four-core board with a tight memory budget, one shared buffer instead of four is the difference between affordable and not.

There is a second benefit that matters for program size. bpf_ringbuf_reserve() hands the program a pointer directly into the buffer, which removes the common pattern of staging a sample in a per-CPU array because it is larger than the 512 bytes of stack a program is allowed. Fewer maps, less copying, smaller programs.

Layer 3: power and scheduling, where the expensive regressions hide

For a battery or thermally constrained device the most consequential misbehaviour is usually not a functional bug. It is a power state regression, and it produces no error anywhere.

The CPUIdle subsystem characterises each idle state by two parameters. The target residency is the minimum time the hardware must spend in the state, including the time needed to enter it, in order to save more energy than a shallower state would. The exit latency is the maximum time from a wakeup request to the first executed instruction. When a device stops entering its deeper states, battery life and thermal headroom both regress with nothing in the logs.

The mechanism that causes it is usually PM QoS. The documentation is explicit that idle governors are expected to regard the minimum of the global effective CPU latency limit and the effective resume latency constraint for that CPU as the upper limit for the exit latency of any state they may select. A driver or a user-space process that asserts a tight latency constraint and never releases it therefore holds the entire system out of deep idle. Instrumenting those assertions is one of the highest-leverage things you can do on a battery-powered product, because a single stuck constraint from an unrelated driver can change idle power draw with no functional symptom at all.

The blunt version of the same failure is booting with idle=poll. The kernel documentation calls this somewhat drastic, and notes that preventing idle CPUs from saving almost any energy may not be its only effect: on Intel hardware it also prevents CPUs from using P-states that require CPUs in a package to be idle, so it can hurt single-thread performance as well as efficiency. That second consequence is specific to Intel and does not transfer to an ARM SoC, but the first one does. Whether a shipped device actually reaches its deepest advertised idle state is worth checking automatically rather than assuming.

The real-time throttle nobody instruments

If any part of the product uses SCHED_FIFO or SCHED_RR, the kernel caps how much CPU those threads may take. The defaults are a period of 1,000,000 microseconds and a runtime of 950,000, which leaves 0.05 seconds per second to SCHED_OTHER. The documentation states the reasoning plainly: the defaults were chosen so that a run-away real-time task will not lock up the machine but leaves a little time to recover it.

The consequence is that a real-time thread consuming more CPU than intended is not reported as a scheduling problem. It is throttled, and the user sees a periodic hang. Instrumenting throttle events turns a vague support description into a categorised, actionable failure. If you are writing the user-space side of such a thread, the constraints that keep it out of trouble are covered in Real-Time Linux with PREEMPT_RT — Part 4: Writing RT-Safe User-Space Code.

What to build first

1. Audit the kernel configuration on every product line before designing anything. If CONFIG_BPF_SYSCALL or CONFIG_BPF_EVENTS is unset, or DEBUG_INFO_BTF is missing, that is the first work item and it is a BSP change, not an application one.
2. Ship an instrumentation agent in the firmware image, disabled by default and gated behind signed remote enablement. Never expose program loading to the network.
3. Publish a versioned event schema and treat schema evolution as a breaking-change process, the same way you treat a protocol change.
4. Add idle-state residency and PM QoS constraint tracing before any application metric. Application metrics can wait; a battery regression that ships cannot be recalled.
5. For any product using SCHED_FIFO or SCHED_RR, instrument throttle events, blocked time and wakeup latency as first-class metrics.

What not to do

• Do not enable always-on tracing across a fleet. Even efficient instrumentation costs something, and it widens the surface for log exfiltration.
• Do not send free-form strings where a structured event will do.
• Do not confuse a heartbeat with health. A heartbeat says a network stack is running, not that the audio pipeline met its deadline.
• Do not assume the BPF JIT is enabled because the sysctl documentation says so. Read the value on the board.
• Do not ship without a signed remote-enable path for deeper diagnostics. If your first field regression needs an over-the-air update before you can even look, you have lost weeks.

Key takeaways

• Embedded Linux observability is three layers: eBPF instrumentation in the kernel, structured events, and power and scheduling diagnostics. Logs are an evidence source, not a strategy.
• Whether eBPF runs on your board is a kernel configuration question. CONFIG_BPF_SYSCALL is default n, and CONFIG_BPF_EVENTS is what allows attaching to kprobes, uprobes and tracepoints.
• The BPF JIT is on by default on arm64 and off by default on 32-bit ARM, because the default is derived from an architecture opt-in rather than from the constant the sysctl documentation quotes. Big-endian 32-bit ARM has no eBPF JIT at all.
• Missing kernel BTF is what usually stops portable programs from loading on a BSP kernel; the kernel exposes it at /sys/kernel/btf/vmlinux.
• The BPF ring buffer shares one buffer across CPUs and preserves event ordering, which matters more on a small device than on a server.
• A stuck PM QoS latency constraint holds the whole system out of deep idle with no functional error, and a real-time thread over its budget is throttled rather than reported.

Frequently asked questions

Do I need eBPF, or is ftrace enough?
ftrace is enough for a single device on a bench. eBPF matters for a fleet because programs are verified before they run, are compiled to native code, and return structured data through maps, so instrumentation can be enabled remotely on a shipped unit without a rebuild.

Why will my eBPF program not load on a vendor BSP kernel?
Most often the kernel lacks BTF type information, so libbpf cannot relocate field offsets for that kernel. Check whether /sys/kernel/btf/vmlinux exists and whether CONFIG_DEBUG_INFO_BTF was set in the build. If it was not, check the BSP for DEBUG_INFO_REDUCED, which blocks it. CONFIG_BPF_SYSCALL being unset produces the same outcome for a different reason.

Is the BPF JIT enabled by default?
It depends on the architecture. The default is derived from CONFIG_BPF_JIT_DEFAULT_ON, which follows an architecture opt-in that arm64 selects and 32-bit ARM does not. The sysctl documentation states 0 as the default, which is the generic case and not correct for arm64.

Why do my devices lose battery life with no error in the logs?
A latency constraint asserted through PM QoS and never released prevents idle governors from selecting deep idle states, because they must treat that constraint as the upper limit on exit latency. Nothing fails, so nothing is logged.

My real-time thread causes periodic hangs. Is that a scheduling bug?
It is more likely the real-time bandwidth cap. The defaults allow real-time tasks 950,000 microseconds out of every 1,000,000, leaving 0.05 seconds per second for everything else. A thread over that budget is throttled, which users experience as a hang.

Further reading

• libbpf overview — BPF CO-RE and kernel BTF
• BPF ring buffer — Linux kernel documentation
• sysctl/net — bpf_jit_enable, bpf_jit_harden, bpf_jit_limit
• kernel/bpf/Kconfig — BPF_SYSCALL, BPF_JIT and the JIT default
• kernel/bpf/core.c — where bpf_jit_enable is initialised
• lib/Kconfig.debug — DEBUG_INFO_BTF and its pahole and DEBUG_INFO_REDUCED dependencies
• kernel/trace/Kconfig — what BPF_EVENTS actually depends on
• CPU Idle Time Management — target residency, exit latency and PM QoS
• Real-Time group scheduling — the bandwidth defaults
• Documentation/ABI/testing/sysfs-kernel-btf