Embedded Linux Log Storage: RAM, Flash or Remote?

For most products, keep the full log in RAM, forward it to a server whenever the device is connected, keep kernel crash records in pstore, and write only a small, bounded set of application events to flash. Persisting the complete log to flash is the better choice when devices stay offline for long periods and the log is the only record of a failure, or when a local audit record is required. The main trade-off is that an ordinary power loss removes any log that was neither forwarded nor persisted.

An embedded Linux product needs an answer to one question: after a device fails in the field, what record of the failure exists? The answer depends on where the log was written. This article compares three places for embedded Linux log storage (RAM, local flash and a remote server) and recommends a default for products that ship in volume. The reasoning follows an architecture decision record: context, options, decision and consequences.

The context

A development board writes logs to the root filesystem and the write load goes unexamined. A shipped product has different constraints. Flash has a limited number of program and erase cycles, so a service that logs continuously reduces the usable life of the flash. Some products use a read-only root filesystem, so a writable location for logs must be designed explicitly. Power can be removed at any time, so data held only in memory or in a write cache may not survive. Finally, the device may have no network, a metered link, or a link that is unavailable when the failure occurs.

The decision is therefore not only “RAM or flash”. It is a choice about which records must survive a reboot, which must survive a power loss, and which must leave the device. Related decisions are covered in our posts on root filesystem formats and device heartbeat design.

The options

Option 1: Keep logs in RAM

With systemd, Storage=volatile in journald.conf keeps log data only in memory under /run/log/journal. The manual documents RuntimeMaxUse= as the limit for that location, with a default of 15% of the file system capped at 4G. On a small device that default may be larger than you intend, so set an explicit value. With BusyBox, syslogd -C[size_kb] logs to a shared-memory buffer that logread reads. The buffer is provided by FEATURE_IPC_SYSLOG (enabled by default) and sized by FEATURE_IPC_SYSLOG_BUFFER_SIZE, which the BusyBox source gives a default of 16 KB and a minimum of 4 KB.

Advantages: no flash wear, no dependency on a writable partition, and it works with a read-only root filesystem.
Disadvantages: everything is lost at reboot or power loss. The history before a failure is gone unless it was sent elsewhere.

RAM-only logging is sufficient when a failure can be reproduced or the loss of recent history is acceptable.

Option 2: Write logs to local flash

Storage=persistent stores the journal under /var/log/journal. The manual describes how journald starts with volatile storage and switches to persistent storage when journalctl --flush runs, which happens at boot through systemd-journal-flush.service. The size limit is SystemMaxUse=, with a default of 10% of the file system capped at 4G; only archived files are deleted to stay within it. Data is synced every five minutes by default (SyncIntervalSec=), and messages of level CRIT, ALERT and EMERG trigger an immediate sync, while ERR messages do not.

Advantages: logs survive reboots and are available after a long offline period. No server is needed.
Disadvantages: continuous writes consume flash endurance. The sync interval means recent entries may not be on flash at the moment of a power loss. A read-only root filesystem needs a separate writable partition for /var/log.

Check how your image maps /var/log. In Poky, /var/log is by default a symlink to volatile storage (/var/volatile/log), so on a stock image journald’s persistent storage may not reach flash. In current Yocto releases this is controlled by FILESYSTEM_PERMS_TABLES containing files/fs-perms-volatile-log.txt; older releases used VOLATILE_LOG_DIR. Removing that entry (or, on older releases, setting VOLATILE_LOG_DIR to “no”) makes /var/log a real directory. Verify the result on your own image.

Option 3: Send logs to a remote server

systemd provides systemd-journal-upload, which transfers journal content over HTTP (HTTPS by default, port 19532) to systemd-journal-remote. With --save-state it stores the last uploaded cursor, by default in /var/lib/systemd/journal-upload/state, so it can resume. BusyBox syslogd offers -R HOST[:PORT], with port 514 as the default, and -L to log locally as well as over the network (with -R alone, only the network is used).

Advantages: records survive device loss and can be searched across a fleet.
Disadvantages: it needs connectivity, server infrastructure and credentials. The upload state file also needs a small writable location.

Remote forwarding alone is the better choice when devices are connected almost continuously and flash endurance or a read-only root filesystem rules out local persistence.

A complement: crash records with pstore

The kernel can store oops and panic records separately from normal logs. The ramoops backend keeps them in a reserved RAM region and requires RAM that keeps its contents across a restart. It is configured by module parameters or by a reserved-memory device tree node with compatible = "ramoops", and records appear as dmesg-ramoops-N in pstore. The kernel documentation notes that RAM may be somewhat corrupt after a watchdog-triggered reset but is usually restorable, and that the optional ecc parameter (software ECC protection of the region) can be useful in that case. The pstore/blk backend writes to a block device or MTD partition instead.

raghu@techveda.org:~$ sudo mount -t pstore pstore /sys/fs/pstore
raghu@techveda.org:~$ ls /sys/fs/pstore
dmesg-ramoops-0

Many systems already mount pstore at this location. The directory is empty until a crash has been recorded.

The decision for embedded Linux log storage

For the common case, combine four mechanisms. Keep the full log in RAM with an explicit size limit. Forward it to a server whenever the network is available. Use pstore for kernel crash records. Persist a small application-level event record to flash, such as boot reasons, update results and fatal errors, with a fixed size limit and rate limiting. The application writes this record to a size-limited file on a writable partition; journald with Storage=volatile does not keep it. This approach to embedded Linux log storage limits flash writes, works with a read-only root filesystem provided a small writable partition holds the event record and the upload state, and still preserves records of the most serious failures.

# /etc/systemd/journald.conf.d/10-product.conf (example values)
[Journal]
Storage=volatile
RuntimeMaxUse=8M

Persistent journald storage is the right choice in three cases. The first is development and bring-up, where flash wear is not a concern. The second is a device that is offline for weeks and whose log is the only diagnostic record. The third is a product with a requirement to keep an audit record locally. In these cases, set SystemMaxUse= explicitly, place the journal on a writable partition sized for the expected write endurance, and decide how often data must be synced. The BusyBox in-memory buffer and logread remain a reasonable choice on devices too small for systemd; see systemd vs BusyBox init for that decision.

Consequences

Loss of history. A power loss removes the RAM log. Forwarding reduces this exposure only when the link is up, so the team must accept that some failures will have no log before the event.

Forwarding requires maintenance. You need a receiving service, certificates, retry behaviour and a policy for bandwidth. The upload state file needs writable storage.

Rate limits can drop messages. The journald manual documents a default limit of 10000 messages in 30 seconds per service. A service that logs heavily can lose messages, and the limits can be changed per service.

Sizing must be specified. Embedded Linux log storage has a cost in memory or flash. The RAM limit competes with application memory, and the flash event record needs a size and a write-endurance budget. Both should appear in the product requirements.

Hardware dependency. Crash records in ramoops depend on a board that preserves RAM across reset. This must be tested on the target, including after a watchdog reset.

Key takeaways

Treat embedded Linux log storage as a design decision: decide which records must survive a reboot, a power loss, or the loss of the device itself.
Default to a bounded RAM log, remote forwarding, pstore crash records and a small flash event record.
Persist the full journal only when devices are offline for long periods or an audit record is required.
Set RuntimeMaxUse= or SystemMaxUse= explicitly instead of relying on percentage defaults.
Test crash-record retention on the real board, including after a watchdog reset.

Our Embedded Linux and Yocto training covers root filesystem and storage design.

Was this worth your time?

Frequently asked questions

What does Storage=volatile do in journald?
It keeps log data only in memory under /run/log/journal, so the data is lost at reboot.

Does journald write every message to flash immediately when persistent storage is used?
No. By default it syncs every five minutes. Messages of level CRIT, ALERT and EMERG trigger an immediate sync, but ERR messages do not.

How can I keep kernel panic information when logs are in RAM?
Use pstore. The ramoops backend stores records in reserved RAM that must survive a restart, and the pstore/blk backend writes to a block device or MTD partition.

What can I use for logging on a device without systemd?
BusyBox syslogd can log to a shared-memory buffer with the -C option, which logread reads, or send logs to a remote host with -R.

Embedded Linux Log Storage: RAM, Flash or Remote?

The context