Read-Only Root Filesystem on Embedded Linux: Where Should Writable State Live?

A read-only root filesystem reduces the risk of corrupting the system image and keeps every unit identical, but the device still has to store a hostname, a machine ID, network settings, certificates, logs and application data somewhere. This article examines how to structure the writable side of a read-only root filesystem. It goes one level below our comparison of root filesystem formats, which recommends a read-only root with a separate writable partition but does not compare the ways of connecting the two.

The context: what a read-only root filesystem forces you to decide

With Yocto, the read-only-rootfs image feature builds an image in which, as the manual states, any attempt by the target to write to the root filesystem at runtime fails. Post-install scripts must therefore complete during the build, unless read-only-rootfs-delayed-postinsts is set, which assumes the root will be made writable for the first boot. In Buildroot, BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW controls whether the root is remounted read-write during boot; disabling it keeps the root read-only.

Some writable state is disposable, such as sockets and caches. Some must survive a reboot, such as device identity and configuration. Some must survive an update and remain readable after a rollback. This article assumes image-based updates; a product that installs packages on the device keeps a writable root and is outside this decision.

The options: three ways to provide writable state

Option 1: tmpfs for data that may be lost

Directories that hold only runtime data are backed by tmpfs. On Yocto with systemd, the volatile-binds recipe generates one service per entry in VOLATILE_BINDS. Each service makes a directory under the tmpfs /var/volatile appear at the read-only location, by default through an overlay. The default entries cover /var/lib, /var/cache, /var/spool and /srv.

• For: no flash writes, and a clean state after every boot.
• Against: all contents are lost at reboot, and the data occupies RAM.

Option 2: named persistent directories on a data partition

State that must persist is placed on a separate partition, and each directory is connected to the path software expects. The RAUC documentation lists three ways to map paths onto a data partition (symbolic links, bind mounts and an overlay file system) and says the choice depends on the amount and type of data. This option uses the first two. In /etc/fstab, the data partition, one bind mount and a tmpfs mount look like this:

/dev/mmcblk0p4       /data               ext4    defaults,noatime   0  2
/data/lib/myapp      /var/lib/myapp      none    bind               0  0
tmpfs                /var/volatile       tmpfs   defaults           0  0

Every mount point must already exist in the image, because it cannot be created on the read-only root at runtime.

• For: the persistent paths form a short, reviewable list; reset and migration act on known directories; nothing else persists by accident.
• Against: every writer must be found and redirected before release; software that writes to an unexpected path fails with a read-only error.

Option 3: overlayfs with a persistent upper layer

An overlayfs presents the read-only root as the lower layer and a directory on the data partition as the writable upper layer, in the form mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper,workdir=/work /merged. The workdir must be on the same filesystem as the upperdir. Yocto’s overlayfs-etc image feature provides this for /etc, using a custom init script that mounts /etc before the real init starts.

• For: unmodified software keeps working, because a write to any path is written to the upper layer.
• Against: a file kept in the upper layer takes precedence over the same file in a new image, so an update can fail to deliver changed defaults; the kernel restricts offline changes to the lower layer; early boot needs more setup; and unintended changes persist.

The decision: explicit state directories by default, overlayfs by exception

For a product that ships to customers, use Option 1 and Option 2 together. Build a read-only root filesystem, place disposable data on tmpfs, and place every persistent item in a named directory on the data partition. The result is a written inventory of device state.

Option 3 is the better choice for a large body of existing software that writes to many paths and cannot be changed in time, or for a device where users configure software themselves, such as a development platform. Apply the overlay to the narrowest tree that works, such as /etc alone. If persistence is not required, a tmpfs upper layer returns the system to the image state at every reboot.

Consequences for updates, identity and rollback

Persistent overlays and updates. An A/B update replaces the lower layer offline. The kernel documentation allows that only if the metacopy, index, xino and redirect_dir features have not been used, so mount the overlay with redirect_dir=off,index=off,metacopy=off,xino=off. With redirect_dir disabled, renaming a lower or merged directory fails with EXDEV, which some software does not handle.

Default volatile directories. On Yocto, the default VOLATILE_BINDS makes all of /var/lib volatile, which can hide a persistent bind mount below it. Remove /var/lib from VOLATILE_BINDS and bind each persistent subdirectory explicitly.

Device identity. If /etc/machine-id is empty on a read-only root, systemd bind-mounts a temporary file over it, so the ID changes between boots. Provision it, together with SSH host keys and certificates, from the data partition.

Data compatibility across rollback. The RAUC documentation asks whether a new application can read and convert old data, and rates fallback as tricky with single data storage and easy with redundant storage. Decide before the first field release; layouts are covered in Choosing an A/B Update Layout for Your Product.

Integrity and delivery. The root can be protected with dm-verity; the data partition is outside that protection. Large payloads can go to the data partition as separate artifacts, as in Containers vs Packages. We teach this design in TECH VEDA’s Embedded Linux with Yocto training.

Common wrong turns with a read-only root filesystem

• Remounting the root read-write to apply one change. It is one command and the change takes effect. The change is not part of the image, so the next update discards it, and the unit no longer matches the tested image.
• Placing the whole root under a persistent overlay to clear read-only errors. The errors stop, but the state inventory is lost, and any modified file can later hide a newer version from an update.

Checks before release on the target

1. The first command below must print ro as the first option.
2. The sudo touch test must report Read-only file system; without sudo, Permission denied proves nothing.
3. Run a full feature cycle, then search the journal of the current boot before rebooting; on a default Yocto image the log is on tmpfs and is lost at reboot.
4. Compare the directories on the data partition with the state inventory; an unlisted directory is an undocumented writer.
5. Update to the new release, roll back, and confirm the previous application starts and reports the same configuration values.

raghu@techveda.org:~$ findmnt -no OPTIONS /
ro,relatime
raghu@techveda.org:~$ sudo touch /testfile
touch: cannot touch '/testfile': Read-only file system
raghu@techveda.org:~$ journalctl -b | grep -i "read-only file system"

Output is from util-linux and GNU coreutils; BusyBox touch prints touch: /testfile: Read-only file system.

Key takeaways

• A read-only root filesystem moves the design problem to its writable state; decide it explicitly.
• Default to tmpfs for disposable data and named directories on a data partition for persistent data.
• Use overlayfs for the narrowest tree that needs it, with mount options that allow the lower layer to be replaced.
• Provision device identity, and decide single or redundant data storage, before the first release.

Frequently asked questions

Is overlayfs required to make a read-only root filesystem usable?
No. Tmpfs covers data that may be lost at reboot, and bind mounts or symbolic links to a data partition cover named state that must persist. Overlayfs helps when software writes to many unpredictable paths, but it has the update-time limits described above.

What happens to an overlayfs upper layer when the root filesystem is replaced by an update?
The kernel documentation allows offline changes to the lower layer only when the metacopy, index, xino and redirect_dir features have not been used, and it prohibits reusing an upper layer with different lower layers when index or metacopy is enabled. A file in the upper layer also takes precedence over the same file in the new lower layer, so a persistent upper layer can hide files delivered by an update.

Should the data partition be shared between the A and B root filesystem slots?
It can be, and it is the simpler setup. The RAUC documentation notes that it has no backup by default and that fallback is tricky, which in practice means old software may have to read data that newer software converted. Redundant storage makes fallback easier but needs a copy and migration on every update.

Further reading

• Overlay Filesystem: kernel documentation
• Yocto Project: Creating a Read-Only Root Filesystem
• Yocto Project: Image features reference
• OpenEmbedded-Core: volatile-binds recipe
• OpenEmbedded-Core: overlayfs-etc class
• RAUC: Advanced topics, Data Storage and Migration
• machine-id(5) manual page
• The Buildroot user manual