An A/B Update That Could Not Read the U-Boot Environment

The symptom

The board was an Avnet module built around NXP’s i.MX8M Plus, running a Yocto image with Mender integrated for A/B root filesystem updates. In an A/B design the bootloader owns the decision: it reads stored state, picks one of the two root filesystems, and tells the update client which one it chose. On these boards that state lives in the U-Boot environment.

The image booted. Then every update operation failed the same way, as reported by pgthompson24 on the Mender community forum:

Failed to read the current active partition: No match between boot and root partitions.

Nothing was wrong with the board. It booted, it ran, the filesystem was intact. The client simply could not establish where it was running from, and without that it installs nothing.

The obvious cause, and what correcting it revealed

Userspace reads the U-Boot environment through fw_printenv and fw_setenv, which learn where it lives from /etc/fw_env.config. Working through Mender’s integration checklist, the reporter found that file naming the wrong device: /dev/mmcblk0, when the board boots from /dev/mmcblk1.

That is a real defect and worth fixing. It is also the kind of finding that ends an investigation early, because it explains the symptom well enough to satisfy. Correcting it did not clear the error. It made it longer:

Failed to read the current active partition: No match between boot and root
partitions.: Failed mender_saveenv_canary check. There is an error in the
U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader
environment location and the location specified in /etc/fw_env.config.
2) 'mender_setup' is not run by the U-Boot boot script: exit status 1

The longer message is progress. With the device name corrected, userspace could finally reach a U-Boot environment, which let a deeper check run at all — and that check failed. The canary it names is a deliberate tripwire: a value the bootloader is supposed to write into the environment during boot, which the client reads back from Linux. If the bootloader never writes it, or writes it somewhere userspace cannot see, the canary is missing and the client refuses to proceed rather than guess. The message names the two situations that produce it.

Reading the U-Boot environment from both sides

The next observation settles which of the two it is, and needs no tooling beyond a serial console. The reporter interrupted the boot to reach the U-Boot prompt, listed the environment there, then listed it again from Linux:

=> printenv

fw_printenv

The two produced entirely different sets of variables. Not different values for the same variables, which would suggest a stale copy or a failed write. Different variables. The bootloader and userspace were describing two unrelated environments, and none of Mender’s variables were in the one U-Boot was using.

Was the right bootloader even built?

At this point the obvious suspicion is that the Mender integration never reached the bootloader at all, and the thread goes after that first. Josef, posting as TheYoctoJester, suggested establishing which U-Boot the build actually uses. On a Yocto build that is answerable directly:

raghu@techveda.org:~$ bitbake -e virtual/bootloader | less
raghu@techveda.org:~$ bitbake -vn virtual/bootloader
raghu@techveda.org:~$ bitbake-layers show-appends

The answers came back clean. The build selected u-boot-imx as the bootloader provider, and the layer’s own bbappend for it was being applied, pulling in u-boot-mender.inc. The U-Boot binary was being patched for Mender exactly as intended.

Which eliminates the first of the two causes the canary named, and leaves the second.

Root cause

Patching the U-Boot binary is not the same as patching the boot process, and on most Arm boards those are two different artifacts.

Mender’s integration hinges on mender_setup, a script that must run at the very start of the boot command. It performs no boot steps itself; it reads the update state, decides which partition is active, and populates the variables everything downstream depends on. The documentation is explicit that the board’s boot command has to call it — bootcmd=run mmcboot becomes bootcmd=run mender_setup; run mmcboot — with the kernel and root arguments then referencing ${mender_uboot_root} and ${mender_kernel_root}.

On these boards the boot command does not live in the U-Boot binary. It lives in a boot script, compiled from a boot.cmd.in template into a boot.scr that U-Boot loads and runs, and patching U-Boot leaves that script untouched. whsplk made this point in the thread, pointing at the per-BSP patches in meta-mender-community that exist to do exactly this; the Toradex and Odroid layers each carry one. The i.MX8M Plus build had no equivalent.

So U-Boot booted the board the way the vendor BSP always had, from its own U-Boot environment, never running mender_setup, never consulting the update state, never writing the canary. The client, reading the other environment, found no Mender variables and refused to act.

The consequence is larger than an update that will not install. That same environment carries the boot counter that drives rollback, so a bootloader which never runs the Mender logic is not counting failed boots either. The rollback the A/B layout exists to provide would not have been there when it was needed.

One honest note: the thread ends with the diagnosis, not with the reporter confirming a fix. What follows is the documented integration those two causes point at, not a repair anyone posted.

The fix, and the check that would have caught it

Where a BSP has no Mender boot script patch, the integration is done by hand and the documented steps are specific. The boot script must call mender_setup first and reference the Mender root variables. The board’s U-Boot configuration needs CONFIG_BOOTCOUNT_LIMIT and CONFIG_BOOTCOUNT_ENV for rollback, and CONFIG_ENV_IS_IN_MMC for the U-Boot environment itself, with the other CONFIG_ENV_IS_IN_ and CONFIG_BOOTCOUNT_ options off. BOOTENV_SIZE in the bitbake recipe must equal CONFIG_ENV_SIZE in the board header. Where automatic patching is in play, its output is a useful starting point:

raghu@techveda.org:~$ bitbake -e u-boot | grep '^MENDER_UBOOT_AUTO_CONFIGURE='
raghu@techveda.org:~$ bitbake -c save_mender_auto_configured_patch u-boot

The check that would have caught this on day one is the comparison the reporter eventually made: stop at the U-Boot prompt, run printenv, boot to Linux, run fw_printenv, confirm the two agree. It takes a minute and it is the only test that exercises both sides of the boundary. Everything above it — the config file, the recipe, the layer, the client — can be individually correct while the two halves address different storage.

The same failure elsewhere

This is a failure class, not a Mender problem. It recurs wherever an updater and a bootloader share the U-Boot environment.

On RAUC the shared variables are BOOT_ORDER and the per-slot BOOT_<slot>_LEFT counters. A report on an i.MX8M Mini module showed both slots marked bad and fw_printenv returning a configuration error; the maintainer’s answer was that U-Boot and userspace must have the same idea of where the environment lives. A Raspberry Pi 4 Buildroot build hit the same wall, RAUC unable to get the primary slot because fw_printenv exited non-zero.

The arithmetic variant produces the purest version of this symptom: an update that installs, reports success, and boots the old slot forever. Here both sides can read a U-Boot environment, so nothing errors — they simply address different bytes. U-Boot takes its location from CONFIG_ENV_OFFSET, plus CONFIG_ENV_OFFSET_REDUND when redundancy is enabled, counted from the start of the device. /etc/fw_env.config states device, offset and size independently, and is commonly written relative to a partition. If the two do not land on the same bytes, fw_setenv updates a region U-Boot never reads: the tool reports success, the file shows the new value, and the board keeps booting what it booted before.

Two practices close this off. Derive both descriptions from one source in your build rather than by hand. And make the two-sided comparison a standing item in board bring-up, before any update logic is layered on top. We cover this integration work in our Embedded Linux and Yocto training.

Key takeaways

• An error message getting longer after a fix can be progress. It often means an earlier failure was masking a deeper check.
• Patching the bootloader binary and patching the boot process are different things. On Arm boards the boot command usually lives in a boot.scr built from boot.cmd.in, which a U-Boot bbappend does not touch.
• Compare the U-Boot environment from both sides during bring-up: printenv at the U-Boot prompt against fw_printenv in Linux. Different variables means two environments.
• Update state and rollback state are the same state. A bootloader not reading one is not maintaining the other, so rollback is absent rather than degraded.
• A canary that fails loudly is worth building. Guessing the active partition would have been worse than refusing to run.

Frequently asked questions

Why did fixing /etc/fw_env.config make the error longer?
The wrong device name stopped userspace reaching any environment at all, which masked a later check. With the device corrected, the client could read an environment and went on to verify that the bootloader had written its canary value into it. That check then failed, and its message names the two situations that cause it.

What does mender_setup actually do?
It is an environment script that must run at the very start of the boot command. It performs no boot steps itself, but it reads the update state, works out which partition is active, and sets the variables the rest of the boot uses for the kernel and root arguments.

How do I check that U-Boot and Linux agree about the environment?
Stop at the U-Boot prompt and run printenv, then boot to Linux and run fw_printenv. If the two show different variables rather than different values, the bootloader and userspace are using separate environments and no update or rollback logic above them can work.

Further reading

• Mender Hub: Mender on iMX8 Plus U-Boot Setup Errors — the primary record of this incident.
• Mender documentation: Manual U-Boot integration
• meta-mender-community — the per-BSP boot script patches.
• RAUC discussion #1043 — the same class on an i.MX8M Mini module.
• RAUC issue #785 — the same class on a Raspberry Pi 4 Buildroot build.
• RAUC documentation: Integration