The Fetch Error That Fixed Itself: A BitBake SRCREV Mystery

On Monday the build failed in do_fetch with an error nobody on the team had seen before. The next Tuesday it built cleanly — on the same machine, from the same manifest, with no changes in between. The second fact worried the team more than the first. A failure you fixed is a solved problem. A failure that went away on its own is a mechanism you do not understand yet. It will happen again.

This failure is worth understanding. It comes directly from the way most silicon vendors publish their BSPs. And the error it produces makes you look for the problem in your own setup, which is the one place it is not.

The setting: syncing a vendor release on day one

The team was bringing up a gateway product on a vendor reference SoM, using the vendor’s Yocto-based BSP. The vendor had just announced its quarterly release: release notes, a fresh manifest, and new branches on its public GitHub mirrors. The release contained a U-Boot fix the team had waited three months for. So they set it up the normal way:

raghu@techveda.org:~/bsp$ repo init -u https://github.com/vendor/bsp-manifest -b vendor-linux-rel -m rel-2.1.0.xml
raghu@techveda.org:~/bsp$ repo sync
raghu@techveda.org:~/bsp$ bitbake core-image-minimal

repo sync finished without errors. Remember that detail. All the metadata was in place.

The symptom: an error that looks like a local problem

The build ran for a while, then stopped in the bootloader recipe:

ERROR: u-boot-vendor-2026.04-r0 do_fetch: Fetcher failure: Unable to find
 revision e7b3a95c412df0a6b81c2f7d9e04a15c6b3d8f42 in branch vendor_v2026.04
 even from upstream
ERROR: u-boot-vendor-2026.04-r0 do_fetch: Fetcher failure for URL:
 'git://github.com/vendor/uboot-vendor.git;protocol=https;branch=vendor_v2026.04'.
 Unable to fetch URL from any source.

The team tried the usual recovery steps. They deleted the recipe’s download under DL_DIR. Same error. They ran cleanall. Same error. A colleague tried on a different machine, on a different network, with no proxy. Same error. The working theory became: the recipe in this release is broken. So they looked at the recipe:

SRC_URI   = "git://github.com/vendor/uboot-vendor.git;protocol=https;branch=${SRCBRANCH}"
SRCBRANCH = "vendor_v2026.04"
SRCREV    = "e7b3a95c412df0a6b81c2f7d9e04a15c6b3d8f42"

A pinned commit on a named branch. This is what every vendor U-Boot recipe looks like. Nothing unusual. Nothing visibly wrong. So why can nobody, on any network, fetch this commit?

The mechanism: what BitBake actually checks for a SRCREV

To read this error correctly, you need to know three things about BitBake’s git fetcher.

What it checks. For a git:// URL, the fetcher keeps a bare mirror clone of the full repository under DL_DIR/git2/. It fetches refs/heads/* and refs/tags/* from the remote. Then it checks that the pinned revision is part of the branch named in the recipe. From lib/bb/fetch2/git.py (current BitBake; on master the file is lib/bb/fetch/git.py after the 2026 rename):

def _contains_ref(self, ud, d, name, wd, tag=False):
    git_ref_name = 'refs/tags/%s' % ud.parm['tag'] if tag else ud.revision

    if ud.nobranch:
        cmd = ud.basecmd + ['log', '--pretty=oneline', '-n', '1', git_ref_name, "--"]
    else:
        cmd = ud.basecmd + ['branch', '--contains', git_ref_name, '--list', ud.branch]

With branch= set, the check is exactly git branch --contains <SRCREV> --list <branch>. The commit must exist, and it must be reachable from the tip of that branch in the remote repository.

Why it can fail when your setup is fine. If the check fails, the fetcher fetches everything from the remote once more and checks again. Only then does it stop:

if not self._contains_ref(ud, d, ud.name, ud.clonedir):
    raise bb.fetch.FetchError("Unable to find revision %s in branch %s even from upstream"
                              % (ud.revision, ud.branch))

So the error has only two possible causes, and both are on the remote side. Either the commit is not in the public repository at all. Or the commit exists but is no longer reachable from the named branch — this is what happens when a vendor rebases or force-pushes a release branch, and it produces the identical error message.

Why it recovers on its own. The update path fetches all heads and tags from the remote every time the check would fail:

def clonedir_need_update(self, ud, d):
    if not os.path.exists(ud.clonedir):
        return True
    ...
    if not self._contains_ref(ud, d, ud.name, ud.clonedir):
        return True

As soon as the commit appears on the remote, the next bitbake run fetches it and continues. No local cleanup is needed, because no local cleanup was ever relevant. The build “fixed itself” because something changed on the remote.

The turning point: released does not mean fetchable

At this point the investigation stops being about the build. It becomes about the order in which the vendor publishes a release. The check that should have been the first step — it takes thirty seconds — is to ask the vendor’s public repository what it has:

raghu@techveda.org:~$ git ls-remote https://github.com/vendor/uboot-vendor.git vendor_v2026.04
9c41d7e2f0a8beaf30dd12c66e4a9271c5f80d13        refs/heads/vendor_v2026.04

The branch tip on GitHub is 9c41d7e2. The recipe pins e7b3a95c. The pinned revision is not an ancestor of the public tip. It is newer than anything in the public repository. The release metadata points to a commit that does not exist on the public internet.

This sounds impossible until you look at how these releases are built. The vendor develops on an internal git server. Their release automation updates SRCREV in the layer as soon as a commit passes internal CI. The release is then published in stages. The manifest and release notes go out first. The source mirrors on GitHub are synchronized by a separate process, on a separate schedule.

In the incident behind this story, public data allowed us to reconstruct the full sequence afterwards. The commit was created on the vendor’s internal server, and the layer pinned it as SRCREV the same day. The manifest that names this layer went public about a week later. That is the day the release became available to sync — and impossible to build. The public U-Boot mirror received the commit a full week after that. GitHub recorded the change:

raghu@techveda.org:~$ curl -s "https://api.github.com/repos/vendor/uboot-vendor/events?per_page=100" 
    | jq -r '.[] | select(.type=="PushEvent")
             | "(.created_at) (.payload.ref) (.payload.before[0:12]) -> (.payload.head[0:12])"'
2026-09-17T19:06:41Z refs/heads/vendor_v2026.04 9c41d7e2f0a8 -> e7b3a95c412d

One push event. It moves the branch directly from the old release’s revision to the new one — fifteen days after the recipe started pinning it, and seven days after the manifest went public. Every engineer who synced during that window got the same failure and tried the same recovery steps. Everyone who retried after the push found that it “worked now”.

The release tag confirms the timeline from a second direction. An annotated tag records when it was created, not only what it points at:

raghu@techveda.org:~$ git cat-file -p refs/tags/rel-2.1.0 | head -4
object 3d1c8a97e5b24f60d9a1c7e83f52b0a4d6e91c25
type commit
tag rel-2.1.0
tagger Release Automation <rel@vendor.example> 1789677300 -0700

Convert that epoch with date and you get the evening of the mirror push. The public release was assembled days after it was announced. None of this is wrongdoing. It is a pipeline whose internal order is invisible from outside. But it means “the vendor released version X” and “you can build version X” are two different events, sometimes weeks apart. The only visible symptom in between is a fetcher error that names no cause and suggests no action.

Next time: a thirty-second check instead of three lost hours

Once you know the mechanism, the whole debugging session reduces to one command and one comparison. When do_fetch reports a revision it cannot find “even from upstream”, ask the remote before touching DL_DIR:

raghu@techveda.org:~$ git ls-remote https://github.com/vendor/uboot-vendor.git vendor_v2026.04

Then compare the recipe’s SRCREV against the answer:

raghu@techveda.org:~/uboot-vendor$ git merge-base --is-ancestor <SRCREV> <public-tip> 
      && echo "reachable - problem is local after all" 
      || echo "not on the branch"

There are three possible outcomes.

If the SRCREV is reachable from the public tip, the problem really is on your side — proxy, DNS, or a corrupted DL_DIR mirror. The usual cleanup steps now make sense.

If the commit is missing from the remote, you are inside a mirror-lag window. Nothing you do locally will help, and nothing you do locally is needed. The first build after the vendor’s push will succeed without any change.

If the commit exists but is not on the branch, the vendor rewrote history. That is a larger problem, but it produces this same error message.

While a mirror-lag window lasts, you have two reasonable options. You can build the vendor’s previous release manifest; its revisions have been public for months. Or, if you need the new content, you can pin the advertised public tip locally — and record that you did:

# local.conf - TEMPORARY: public mirror lags release metadata, see ticket #1234
SRCREV:pn-u-boot-vendor = "9c41d7e2f0a8beaf30dd12c66e4a9271c5f80d13"

Be aware of the limit of this override: it builds a revision that is not the one in the release notes. That is acceptable during bring-up. It is never acceptable for a release build. Remove it as soon as the mirror is updated.

Long term: remove the dependency on vendor push timing

The durable fix is to fetch from infrastructure you control. OpenEmbedded has supported this for years. On a build machine that has fetched successfully, let BitBake produce mirror tarballs of its git clones:

# local.conf on the machine that populates the mirror
BB_GENERATE_MIRROR_TARBALLS = "1"

Publish the resulting DL_DIR tarballs on an internal web server. Then point every other build at that server first. own-mirrors.bbclass in openembedded-core does this with two lines of configuration:

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "http://mirror.internal.example/sources/"

The class adds your mirror to the front of PREMIRRORS for every fetcher scheme, including git://. Upstream is then only contacted for revisions your mirror has never seen. A team that runs this pattern meets a vendor mirror-lag window exactly once, on the single machine that populates the mirror — not on every developer desk and CI runner. The same setup protects against the more serious variants of this failure: vendor repositories that change hosting, branches that get rebased, releases that get withdrawn. Sources that built once stay buildable, on your schedule. (How long each part of a vendor BSP stays supported is a separate planning problem; we covered it in Your Product Has Five Support Clocks.)

Postscript: reading a vendor release from the outside

The techniques used in this incident fall into two groups. Both work without any help from the vendor.

The fast checks, for the moment something fails: git ls-remote shows what the remote can serve right now — the fact the error message never tells you. git merge-base --is-ancestor shows where a pinned revision stands relative to the public tip.

The deeper reconstruction, for when you want the timeline: the layer’s git history dates every SRCREV update, which documents the vendor’s internal pipeline. git cat-file -p on an annotated release tag shows the tagger date — when the release was really assembled, which is often later than the announcement. A public forge’s events feed records when each branch moved, and from which commit to which. Save it early; GitHub keeps about ninety days. With these sources you can usually reconstruct a vendor’s release timeline to the hour. That turns “it fails, sometimes, for some people” into a dated window with a clear cause.

The habit behind all of this is simple. When a fetch error says a revision does not exist, believe it. The error is not a symptom of your network. It is a statement about the remote — and the remote keeps records.

Vendor release structure, fetcher internals and mirror strategy make the difference between a smooth BSP upgrade and weeks of lost time. Our Embedded Linux BSP Development course covers them on real vendor BSPs.

Key takeaways

• BitBake’s git fetcher does more than fetch a commit. With branch= in SRC_URI, the SRCREV must be reachable from that branch on the remote (git branch --contains). “Unable to find revision … even from upstream” means the remote really lacks it — not pushed yet, or removed by a rebase.
• Vendor BSP releases are published in stages: recipes pin internal revisions first, manifests and release notes go public next, and source mirrors sync last. Anyone who builds in the gap gets a do_fetch failure with nothing wrong on their side. The failure clears on its own after the mirror push, because the fetcher re-fetches all refs on every retry.
• Before touching DL_DIR, run git ls-remote <repo> <branch> and compare with the recipe’s SRCREV. Reachable: the problem is local. Missing: mirror lag — wait, or build the previous release. Present but unreachable: rewritten history.
• A temporary SRCREV:pn-<recipe> override to the advertised public tip lets bring-up continue, but it builds a revision the release notes do not describe. Comment it, track it in a ticket, and remove it.
• BB_GENERATE_MIRROR_TARBALLS plus own-mirrors.bbclass (SOURCE_MIRROR_URL) makes downstream builds independent of vendor push timing and history rewrites. Only the machine that populates the mirror is ever exposed to the vendor’s schedule.
• Forge events feeds and annotated-tag tagger dates let you reconstruct a vendor’s release timeline from public data. Save the events feed early; retention is short.

Further reading

• BitBake User Manual: Fetchers — the git fetcher, SRCREV and branch=.
• bitbake.git — lib/bb/fetch/git.py — the containment check and the error in this story.
• Yocto Project Reference Manual: variables — PREMIRRORS, SOURCE_MIRROR_URL, BB_GENERATE_MIRROR_TARBALLS.
• openembedded-core — own-mirrors.bbclass — the two-line internal source mirror.
• repo Manifest Format — how vendor manifests pin layer revisions.