The Danger of Cargo-Cult Engineering: Why a Correct Fix Can Break Your System

You are bringing up a sensor on a new board. The vendor’s evaluation board already has a device tree node for the same sensor (the description that tells the Linux kernel what hardware is on the board), so you copy it, change the I2C address, and boot. The driver fails to probe. The chip-ID read gets no response on the bus.

Two days later, a scope on the reset pin shows it held low. The copied node described the reset line as active-low. On the evaluation board it is. On your board, the hardware team added an inverter on that line, so the sensor was held in reset the whole time.

The node was not wrong. It was correct for a board you do not have. The same thing happens with a retry copied from a forum answer, a flag taken from another team’s repository, or a fix an AI assistant suggested.

What is cargo-cult engineering?

The term comes from Caltech’s 1974 commencement address by the physicist Richard Feynman, published that June as “Cargo Cult Science”. He used it for work that copies every visible step of a method but leaves out the part that made the method work. He described it this way:

“They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work.”

Feynman based the name on a simplified account of Melanesian movements, and most anthropologists now reject the label as misleading. Engineers use it, often as “cargo cult programming”, for one narrow idea: the form of a practice has been copied, and its condition has not.

Every line depends on a condition

Almost every line in a working system is there because of some condition. The reset polarity matches the way the board is wired. A delay allows for how long a supply takes to become stable. A retry allows for a service that is slow to start.

When you copy the line, the condition does not come with it. Nothing in the text of GPIO_ACTIVE_LOW tells you it depended on the absence of an inverter.

“It worked there” is evidence about there. It tells you nothing about here until you have checked the condition.

This is different from not understanding the code. You may know exactly what msleep(100), a volatile qualifier or a retry loop does. The harder question is whether the condition each of them depends on exists in your system at all. Only the first kind of knowledge is carried by the copy.

Examples of cargo-cult code in embedded and Linux projects

• A Yocto bbappend (a file that changes someone else’s build recipe) from a forum, adding INSANE_SKIP to switch off a build-time quality check. The warning was harmless for the person who posted it. In your build it may be reporting a real problem, and now nothing reports it.
• volatile used to make a shared variable safe across CPUs. The kernel’s own documentation says its use in kernel code “is almost never correct”. It only stops the compiler from optimising the accesses. It does not make updates atomic or ordered between CPUs, which is the condition people copy it for. That needs locks or atomic operations.
• A device tree node an AI assistant generated for a closely related SoC. The pin and clock names compile on your part, but they select the wrong pads.
• A retry loop copied into a test script. In the original repository, a service was slow to start. In yours, the retry hides a race in the code under test that fails one run in three.
• Tests an assistant generated from the code rather than from the requirement. They pass because they check what the code does today, including its bugs.

Why cargo-cult fixes spread

A copied form usually appears to work on the day it is added, so it stays. Then it gets copied again.

The next engineer copies it from your driver, not from the original situation. “It is in three drivers already” then sounds like a reason, but all three copies come from one condition on one board, checked once at most.

AI assistants can make this worse. An assistant has seen the most common form of a pattern many times, often from older vendor code. It has seen the condition far less often, because the condition was usually never written down. Ask an agent to fix a flaky test and it will often add a retry, a sleep or a longer timeout.

Ask it why the line is needed, and it will give you a reasonable-sounding condition. It may also write that condition into a comment or commit message, where the next engineer will trust it more than a bare line.

An assistant’s explanation of why a line is needed is a claim to check, not a check.

How to check whether a line is cargo-cult code

For anything you copied, generated with an assistant, or found in a codebase and cannot explain, ask two questions.

What condition does this depend on? Name it in one sentence: “the rail takes 80 ms to become stable”, “the reset line is active-low on this board”. If you cannot name it, that is your first finding. With generated code, you can ask the assistant to state the condition for each non-obvious line, as a list of claims to verify.

Does that condition exist here? Check it against something real: the schematic, the datasheet, the timing on your board, the log from your build. Not against the source you copied from, or the assistant’s explanation of it.

For code already in the tree, start with its history. Run git blame or git log -S and read the commit that added the line. The condition is often recorded there. If it is not, use a removal test: on a branch, before you delete the line, write down what you expect to break. Then remove it and test. If you could not predict anything, you do not yet know its condition.

If nothing breaks, do not delete it yet. Your tests may simply not cover the condition, such as a colder board or an older silicon revision. Keep the line, write down the condition you suspect, mark it unverified, and ask whoever owns that code. For power sequencing, reset timing or anything safety-related, read the datasheet first. Removing a line to see what happens can damage hardware.

How to stop your code becoming someone else’s cargo cult

A few years into your career, other engineers copy your driver, recipe or build script exactly as you left it.

So write the condition next to the form. A comment that says /* ACTIVE_HIGH: rev B adds an inverter on RESET_N */ is worth more than one that says /* active low */, because the first tells the next engineer when the line no longer applies. State only conditions you have checked, including in comments and commit messages an assistant drafted for you. How to write those commit messages is the subject of The Code Is Yours. The Understanding Is Not.

If you review other people’s changes, make one request: for any delay, retry, skipped check or unusual flag, the pull request description should name the condition it depends on and say how it was checked.

Try this on your own codebase

Pick three lines in your project that nobody on the team can explain: a delay, a flag, a compiler option, a skipped check. For each, read its history, write the condition you think it depends on and what you expect to break without it. Then test one on a branch.

Before you keep any line, copied or generated, name the condition it depends on and check that condition here.

Copying is not the problem, and neither is using an assistant. Good examples save real time. The danger is keeping the fix and assuming the condition was the same.

This is one reason TECH VEDA sessions are live and built around problem scenarios: when you work through a scenario with a mentor and other engineers, “why is this line here?” gets asked at the moment you would otherwise copy it.

Source: Richard P. Feynman, “Cargo Cult Science”, Caltech commencement address, 1974; published in Engineering and Science, June 1974.

— Raghu Bharadwaj