The Mid-Career Engineer’s Reset: What to Keep, Deepen, and Relearn

Consider this illustrative example. An engineer with eight years of bare-metal firmware joins an embedded Linux product. In the first design review, they propose mapping the peripheral registers into the application and writing to them directly. “That is how we always did it,” they say. “It is the shortest path to the hardware.”

A colleague asks what justified it. On the microcontroller, nothing else owned the peripheral. On this board, a kernel driver may already own it. Whether the access works depends on how the kernel is configured; whether it is a sound design is a separate question.

The habit may have been justified in firmware. The mistake is carrying it over without carrying over, and rechecking, the reason. In my experience, a mistake like this shows up as an intermittent fault in integration, not as an error when the code is written.

Confidence carries no label for the platform it was earned on.

Principles, local policies and unchecked assumptions

You do not need to discard your experience when you move from firmware to Linux. You need to identify which decisions were principles, which were local policies, and which have not yet been checked against the new platform. A principle, such as keeping an interrupt handler short, holds on both. A local policy, such as the application owning the hardware, answered the conditions of a single image on a single chip. An unchecked assumption is one you have not yet tested either way.

Here is a second example, with no kernel in it. Suppose enlarging a frame queue stopped dropped frames on your previous product, because the load arrived in short bursts and the application caught up between them. The rule you carried forward is “when frames drop, enlarge the queue”. It lost its condition. If frames keep arriving faster than the application processes them, a larger queue only delays the next drop, and both explanations fit a short, successful test.

Ask of any old rule: what had to be true for this to be good advice, and is it true here?

A copied fix can lose its context. So can your own experience. This audit is about the second case: decisions you understand, trust and repeat because they served an earlier platform. For the first case, see the article on copied code. For the problem of corrections that never reach you, see Ten Years of Work Is Not Ten Years of Experience.

What the research measured

Research on how people correct mistakes finds that errors made with high confidence are often corrected more readily than errors made with low confidence, once the right answer is shown. Studies of this effect have mostly asked people general-knowledge questions.

That does not mean confidence usually signals error: overall, confidence and correctness are highly correlated. The narrower lesson is this.

A confident mistake need not resist correction once it is exposed.

The correction has to be shown, which is why step 3 of the audit checks a primary source. This research looked at general-knowledge errors, not engineers changing platforms, so the audit below is an engineering recommendation informed by it, not a validated intervention.

Three audit examples

These examples are illustrative, with invented ratings. The engineer from the opening writes ten assumptions from memory, rates each for confidence from 1 to 5, notes where each was learned, and checks each against the Linux platform. Three of the ten:

• “The application owns the hardware, so it writes the registers directly.” Confidence 5. Learned on firmware. The kernel’s Kconfig help for IO_STRICT_DEVMEM says that, when it is disabled, userspace can access all io-memory regardless of whether a driver is actively using that range. It calls accidental access “obviously disastrous” and says specific access can be used by people debugging drivers. When the option is enabled, /dev/mem allows only idle ranges. Result: relearn. Find the driver that owns the peripheral and the interface it exposes, and treat /dev/mem as a debugging tool. Check how your own kernel is configured.
• “Register accesses must happen exactly as written, in order.” Confidence 5. Learned on firmware. In firmware, the engineer used a volatile pointer. The requirement stands, but the means differ. The kernel documentation says ioremap makes bus memory accessible through readb, readw, readl, writeb, writew, writel and other MMIO helpers, and that the returned address is not guaranteed to be usable directly as a virtual address. Result: deepen. Learn the accessors before you write driver code.
• “Keep the interrupt handler short: acknowledge the interrupt and defer the rest.” Confidence 5. Learned on firmware. The kernel’s hacking guide says the handler has to be fast, and that frequently it simply acknowledges the interrupt, marks a software interrupt for execution and exits. The mechanism differs from your firmware’s, but the reason behind the rule holds. Result: keep. Which knowledge lasts across tools is argued in Tools vs Fundamentals.

This is an illustration, not evidence about how often each result occurs, and your own items may all be keep.

What I recommend: a keep, deepen and relearn audit you can copy

1. Choose ten decisions or beliefs you hold from memory and expect to use on the Linux platform: practices, commands, settings, estimates. Next to each, write where you learned it.
2. Write each one down alone, and give it a confidence from 1 to 5. Do this before you open anything. Note the reason it was true in firmware.
3. For each, ask what made it true, and whether the Linux platform has that. Check the primary source: kernel documentation, your SoC vendor’s Linux notes, the product’s requirements.
4. Sort the results. Keep means you have verified that the knowledge applies and that its relevant conditions still hold. Deepen means its applicability remains uncertain, or you cannot yet explain the conditions well enough to rely on it. Relearn means the assumption, policy or procedure needs to change for the new platform.
5. Prioritise items whose failure would have the greatest consequence. Within that group, examine high-confidence assumptions carefully. For relearn items, write the corrected form from memory the next day, update the engineering artifact where the assumption lives, and assign an owner.

Test a keep. Remove the board, operating system and tools from the memory. If you would still do it on the first morning of an investigation, it travels. Then write it so a colleague can apply it in your absence: not “run the test for longer”, but “keep the workload running and watch whether the queue returns to its usual level”.

Aim a deepen at your recurring handoff. Listen for a sentence you have said several times this year, such as “once it gets into the kernel, I need someone else”. Pick the handoff that limits the work you want to own next, and set one assignment that carries a single real decision across it: not “learn the kernel”, but “trace one interrupt from the device to the application and account for the time it waits”.

Challenge a relearn with a colleague. Ask them to bring a case where your familiar rule would fail. If they find one, update the design note or test procedure while the example is fresh.

Your experience becomes more useful when its limits are written down.

What this audit cannot tell you

This audit examines the assumptions you list; it cannot reveal requirements you have not found yet. Memory, boot, timing and update behaviour on Linux differ from firmware and need their own entries, and new hardware constraints, regulation and customer expectations need investigation with the people responsible for them.

A familiar chip family does not settle the audit either. The Linux board may use different drivers, interfaces and boot flow from the bare-metal SDK you know.

“Keep” is a valid result, but it should come from checking, not from the starting assumption.

What to try this week

Pick one upcoming review, release or investigation. Write one line for each part of the reset. For the engineer in the opening, the page might read:

• Keep: Ask whether the test reproduces the workload and duration that caused the field failure.
• Deepen: Trace one interrupt from the device to the application and account for the time it waits.
• Relearn: Treat direct register access from the application as a proposal to justify against the driver that owns the peripheral.

Take the page into the review. Afterwards, ask what changed: did you request a more revealing test, explain a trace you would once have forwarded, or withdraw a recommendation when the evidence went against it? For a monthly version that repeats a whole task, see Why Learning Should Be a Process, Not an End Goal.

Your move: find the last engineering decision you signed off on but could not fully explain. Before your next review, investigate the one part you took on trust. TECH VEDA sessions use the same order: you commit to an answer first, and the mentor then shows the current behaviour.

— Raghu Bharadwaj