Four Levels of Knowing. Most Engineers Misjudge Which One They Are On.

A device does not appear. The driver is in the tree, the node is in the device tree, and probe() never completes. Four engineers can look at that log and all four will say they know what deferred probe is. They will mean four different things by it.

The four levels on one problem

Familiarity is recognition. You have read about deferred probe. When -EPROBE_DEFER appears you know you have seen the term before. Asked what it does, you can describe it in general terms and not precisely.

Recall is production without the source. You can state that a driver returns -EPROBE_DEFER when a resource it needs is not yet available, that the kernel will retry it later, and that there is a list of still-pending devices at /sys/kernel/debug/devices_deferred. You could write that on a blank page with nothing in front of you.

Understanding is the mechanism, and the difference from recall becomes clear the moment somebody asks why. The short answer is that the driver returns -EPROBE_DEFER and the kernel tries again later. The full one has three parts. probe() is called before the kernel can know whether a clock, regulator or GPIO the driver needs has a driver of its own yet, so the device goes back on the deferred list and is retried each time some other driver binds successfully. That retry loop is really a discovery mechanism, and an expensive one, because it discovers by repeated failure an ordering the firmware description already contained — which is why fw_devlink now reads the consumer and supplier relationships out of the device tree and blocks the consumer until its supplier has probed. And because a supplier that never gets a driver would otherwise block its consumer forever, deferred_probe_timeout exists to give up after a point and fail those requests. Once you have all three, you can predict things nobody told you. You can say which of two drivers binds first from the device tree alone, and what the log looks like when a supplier simply never arrives. This level is developed properly in Where Is Your Mental Model?, and it is the level most study material is written to reach.

Competence is that same knowledge used under conditions you did not choose. The board boots nine times out of ten. The vendor tree carries four hundred patches nobody documented. There is a delivery date this week, and the question you have to answer is not what deferred probe is. It is whether the deferral you can see is the fault or a symptom of it, and whether the fix you have when the time runs out is one you are prepared to approve.

All four feel the same to the person who has them. That is the entire difficulty.

Why your own level is the hardest one to judge

This is the illusion of competence, and it operates at all four levels. Each level is difficult to identify while you are at it, and obvious afterwards. Somebody at the recall level cannot easily tell that they are not at understanding, because recalling something gives you no sign of the difference. The sensation of having an answer is identical. It is only when the situation shifts slightly — a variant nobody showed you, a board that behaves differently from the one in the article — that the levels separate, and by then you are already committed.

This is why the useful diagnostic is never how well you know something. It is what your knowledge does when the problem is not the one you rehearsed.

One thing applies to all four and is not a level at all. Hours you were only half present for teach you nothing, equally at every level, which is why two engineers can do the same exercise and only one of them moves up.

Ask what breaks first, not how much you know. Each level fails in a way the others do not.

An engineer who understood it and still could not fix the board

Consider an engineer who genuinely has the model described above. Not someone pretending. They can explain the deferred probe mechanism completely, including the parts most people skip.

They are given a board that fails to bring up an audio codec on roughly one boot in ten. They begin with the check the model implies. On a failing boot they read /sys/kernel/debug/devices_deferred and it is empty, which by the model means nothing is pending and there is nothing to wait for. That is the first surprise, and it costs an hour, because a correct model with one missing fact produces confident work on the wrong problem.

What the model does not contain is anything about this product. The I2C expander that carries the codec’s regulator is built as a module on this configuration, and it is loaded from user space by a startup script. The vendor tree sets deferred_probe_timeout to a low value, because somebody two years ago wanted faster boots on a different board. On the nine good boots the module loads before the timeout expires. On the tenth the filesystem is a little slower, the timeout expires first, the kernel gives up waiting on a supplier that has no driver yet, the codec’s request fails permanently, and the deferred list is empty afterwards precisely because the kernel has stopped waiting.

Everything they understood was correct. The mechanism behaves exactly as they described it. What they did not have was the habit of asking, early and cheaply, which parts of this system are not the standard system — and the reason they did not have that habit is that very little material teaches it, because it is not knowledge. It is a reflex you only build by being wrong about a specific board while somebody waits.

The competent approach to that fault is not smarter, and it does not come from having read more. It is the same model, plus enough attention in the first ten minutes to notice that one detail in front of them did not belong to the standard system. That is trained attention, and it is what decides whether hours spent at any of these levels actually change the level. In practice it is one question asked forty minutes earlier: what is different about this tree. Then initcall_debug and the dmesg timestamps on a good boot beside a bad one, which shows the ordering changing rather than the mechanism failing. And then the harder part, which is not technical at all: deciding whether to raise the timeout and ship, knowing that raising it hides the race rather than removing it, or to build the expander in and justify the boot-time cost to a person who has already objected to it.

Being right about the subsystem is not the same as being right about the board in front of you.

The misdiagnosis, and what it costs

Effort applied at the wrong level produces almost nothing, and it does not feel like wasted effort while it is happening, because it is real work. An engineer at familiarity who believes they are at understanding reads more material; it reads as easily as the last material did and their position does not change. An engineer at recall who believes they are at understanding does more exercises of the kind they can already do. An engineer with genuine understanding who believes it is competence takes a real problem and spends hours on a fact that was never in any article.

That last case is the one that damages people, because they decide their understanding must have been faulty and return to the material, applying the remedy for a level they passed months ago. Usually they understood it perfectly well. They had built the model and had never used it anywhere that mattered, and those are different achievements.

Reading again is the correct remedy at exactly one of the four levels, and it is not the level most people are on when they choose it.

Two limits of the model

Two objections are worth making, because the model is more useful once it is bounded.

The first is that these are not a rank you have as a person. They are per subsystem, and often per board. The same engineer is competent in the clock framework and merely familiar with the display pipeline, at the same time. Anyone who has watched a strong engineer move to an unfamiliar area has seen this: nothing about them changed, and the level did.

The second is that the ordering is looser than a numbered list suggests. You do not finish one and begin the next. In practice you remain at recall on a subject for months while the parts of it you have actually had to use reach competence, and the overlap between them is where most real work happens. The order matters for one purpose only, which is choosing the remedy. You cannot fix a familiarity problem with a competence exercise, and putting somebody who has read one article about clocks in front of a failing board under time pressure does not build competence. It builds the belief that they are not capable of this work.

Using the levels

Neither of the lower levels is a failure state. Familiarity is the first pass and is supposed to be quick. Recall can be looked up instantly now and it has not stopped mattering, and both of those are true at once: you cannot reason about a system whose parts you have to look up one at a time, because reasoning needs them present at once.

The fourth level is the difficult one to arrange, because three of the conditions that define it are ones you do not naturally give yourself: a problem you did not choose, a deadline you did not set, and a cost for being wrong. That is difficult, not impossible. Upstream work supplies all three — a bug reported by somebody else, on hardware you do not have, reviewed in public by a maintainer who will say so when you are wrong. If that is not available to you, the conditions can be approximated deliberately.

• Take a visibly failing fault from a subsystem you have not read. Choosing your own practice material removes the hardest condition every time, and it is the one everyone removes without noticing.
• Set a time limit for one session. The purpose is not speed. It is to watch the point where you stop listing causes and start choosing between them, because that transition is what real work consists of.
• Ask what is non-standard before you ask what is wrong. Out-of-tree patches, modules that could have been built in, a boot parameter somebody set two years ago. On a vendor tree, that question repays the time more often than any other.
• Compare a good run with a bad one rather than studying the bad one. Two dmesg logs side by side, or initcall_debug on both. Intermittent faults are ordering problems far more often than they are logic problems, and ordering is invisible in a single capture.
• State a conclusion to somebody before the result is known. A private prediction can be revised after the fact. One you have said out loud cannot, and that small cost of being wrong is the essential part.
• Keep the record. A written account of one problem solved under those conditions is better evidence, to an employer and to you, than a list of topics covered, because it is the only evidence of the fourth level rather than the third.

This is also what our sessions are for, and it is worth being exact about the claim. Reading, rebuilding and explaining you can do alone, and you should; nobody needs a classroom to build the model. What is awkward to arrange for yourself is a problem you did not pick, a session that ends whether or not you have finished, and other engineers who will hear your reasoning while you are still forming it and tell you where it went wrong.

Being wrong in front of somebody early costs little. The same error later costs a great deal, and by then nobody is watching closely enough to correct it.

Go back to the codec that appears on nine boots out of ten. Knowing the term, being able to state the rule, being able to derive the behaviour from the device tree, and being able to decide, in the time you actually have, which fix you are prepared to defend are four different positions. Only the last one is the job, and the honest question is not which of them you could reach if you studied — it is which one you were on the last time a board behaved like this.

— Raghu Bharadwaj