Hard debugging is rarely defeated by a lack of tools. It is defeated by three quiet habits: assuming the fault is where the symptom appears, clinging to the first explanation, and hoping a tool will do the thinking. A difficult fault is usually lost to those habits before you read a line of code. The engineers who resolve hard faults are the ones who notice these defaults and replace them with a patient, evidence-first mindset.
Most hard bugs are lost before we touch them, in the attitude we bring to the session.
When something breaks, the average person rushes in with three quiet habits: they assume the fault lives exactly where it shows up, they cling to the first explanation their mind offers, and they hope a tool or a smarter person will tell them what to do next. Those habits feel natural, but on hard faults they are exactly what keep us stuck.
Put two engineers on the same failing board. One finds a way through in an afternoon; the other is still going three days later. The difference is rarely raw intelligence or how many commands they know. It is the mental posture each brings to the work before the first step. Handling a hard debug session is less about knowing every tool and more about managing your own assumptions, reactions, and impatience.
A tough problem is usually lost in your mindset before it is lost in your methods.
Habit one: starting too narrow
The first habit is to fix on the most visible symptom and refuse to look anywhere else. Something breaks, so we stare at the last thing we changed, and we return to it because it is familiar and close at hand. When the answer is not there, we look harder in the same place instead of stepping back.
Here is what that looks like on real hardware. A device keeps dropping off the bus. You are a kernel person, so you open the driver and read it, carefully, for three days: the probe path, the error handling, the power-management callbacks. Every line is correct, and the device still fails. The fault was a layer below the entire time, in a marginal connector and a power rail that sagged under load. Reading C could not localise it, because the problem was not written in C.
The mindset shift is to start wide. A real system is a stack of layers: application, kernel, driver, board, power, and the signals on the wires. What you see on the surface is often just where a deeper fault finally shows itself. This is why David Agans opens Debugging: The 9 Indispensable Rules with the rule that comes before all the others: “Understand the System.” If your model of the system stops at the software, the layer that owns the fault is not in your model, and you cannot search a layer your model does not contain. Engineers who handle hard sessions well resist the pull toward the layer they know best; they pause, sketch the whole picture, and choose where to look.
Do not confuse where it hurts with where it started.
Habit two: trusting the first explanation
The second habit is to treat your first explanation as the truth. Within seconds, the mind produces a story: it must be this driver, this flag, this commit. From that moment it collects only the facts that support the story and quietly ignores the rest. You feel as though you are analysing, when you are often defending a guess.
“The first principle is that you must not fool yourself and you are the easiest person to fool.” – Richard Feynman
The mindset shift is humility. Treat the first theory as a possibility, not a verdict, and read all of the evidence the system already gave you (the full message, the log, the oops, the register that held the bad value) before you decide what you believe. Agans states this as a rule of its own, “Quit Thinking and Look”: observe the actual behaviour before you explain it. And when you do change something, change one thing at a time, so each result answers exactly one question; change several at once and you can no longer tell which change mattered.
Strong debuggers care more about being right in the end than feeling right in the moment.
Habit three: outsourcing the thinking
The third habit is modern and seductive: believing a tool can do the thinking for us. Stuck, we search, paste, prompt, or ask around, and treat the reply as a substitute for understanding. An assistant answers inside whichever layer you named; it never held a model of your board, your load, or your timing, so it cannot tell you that the layer you asked about was the wrong place to look. The reply may be fluent, fast, and confident, and a fluent explanation is still not a tested one.
“The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.” – Brian Kernighan
Tools are useful, sometimes indispensable, but they are at their best when they test a question you have already framed. A print statement, a trace, an oscilloscope, or an AI prompt can help you inspect; it cannot decide what deserves attention, which pattern matters, or which assumption to challenge next. That part remains yours.
A good tool answers questions. A good mindset decides which questions are worth asking.
The mindset that handles hard faults
These habits are not signs of a weak engineer. They are the natural defaults of any fast mind under pressure, and hard debugging is the practice of noticing those defaults and replacing them with better ones: patience instead of haste, curiosity instead of ego, observation instead of premature certainty. On the next hard fault, run a small, repeatable routine.
- Pause before acting and name the layers the failure passes through: application, kernel, driver, board, power, and signal.
- Read everything the system is already telling you (the full error, the log, the oops) before you form a theory.
- Name your current hypothesis about which layer owns the fault, but stay willing to discard it.
- Change one thing at a time, and ask one clear question with each change.
- Reach for the one instrument, or the one AI prompt, that tests that single hypothesis, and treat its answer as evidence about one layer, not a verdict on the whole system.
- Let evidence, not relief, tell you the fault is solved. A disappearing symptom is not the same as a fixed cause.
Tools will keep improving, and an assistant will keep producing quicker and more plausible answers. None of that removes the work at the centre of every hard session, which is human judgement: deciding what to believe, what to doubt, and what to test next. That judgement is what turns a chaotic failure into a manageable investigation, and it is the habit we build deliberately in TECH VEDA training: understand the system first, read the evidence before you commit to a theory, and use tools only to confirm a conclusion you have already reasoned your way toward. It is also what makes an engineer trustworthy when the easy answers fail.
– Raghu Bharadwaj
