From Nobody Wants to Die - a detective replaying a reconstruction of what actually happened. The machine rebuilds the evidence; deciding what it means is still his problem.
Yesterday I spent most of an afternoon chasing the wrong root cause, with total, semi-earned confidence, straight past the evidence pointing at the right one.
The bug came in looking like pollution of a global namespace. And my brain took one look at it and said: I know this one. It had a shape I recognised - something is overwriting it, and it's probably a race condition. So off I went to prove it. Not to investigate it, to prove it.
So I did the proving. I followed execution, I added debug logs. And somewhere in the middle of all that, the actual answer scrolled past me. I could visibly see a piece of the system writing to that key, overwriting it. I looked directly at it.
I noticed it, but it had no meaning
That's the sentence that's rattling around in my noggin.
I didn't miss the contradictory evidence. Missing it would almost be fine - screens full of noise, nobody reads everything. That's not exactly what happened.
I noticed it. But it had no meaning. I didn't apply the meaning to it.
My eyes did their job, and the information physically arrived. But my brain - so busy confirming the thing it had already decided - filed it under irrelevant without consulting me. Because I was very confident, over my years of debugging and pattern matching, I simply did not register it. The root cause turned out to be a piece of initialisation logic that clobbered the namespace and the last one to run, won. Looking back now the contradictory evidence was worryingly obvious. I was convinced it was one thing, and there it was the whole time, saying: hi I'm over here.
The prediction engine
People describe the brain as a prediction engine, and I find this really, really interesting - not the description itself so much as what we all do with it. You make a prediction, then you attempt to match your reality to that prediction. And whether things turn out worse, or better, or simply different, what actually matters is how you use that information for future predictions in similar patterned situations. That's more or less the job of having a head - what can I remember from a previous similar situation that will keep me safe in this new situation.
I've written before about brains as prediction engines, but that post was about AI - what machines can't bring to the table. This one is about the engine misfiring inside your own skull, in the one activity where I'd argue the loop runs at its purest: debugging. Your brain will see an issue or a bug and will start looking for the root cause, and it makes subconscious predictions about what it's likely to be. Before you've opened a single log. Before you've decided to have a theory, you inherit a theory - you don't get a say in that part, which is the detail I think most debugging advice subtly ignores.
Pattern signatures
Here's the mechanism, as best I can describe it from the inside. Your brain creates a pattern signature of an error - this shape of stack trace, this specific smell of "works locally, dies in prod" - and it maps that signature back to a likely root cause based on your own experience. And then you simply go prove that. For the most part, you are proven correct - which only reinforces the pattern recognition. The signature sharpens, and next time is faster. This is what experience is, mechanically.
The less experienced you are, the less predictions you have. Every bug is a slow, humbling search - everything checked because nothing can be ruled out. The more experienced you are, the more predictions you have, and that can be a help - a great help - but sometimes a hindrance. Because when you become very experienced, you start to predict very confidently, and you will actually miss certain flags, certain disconfirming evidence, on the way to a different root cause. Confirmation bias can really scupper you when trying to find an issue, and the cruel bit is that it scuppers you harder the better your priors are - a junior's wrong guess collapses in ten minutes because they hold it loosely; my wrong guess yesterday survived the whole afternoon because it arrived donning the full bravado of every previous time it had been right.
Have we seen this shape before?
I believe in this mechanism enough that I've been building it. Norrin is a small TypeScript module that takes distributed traces and reduces them to fingerprints of what happened - stripping out the trace ids, the timestamps, the hostnames, the jitter, all the stuff that makes two occurrences of the same incident look superficially different - so that when the same shape turns up again, you get "we've seen this before, 91% match" instead of relying on whoever was on call last time having a good memory. It's the pattern signature from the section above, taken out of my head and put somewhere the whole team can see it. (Your brain files these signatures under "experience" and then charges you a decade for access. The module is free.)
And building it is what handed me the more acute version of yesterday's lesson, because Norrin's half of the job is just maths: it computes the signature and reports the match. A 91% match doesn't make it stop reading the other 9%. Whereas I do - did - because the moment my internal engine said "race condition", the match stopped being a hypothesis and became the "answer". The tool can't apply meaning at all, and honestly that's what saves it: a match is a prediction, not a verdict. The meaning bit - deciding what a match is evidence of, and noticing what it isn't - stays stubbornly human.
Budgeting for what doesn't fit
The obvious moral is "trust your instincts less", but I don't fully buy that. The prediction engine is the single most valuable thing experience builds; deliberately dulling it would be like binding your own nervous system whilst it's trying to do its job. No - prove your predictions. That's what they're for. Go at the likely root cause first, and go fast. You'll be right most of the time, and being right fast is the job.
But budget a little attention for what doesn't fit. Two things I'm actually going to try, rather than just feel wise about: say the prediction out loud - in the incident channel, in a comment, to the duck - because a spoken prediction becomes a hypothesis, and a silent one stays a fact you're merely confirming. And every so often, mid-chase, ask: what have I seen in the last ten minutes that I haven't explained? Not what have I found but crudely what have I seen. Because yesterday's evidence was never hiding. It didn't need discovering. It needed meaning, and I was the only one in the room who could have applied it, and I was busy.
The noticing and the meaning arrived a day apart this time. I'd like to get that gap down. No promises.
TL;DR
- The brain is a prediction engine: it guesses, matches reality to the guess, and feeds the result into future predictions for similar patterned situations. Debugging is this loop at its purest.
- Experience builds pattern signatures: error → likely root cause → go prove it. You're mostly right, which reinforces the pattern.
- The failure mode of expertise isn't wrong predictions - it's confident ones. Disconfirming evidence gets noticed but not registered. It has no meaning until you apply the meaning.
- Don't trust your instincts less. Prove your prediction - but say it out loud so it stays a hypothesis, and keep asking: what have I seen that I haven't explained?
- I've been building Norrin to do the signature half of this honestly: it fingerprints incident shapes, immune to ids, timestamps and jitter - and it can't get attached to a match. A match is a prediction, not a verdict.