The Power of Postmortems

Thu 13 June 2013

I’ve just had to write a postmortem for a problem that happened at work. The problem wasn’t too serious, but it was an avoidable issue. Nothing is a bigger bummer than when you see something that looks obvious in hindsight.

Fortunately, I work with a fantastic team, and so we are in the constant pursuit of doing better. And to do better, first you have to admit that you aren’t perfect, that the process isn’t perfect, and that from time-to-time mistakes will occur. When those problems surface, a postmortem is the best way to suck the marrow of experience from failure.

It’s absolutely critical to approach postmortems with the most objective viewpoint as you can - you need to know why something happened, and fix the core problem. Some companies don’t do postmortems because it flies in the face of cover-your-ass office politics. No one wants to admit they’ve done something wrong, or that they’ve contributed to a problem. Except I have yet to meet someone who hasn’t been part of some problem in some way.

What you’ll tend to find once you start doing postmortems is that it’s never one problem, it’s multiple smaller, often seemingly-benign problems that happen to combine into something much uglier. Seemingly simple things become incredibly under scrutiny.

There isn’t really a ‘perfect’ technique for a postmortem, but over time you will find a way of doing it that works most of the time. Templates are good, but some matters require more reflection than others. Here’s a few pointers that I’ve found that have helped me write pretty good postmortems:

  • Use roles instead of names. Disassociation when investigating is incredibly helpful in getting people to look at the matter with a more exacting, objective eye. Instead of ‘Bob’, you have ‘the incoming on-call person’, or instead of ‘Carol’, you have ‘head of development’. No one likes to be blamed, and if you work in a healthy environment, everyone wants to know how to deal with the problem next time. By using roles, everyone can learn from the experience and no one is left to hang. Using roles also future-proof’s the postmortem, making it valuable well after the event has occurred.
  • Use the ‘five whys’ to get at the root cause of the problem. The ‘five whys’ sometimes yields mundane causes, but also can reveal some really interesting gems that wouldn’t have been thought of without really digging into the problem at hand.
  • If you have a list of protocols, see if the problem was covered by the protocols. In our team we have a set of ‘commandments’, and should an issue not fall under the commandments, it allows for a discussion of how they can be strengthened to cover this edge case. So far, we haven’t seen a problem not covered by the initial set of commandments, but it’s nice to know that there is flexibility in them.
  • Have a list of actionable items that can be drawn from the matter. It’s almost never just one item you’ll find, and that’s fine. The best items are the ones that create permanent, positive change.
  • Have members of the team sign off on the post-mortem. Because time investigating and learning about failure is not time wasted.
  • No problem is too big or small to be investigated. Sometimes little things can become big things. By looking into how much time I waited for the kettle to get hot, I now save up to 10 minutes a day by keeping a 1-litre carafe of tea at my desk. It lets me have a hot cup of tea any time during the day, and means I’m not waiting by the kettle (I’ve timed how long it takes to boil, which is around 7 minutes - by knowing this and using my stopwatch it gives me 5 minutes to quickly check my email in the morning). 10 minutes a day doesn’t sound like much, but can add up to ~40 hours a year! A full work week a year saved by a $15 carafe, and I get hot tea whenever I want.

If people are worried that they’ll be blamed for problems, they will cover them up. This is the worst thing that can happen - it means the problem isn’t surfaced and can’t be prevented or dealt with more efficiently in the future.

If you happen to work in a hostile, political office (it happens), do private postmortems. Before long, you will be learning valuable lessons from your failures and the failures that happen around you, and work will never come easier. Failure is whatever you define it to be, so don’t be afraid to use this powerful analysis technique!