The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.
Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.
I agree that there's an awkward emphasis on how FB prioritizes security and privacy but nothing is deceptive here. Had the audit bug not subsequently cut off access to internal tools and remote regions it would be easy to revert. Had there not been a global outage nobody would have known that the process for getting access in an emergency was too slow.
Huge events like this always have many factors that have to line up just right. To insist that the one and only true cause was a bug in the auditing system is reductive.
Our postmortems have three sections. Prevention, detection, and mitigation. They all matter.
Shit happens. People ship bugs. People fat-finger commands. An engineering team’s responsibility doesn’t stop there. It also needs to quickly activate responders who know what to do and have the tools & access to fix it. Sometimes the conditions that created the issue are within acceptable bounds; the real need for reform is in why it took so long to fix.
No, they just wanted to cover both "what caused it?" and "why did it take too long to fix it?" since both are topics people were obviously extremely interested in.
It would have been surprising and disappointing if they didn't cover both of them.
Seems like appropriate emphasis given how many people yesterday were asking why aren't they back online yet. For every person asking why they deleted their routes there were two people asking why they didn't put them back.
