Everytime i look into such catastrophic issues, it always boils down to lack of robust canarying mechanisms.

They have enough client base that they can even run an A/B test on the whole binary level, but no.

Also, why not have some sort of graceful degradation (well kind of), like: OS Boots, loads CS driver, the driver loads some new feature/config, and before/after new recent thing ("runtime flag") marked whether it successfully worked, and if not on the next reboot that thing gets either disabled, or the previous known good config (obviously some combination of things might cause another issue), but instead of blindly rebooting to the same state....

I think pfsense does this (from memory, been a while using it). Basically dual-partitions, and if it failed to come up on the active partition after an update it'd revert. Granted you need to have the space to have two partitions, but for a small partition/image not so bad.

What surprises me is if its a content update, and the code fell over when dealing with it - just basically bad release engineering isn't it not to cater for that in the first place? i.e. some tests in the pipeline before releasing the content update would've picked it up given it sounds like 100% failure rates.

The problem space kind of dictates that this couldn't be a solution, cause malware could load an arbitrary feature/config and mark it as 0, then the AV would be disabled on next boot, right?

fair point indeed!

Why put effort in engineering when you can just fear monger in marketing and buy politicians in sales?

More importantly, why are CS customers not validating? Upstream patches should be treated as faulty/malicious if not tested to show otherwise, especially if they're kernel level.