If anything, this just shows how short-term our memory is. I imagine crowdstrike stock will be back to where it was by the end of next week.

I bet they don't even lose a meaningful amount of customers. Switching costs are too high.

A real shame, and a good reminder that we don't own the things we think we own.

> this just shows how short-term our memory is.

I've been out of IT proper for a while, so to me, I had to ask "the Russiagate guys are selling AV software now?"

I don't partake in the stock market these days, but this is the kind of event that you can make good money betting the price will come back up.

When a company makes major headlines for bad news like this investors almost always over react and drive the price too far down.

I dunno. The stock price will probably dead cat bounce, but this is the sort of thing that causes companies to spiral eventually.

They just made thousands of IT people physically visit machines to fix them. Then all the other IT people watched that happen globally. CTOs got angry emails from other C-levels and VPs. Real money was lost. Nobody is recommending this company for a while.

It may put a dent in Microsoft as splash damage.

>It may put a dent in Microsoft as splash damage.

I have a feeling that Microsoft's PR team will be able to navigate this successfully and Microsoft might even benefit from this incident as it tries to pull customers away from CrowdStrike Falcon and into its own EDR product -- Microsoft Defender for Endpoint.

My (very unprofessional) guess here is that investors in the near term will discount the company too heavily and the previously overvalued stock will blow past a realistic valuation and be priced too low for a little while. The software and company aren't going anywhere as far as I can tell, they have far too much marketshare and use of CrowdStrike is often a contractual obligation.

That said, I don't gamble against trading algorithms these days and am only guessing at what I think will happen. Anyone passing by, please don't take random online posts as financial advice.

After yesterday, CRWD is still up more than the S&P since the start of the year, and both are up insane amounts.

The stock market is unrelated to reality.

Honestly makes me angry, if we had a sense of justice in this world this would devastate them financially.

With a P/E of over 573? Doubt it will recover that fast.

Worth $3.7B, paid $148M in 2022.

Edited to add: I wonder what the economic fallout from this will be? 10x his monetary worth? 100x? (not trying to put a price on the people who will die because of the outage; for that he and everyone involved needs to go to jail)

Nothing at all.

He will be the guy that convinced the investors and stakeholders to pour more money into the company despite some world-wide incident.

He deserves at least 3x the pay.

PS: look at the stocks! They sank, and now they are gaining again value. People can't work, people die, flights get delayed/canceled because of their software.

Regarding the stock. I'm sure people are "buying the dip".

From an investing perspective, that's fairly foolish until the financial liability of the company has been assessed.

Time will tell whether it's foolish or not.

so much seems based on sentiment now, might not matter as much as it would have 15 years ago.

If you invest based on fundamentals and company finances, you probably haven't had many chances to buy any positions in the last decade. Stock prices are completely unhinged from company financial reports.

"This is just a demonstration about how critical our software is and how undervalued we are. If the whole world economy requires us to run, we should be worth more" /s

I am still waiting for someone saying how generative AI (= chatbots/copilot) would have solved this problem, or even "never let happen"...

Generative AI would have mandated QA as best practice and to limit corporate liability.

Yeah well, obviously they don't need AI for that.

"We have unit tests coverage of 99%".

Let's be honest. :)

>I run a software company that deploys antivirus software automatically to customers' computers and servers. What should I be aware of as part of this and what actions should I ensure occur?

I gave it this question and part of the steps included deploying to avoid peak hours to avoid potential disruptions (such as the weekend for businesses); including a way to detect deployment issues and easily rollback the software in case there is an issue; and testing the deployment in a controlled environment before rolling it out to all customers (this would include doing a small rollout before a larger one). All of these actions are very standard and all would have greatly helped in this situation.

Crowdstrike (Falcon) has all possible compliance certifications (they are PCI, FedRAMP, ISO-27001, etc.), and these often include following standard practices of SDLC, such as testing, deploying, incident management, etc. And they are much more detailed than the answer from AI.

My suspicion, as someone pointed out, is that they deployed this update as a small patch, maybe bypassing some controls and cutting some corners. I really wouldn't blame it on lack of automated tests, etc. Only crowdstrike people know how this thing possibly made it to production.

>What certifications should I have?

ChatGPT Obtaining relevant certifications can enhance the credibility of your antivirus software and reassure customers about its security, reliability, and compliance with industry standards. Here are some key certifications you should consider:

1. Security Certifications: ISO/IEC 27001: This certification specifies the requirements for establishing, implementing, maintaining, and continually improving an information security management system (ISMS). It demonstrates your commitment to managing and protecting sensitive information. ISO/IEC 15408 (Common Criteria): This certification provides assurance that the security features of your software have been rigorously tested and meet internationally recognized standards. 2. Quality Management Certifications: ISO 9001: This certification focuses on quality management systems and ensures that your processes consistently provide products that meet customer and regulatory requirements. ISO/IEC 25010: This standard specifies a model for evaluating software quality, ensuring that your software meets high standards of functionality, reliability, usability, efficiency, maintainability, and portability. 3. Data Privacy Certifications: GDPR Compliance: If you operate in or serve customers in the European Union, demonstrating compliance with the General Data Protection Regulation (GDPR) is essential. CCPA Compliance: For customers in California, compliance with the California Consumer Privacy Act (CCPA) is important. 4. Industry-Specific Certifications: HIPAA Compliance: If your software handles protected health information (PHI) in the United States, ensuring compliance with the Health Insurance Portability and Accountability Act (HIPAA) is crucial. PCI DSS: If your software processes payment card information, compliance with the Payment Card Industry Data Security Standard (PCI DSS) is necessary. 5. Product-Specific Certifications: ICSA Labs Certification: ICSA Labs, an independent division of Verizon, provides testing and certification for security products, including antivirus software. Certification from ICSA Labs can validate the effectiveness of your software. AV-TEST Certification: The AV-TEST Institute is an independent research institute that evaluates and certifies antivirus software for its protection, performance, and usability. An AV-TEST certification can enhance your product’s credibility. VB100 Certification: Virus Bulletin (VB) offers the VB100 certification, which is awarded to antivirus products that meet specific criteria for detecting known malware without generating false positives. OPSWAT Certification: OPSWAT offers certifications that validate the interoperability and effectiveness of cybersecurity products, including antivirus software. 6. Environmental and Occupational Health Certifications: ISO 14001: This certification demonstrates your commitment to environmental management, which can be important for corporate social responsibility (CSR) and sustainability. ISO 45001: This certification focuses on occupational health and safety management systems, ensuring a safe and healthy workplace for your employees.

"I asked ChatGPT how to prevent this and here's what it said. <generic bullet points about software testing> AI would have prevented this."

Every major outage when I worked at Google caused stock price to increase for this exact reason.

IT is always considered a cost until execs realize its critical to the company's existence. Keeping the lights on always seems to be undervalued. :(

You’re joking but I actually think this is part of how the CEO will frame things to investors.

Kurtz response is ridiculous blaming the customer on X. He will probably find another company to hire him as CEO tho. Just an upside down world in the C-suite world.

Don't forget the golden parachute. These guys always seem to fail upward.

That guy is gonna fail all the way right up to the top. Sheesh.

who is hiring these fucking idiots? they need to be blacklisted

Crowdstrike is run by humans just like you and me. One mistake doesn’t mean they are completely incompetent.

> One mistake doesn’t mean they are completely incompetent.

They are completely incompetent because for something as critical as crowdstrike code, you must build so many layers of validation that one, two or three mistakes don't matter because they will be caught before the code ends up in a customer system.

Looks like they have so little validation that one mistake (which is by itself totally normal) can end up bricking large parts of the economy without ever being caught. Which is neither normal nor competent.

Except this isn’t one mistake. Writing buggy code is a mistake. Not catching it in testing, QA, dogfooding or incremental rollouts is a complete institutional failure

Mistakes are perfectly fine, that's why multiple layers of testing exist

> Mistakes are perfectly fine, that's why multiple layers of testing exist

Indeed. Or in the case of crowdstrike, should exist. Which clearly doesn't for them.

The CTO with a shitty track record, not the line employees. He deserves zero reprieve

Reminds me of Phil Harrison who always seems to find himself in an of executive position, botching launches of new video game platforms - PlayStation 3, Xbox One, Google Stadia

CXOs usually have deep connection and great contracts (golden parachutes, etc.) that make them extremely difficult to fire and amiable to hire :)

He founded the company

I didn’t understand why in 2010, it didn’t seem to make most news…

Took out the entire company where I worked.

People thought it was a worm/virus — few minutes after plugging in laptop, McAfee got the DAT update, quarantined the file; which caused Windows to start countdown+reboot (leading to endless BSODs).

Yet another successful loser who somehow continues to ascend corporate ranks despite poor company performance. Just shows how disconnected job performance is from C-suite peer reviews, a glorified popularity contest. Should add the unity and better.com folk here

Eh. To be fair, the higher profile your job is, the more likely you'll be the face of one of these in your career.

Ok but he faced two

“There's an old saying in Tennessee — I know it's in Texas, probably in Tennessee — that says, fool me once, shame on — shame on you. Fool me — you can't get fooled again.”

- GWB

fool me once...