Delta CEO says CrowdStrike-Microsoft outage cost the airline $500 million

It worked but there was a $10 convenience fee.

They need to get a clue under their skin or else they get the hose again.

Is that why they call you hosers?

There's a good 20% of blame belonging to the penny pinchers choosing to allow third-party security updates without testing environments because the corporation is too cheap for proper infrastructure and disaster recovery architecture.

Like, imagine if there was a new airbag technology that promised to reduce car crashes. And so everyone stopped wearing seatbelts. And then those airbags caused every car on the road to crash at the same time.

Obviously, the airbags that caused all the crashes are the primary cause. And the car manufacturers that allowed airbags to crash their cars bear some responsibility. But then we should also remind everyone that seatbelts are important and we should all be wearing them. The people who did wear their seatbelts were probably fine.

Just because everyone is tightening IT budgets and buying licenses to panacea security services doesn't make it smart business.

The crowdstrike driver has the boot_critical flag set, which prevents exactly what you describe from happening

But... the BSOD!

Funny enough, crowdstrike on Linux had a very similar issue a few months back.

I guess microsoft-crowdstrike is fair, since the OS doesn't have any kind of protection against a shitty antivirus destroying it.

I keep seeing articles that just say "Microsoft outage", even on major outlets like CNN.

They're pissing away their operating system goodwill so fast.

They pissed it away {checks DoJ v. Microsoft} 25 years ago.

I don’t know that Microsoft has OS goodwill. People use it because the apps are there, not because Windows has a good user experience.

I think what I was hearing is that the CrowdStrike driver is WHQL approved, but the theory is that it's just a shell to execute code from the updates it downloads, thus effectively bypassing the WHQL approval process.

The driver is wqhl approved, but the update file was full of nulls and broke it.

Microsoft developed an api that would allow anti malware software to avoid being in ring 0, but the EU deemed it to be anti competitive and prohibited then from releasing it.

Delta was the only airline to suffer a long outage. That’s why I say Crowdstrike is the kickoff, but the poor, drawn-out response and time to resolve it is totally on Delta.

We just lost 500 million - we can't afford that right now! /s

According to The headhunters are constantly trying to recruit me for inappropriate jobs it is starting to get traction with companies and they are starting to actually hire fully skilled it departments. Opposed to the ones merely willing to work for near minimum wage which is what they had before.

In some ways it won't really make a difference because fully staffed up I.T departments also needs to be listened to by management, and that doesn't happen often in corporate environments, but still they'll pay the big bucks so that's good enough for me.

Fucking lol.

According to The headhunters are constantly trying to recruit me for inappropriate jobs it is starting to get traction with companies and they are starting to actually hire fully skilled it departments. Opposed to the ones merely willing to work for near minimum wage which is what they had before.

In some ways it won't really make a difference because fully staffed up I.T departments also needs to be listened to by management, and that doesn't happen often in corporate environments, but still they'll pay the big bucks so that's good enough for me.

All the servers and infrastructure should have "lights out management". I can turn on a server, reconfigure the bios and install windows from scratch on the other side of the world.

Potentially all the workstations / end point devices would need to be repaired though.

The initial day or two I'll happily blame on crowdstrike. After that, it's on their IT department for not having good DR plans.

There was no easy automated way if the systems were encrypted, which any sane organization mandates. So yes, did require hands-on-keyboard. But all the other airlines were up and running much faster, and they all had to perform the same fix.

Basically, in macOS terms, the OS fails to boot, so every system just goes to recovery only, and you need to manually enter the recovery lock and encryption password on every system to delete a file out of /System (which isn’t allowed in macOS because it’s read only but just go with it) before it will boot back into macOS. Hope you had those recorded/managed/backed up somewhere otherwise it’s a complete system reinstall…

So yeah, not fun for anyone involved.

I'm not sure any kind of pentest would prevent crowdstrikes backdoor access to release updates at its own discretion and cadence. The only way to avoid that would be blocking crowdstrike from accessing the Internet but I'd bet they'd 100% brick the host over letting that happen. If anything this is a good lesson in not installing malware to prevent even worse malware. You handed the keys to your security to a party that clearly doesn't care and paid the price. My reaction to that legal disclaimer of crowdstrikes stating they take no responsibility for anything they do... responsibility is the only reason anyone would buy anything from them (aside from being forced by legal requirements that clearly didn't have anyone who understood them involved in the legislation).

Yeah, I say I as migrate another service to Azure...

I think what @riskable@programming.dev was saying is you shouldn't have multiple mission critical systems all using the same 3rd party services. Have a mix of at least two, so if one 3rd party service goes down not everything goes down with it

If I were in charge I wouldn't put anything critical on Windows. Not only because it's total garbage from a security standpoint but it's also garbage from a stability standpoint. It's always had these sorts of problems and it always will because Microsoft absolutely refuses to break backwards compatibility and that's precisely what they'd have to do in order to move forward into the realm of, "modern OS". Things like NTFS and the way file locking works would need to go. Everything being executable by default would need to end and so, so much more low-level stuff that would break like everything.

Aside about stability: You just cannot keep Windows up and running for long before you have to reboot due to the way file locking works (nearly all updates can't apply until the process owning them "lets go", as it were and that process usually involves kernel stuff... due to security hacks they've added on since WinNT 3.5 LOL). You can't make it immutable. You can't lock it down in any effective way without disabling your ability to monitor it properly (e.g. with EDR tools). It just wasn't made for that... It's a desktop operating system. Meant for ONE user using it at a time (and one main application/service, really). Trying to turn it into a server that runs many processes simultaneously under different security contexts is just not what it was meant to do. The only reason why that kinda sort of works is because of hacks upon hacks upon hacks and very careful engineering around a seemingly endless array of stupid limitations that are a core part of the OS.

Adding another reply since I went on a bit of a rant in my other one... You're actually missing the point I was trying to make: No matter what solution you choose it's still your fault for choosing it. There are a zillion mitigations and "back up plans" that can be used when you feel like you have no choice but to use a dangerous 3rd party tool (e.g. one that installs kernel modules). Delta obviously didn't do any of that due diligence.

Alternatively, they could have taken Crowdstrike’s offer of layered rollouts, but Delta declined this and wanted all updates immediately to all devices.