5mo ago

CrowdStrike downtime apparently caused by update that replaced a file with 42kb of zeroes

twiiit.com christian_taillon (@christian_tail)

You're correct. Full of zeros at least.

…according to a Twitter post by the Chief Informational Security Officer of Grand Canyon Education.

So, does anyone else find it odd that the file that caused everything CrowdStrike to freak out, C-00000291-
00000000-00000032.sys was 42KB of blank/null values, while the replacement file C-00000291-00000000-
00000.033.sys was 35KB and looked like a normal, if not obfuscated sys/.conf file?

Also, apparently CrowdStrike had at least 5 hours to work on the problem between the time it was discovered and the time it was fixed.

189 comments

Every affected company should be extremely thankful that this was an accidental bug, because if crowdstrike gets hacked, it means the bad actors could basically ransom I don't know how many millions of computers overnight
Not to mention that crowdstrike will now be a massive target from hackers trying to do exactly this
- Don't Google solar winds
  
  Holy hell
  
  Oooooooo this one again thank you for reminding me
  
  That one turns out to have been largely Microsoft's fault for repeatedly ignoring warnings of a severe vulnerability relating to Active Directory. Microsoft were warned about it, acknowledged it and ignored it for years until it got used in the Solar Winds hack.
  
  lesson not learned
- security as a service is about to cost the world a pretty penny.
  
  You mean it's going to cost corporations a pretty penny. Which means they'll pass those "costs of operation" on to the rest of us. Fuck.
  
  Where's my fuckin raise
  
  All the more reason for companies to ignore security until they're affected personally. The companies I've worked for barely ever invested in future cost-savings.
- On Monday I will once again be raising the point of not automatically updating software. Just because it's being updated does not mean it's better and does not mean we should be running it on production servers.
  Of course they won't listen to me but at least it's been brought up.
  
  Thank God someone else said it. I was constantly in an existential battle with IT at my last job when they were constantly forcing updates, many of which did actually break systems we rely on because Apple loves introducing breaking changes in OS updates (like completely fucking up how dynamic libraries work).
  Updates should be vetted. It's a pain in the ass to do because companies never provide an easy way to rollback, but this really should be standard practice.
  
  I thought it was a security definition download; as in, there's nothing short of not connecting to the Internet that you can do about it.
- I've got a feeling crowdstrike won't be as grand of target anymore. They're sure to lose a lot of clients...at least until they spin up a new name and erease all traces of "crowdstrike".
  
  I don't think they will lose any big clients. I am sure they will have insurance to take care of compensations.
  
  That trick doesn't work for B2B as organizations tend to do their research before buying. Consumers tend not to.
- Third parties being able to push updates to production machines without being tested first is giant red flag for me. We’re human … we fuck up. I understand that. But that’s why you test things first.
  I don’t trust myself without double checking, so why would we completely trust a third party so completely.
- I'd assume state (or other serious) actors already know about these companies.
- This is why I openly advocate for a diverse ecosystems of services, so not everyone is affected if the biggest gets targeted.
  But unfortunately, capitalism favors only the frontrunner and everyone else can go spin, and we aren't getting rid of capitalism anytime soon.
  So basically, it is inevitable that crowdstrike WILL be hacked, and the next time will be much much worse.
  
  Properly regulated capitalism breaks up monopolies so new players can enter the market. What you're seeing is dysfunctional capitalism - an economy of monopolies.
  
  Years ago I read an study about insurance companies and diversification of assets in Brazil. By regulation, an individual insurance company need to have a diversified investment portfolio, but the insurance market as a whole not. the diversification of every individual company sum, as a whole of all the insurance market, as an was exposed market, and the researchers found, iirc, like 3 banks that if they fail they can cause a chain reaction that would take out the entire insurance market.
  Don't know why, but your comment made me remind of that.
- Yeah the fact that this company calls it feature that they can push an update anytime without site level intervention is scary to me. If they ever did get compromised boom every device running their program suddenly has a kernel level malware essentially overnight.

Ah, a classic off by 43,008 zeroes error.
- If you listen closely, you can hear this file.

If I had to bet my money, a bad machine with corrupted memory pushed the file at a very final stage of the release.
The astonishing fact is that for a security software I would expect all files being verified against a signature (that would have prevented this issue and some kinds of attacks
- So here's my uneducated question: Don't huge software companies like this usually do updates in "rollouts" to a small portion of users (companies) at a time?
  
  I mean yes, but one of the issuess with "state of the art av" is they are trying to roll out updates faster than bad actors can push out code to exploit discovered vulnerabilities.
  The code/config/software push may have worked on some test systems but MS is always changing things too.
  
  the smart ones probably do
  
  Companies don't like to be beta testers. Apparently the solution is to just not test anything and call it production ready.
  
  That's certainly what we do in my workplace. Shocked that they don't.
  
  When I worked at a different enterprise IT company, we published updates like this to our customers and strongly recommended they all have a dedicated pool of canary machines to test the update in their own environment first.
  I wonder if CRWD advised their customers to do the same, or soft-pedaled the practice because it’s an admission there could be bugs in the updates.
  I know the suggestion of keeping a stage environment was off putting to smaller customers.
- Which is still unacceptable.
- Windows kernel drivers are signed by Microsoft. They must have rubber stamped this for this to go through, though.
  
  This was not the driver, it was a config file or something read by the driver. Now having a driver in kernel space depending on a config on a regular path is another fuck up
  
  What about the Mac and Linux PCs? Did Microsoft sign those too?
- From my experience it was more likely to be an accidental overwrite from human error with recent policy changes that removed vetting steps.
- Which is still unacceptable.

This file compresses so well. 🤏

If it had been all ones this could have been avoided.
- Just needed to add 42k of ones to balance the data. Everyone knows that, like tires, you need to balance your data.
  
  Apply the ones in a star shape to distribute pressure evenly
  
  I mean, joking aside, isn't that how parity calculations used to work? "Got more uppy bits than downy bits - that's a paddlin'" or something.
  
  I once had a very noisy computer.... It was the disk! I just balanced the data and all problems went away.
  
  Perfectly balanced

undefined

d'00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000!
- Damnit you're comment just crashed the rest of the computers that were still up.
- thank you for the visual representation ☺️

Imagine the world if those companies were using Atomic distribution and the only thing you would need to do is to boot the previous good image.
- ohno_anyway.png

The fact that a single bad file can cause a kernel panic like this tells you everything you need to know about using this kind of integrated security product. Crowdstrike is apparently a rootkit, and windows apparently has zero execution integrity.
- I’m not sure why you think this statement is so profound.
  CrowdStrike is expected to have kernel level access to operate correctly. Kernel level exceptions cause these types of errors.
  Windows handles exceptions just fine when code is run in user space.
  This is how nearly all computers operate.
- This is a pretty hot take. A single bad file can topple pretty much any operating system depending on what the file is. That's part of why it's important to be able to detect file corruption in a mission critical system.
  
  This was a binary configuration file of some sort though?
  Something along the lines of:
  undefined
  
  IF (config.parameter.read == garbage) { Dont_panic; }
  Would have helped greatly here.
  Edit: oh it's more like an unsigned binary blob that gets downloaded and directly executed. What could possibly go wrong with that approach?
- Security products of this nature need to be tight with the kernel in order to actually be effective (and prevent actual rootkits).
  That said, the old mantra of "with great power" comes to mind...
  
  with great power, don't lay off the testing team (force return to office or get fired ultimatums)
- Yeah pretty much all security products need kernel level access unfortunately. The Linux ones including crowdstrike and also the Open Source tools SELinux and AppArmor all need some kind of kernel module in order to work.
  
  crowdstrike has caused issues like this with linux systems in the past, but sounds like they have now moved to eBPF user mode by default (I don't know enough about low level linux to understand that though haha), and it now can't crash the whole computer. source
  
  At least SELinux doesn't crash on bad config file
- Does anything running on an x86 processor from the last decade?
- Tell me you don't understand what it is by telling me you don't understand what Crowdstrike does.
- I mean are we surprised by any of this?
- I don't remember much about my OS courses from 20 years back, but I do recall something about walls between user space and kernel space. The fact that an update from the Internet could enter kernel space is insane to me.
  
  You mean you don’t update your kernel?

have they ruled out any possibility of a man in the middle attack by a foreign actor?
- This was not a cyberattack.
  https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
  I guess they could be lying, but if they were lying, I don’t know if their argument of “we’re incompetent” is instilling more trust in them.
  
  "We are confident that only our engineers can fuck up so much, instead of our competitors"
- The CEO made a statement to the effect of "It's not an attack, it's just me and my company being shockingly incompetent." He didn't use exactly those words but that was the gist.
- Or it being an intentional proof of concept
- In the middle of the download path of all the machines that got the update?
- Foreign to who?
  
  "Foreign" in this context just means "not Crowdstrike", not like a foreign government.

How can all of those zeroes cause a major OS crash?
- If I send you on stage at the Olympic Games opening ceremony with a sealed envelope
  And I say "This contains your script, just open it and read it"
  And then when you open it, the script is blank
  You're gonna freak out
  
  Ah, makes sense. I guess a driver would completely freak out if that file gave no instructions and was just like "..."
  
  Except "freak out" could have various manifestations.
  In this case it was "burn down the venue".
  It should have been "I'm sorry, there's been an issue, let's move on to the next speaker"
  
  Great layman's explanation.
  
  Maybe. But I'd like to think I'd just say something clever like, "says here that this year the pummel horse will be replaced by yours truly!"
  
  I'm nominating this for the "best metaphor of the day" award.
  Well done!
  
  Nice analogy, except you'd check the script before you tried to use it. Computers are really good at crc/hash checking files to verify their integrity, and that's exactly what a privileged process like antivirus should do with every source of information.
  
  The funny bit is, I'm sure more than a few people at Crowdstrike are preparing 3 envelopes right now.
  
  This guy ELI5s
  
  Ah yes. So Windows is the screaming in terror version and other systems are the "oh, sorry everyone, looks like there's an error. Let's just move on to the next bit" version.
- Because it's supposed to be something else
  
  At least a few 1's I imagine.
  
  Well, you see, the front fell off.
- The file is used to store values to use as denominators on some divisions down the process. Being all zeros is caused a division by zero erro. Pretty rookie mistake, you should do IFERROR(;0) when using divisions to avoid that.
  
  I disagree. I'd rather things crash than silently succeed or change the computation. They should have done better input and output validation, and gracefully fail into a recoverable state that sends a message to an admin to correct. A divide by zero doesn't crash a system, it's a recoverable error they should 100% detect and handle, hot sweep under the rug.
  
  IFERROR(;0)
  Maybe they should use a more appropriate development tool for their critical security platform than Excel.
- Well, the file shouldn't be zeroes
  
  The front of the file fell off
- Windows

I'm not a dev, but don't they have like a/b updates or at least test their updates in a sandbox before releasing them?
- It could have been the release process itself that was bugged. The actual update that was supposed to go out was tested and worked, then the upload was corrupted/failed. They need to add tests on the actual released version instead of a local copy.
  
  Could also be that the Windows versions they tested on weren't as problematic as the updated drivers around the time they released.
- one would think. apparently the world is their sandbox.

school districts were also affected.. at least mine was.
- I can't imagine how much worse this would have been for global GDP if schools had to be closed for it.

I wonder how many governments and companies will take this as a lesson on why brittle systems suck. My guess is most of them won't... It's popular to rely on very large third party services, which makes this type of incident inevitable.
- oh they'll take it as a lesson all right, up until they get the quote to fix it suddenly the downtime becomes non-issue as long as it "doesn't happen again"

189 comments