Delta Sues CrowdStrike Over Software Update That Prompted Mass Flight Disruptions (reuters.com) 40
An anonymous reader quotes a report from Reuters: Delta Air Lines on Friday sued cybersecurity firm CrowdStrike in a Georgia state court after a global outage in July caused mass flight cancellations, disrupted travel plans of 1.3 million customers and cost the carrier more than $500 million. Delta's lawsuit filed in Fulton County Superior Court called the faulty software update from CrowdStrike "catastrophic" and said the firm "forced untested and faulty updates to its customers, causing more than 8.5 million Microsoft Windows-based computers around the world to crash." [...]
Delta, which has purchased CrowdStrike products since 2022, said the outage forced it to cancel 7,000 flights, impacting 1.3 million passengers over five days. "If CrowdStrike had tested the faulty update on even one computer before deployment, the computer would have crashed," Delta's lawsuit says. "Because the faulty update could not be removed remotely, CrowdStrike crippled Delta's business and created immense delays for Delta customers." Delta said that as part of its IT-planning and infrastructure, it has invested billions of dollars "in licensing and building some of the best technology solutions in the airline industry."
Delta, which has purchased CrowdStrike products since 2022, said the outage forced it to cancel 7,000 flights, impacting 1.3 million passengers over five days. "If CrowdStrike had tested the faulty update on even one computer before deployment, the computer would have crashed," Delta's lawsuit says. "Because the faulty update could not be removed remotely, CrowdStrike crippled Delta's business and created immense delays for Delta customers." Delta said that as part of its IT-planning and infrastructure, it has invested billions of dollars "in licensing and building some of the best technology solutions in the airline industry."
The should have read the fine print (Score:1)
Damages never more than the software costs.
Re: The should have read the fine print (Score:2)
These exclusions donâ(TM)t apply to things like gross negligence or fraud, and whether or not the event was one of those is a question of fact for a jury to decide.
Re: (Score:2)
Delta is demanding from its contractor reparations for damages that it's refusing its customers. When they reject claims for which they have responsibility, how can they go after their contractors on the same basis?
Delta is not the only airline to use CrowdStrike, and they all had outages on July 19. Delta, however, is the only one that couldn't recover until July 24.
Crowdstrike's test and deployment processes to me look like gross negligence: their business is having companies ent
Re: The should have read the fine print (Score:2)
The process that delta uses may not be relevant. What matters is what Crowdstrike said its processes are, what the contract between delta and Crowdstrike says, and what a reasonable person would do based on those assurances. Crowdstrike publicly admitted that it did not test the update. If the contract between delta and Crowdstrike says that Crowdstrike shall test all updates before deployment on infrastructure that duplicates delta infrastructure because delta doesnt want to do that testing itself, Crowdst
Re: (Score:2)
I didn't follow this one beyond the basics. Did crowdstrike really admit in public or to Delta lawyers that they didn't test before releasing??
Re: (Score:2)
Re: (Score:2)
Does the contract really have to say that Corowdstrite must test updates?
Isn't that just assumed?
I mean, isn't it just basic software development practice to actually test software before you release it?
Re: (Score:2)
Delta is not the only airline to use CrowdStrike, and they all had outages on July 19. Delta, however, is the only one that couldn't recover until July 24.
That's one of the strongest parts of the defense against the bulk of the lawsuit.
The hours immediately after the issue? Absolutely the company is on the hook there, and their insurance is already paying out a fortune for those claims. No question about the liability there.
But after that? While the core systems were down for hours airlines had to move things around. All the other airlines were out for about an additional day to reposition their flight crews and aircraft. If you want to look at what is rea
The Person I Would Hate to Be (Score:3, Interesting)
Re: (Score:2)
It's possible that the defendant's lawyers may try to cast shadow on the coder, but the court is likely to consider all aspects of the event, including very poor processes by the company at large. After all, any revision and release processes need to take into account problematic or dangerous updates, and so while you might be able to blame a coder or a code group of buggering it up, the update was green lit, suggesting there are severe systemic issues.
also the lack of update control for that part that (Score:2)
also the lack of update control for that part that crashed.
Re:The Person I Would Hate to Be (Score:4, Insightful)
Re: (Score:2)
I imagine their name will come out at some point during the litigation of this case, and at that point their career and any semblance of a decent life will be over.
Only truly stupid companies wouldn't hire the person. A person is incapable of making a mistake like this. A mistake like this requires a fundamental lack of quality control systems and processes. There is a name you can directly blame. George Kurtz, CEO and Founder of a company who demonstrated that they don't take any basic QA/QC precautions.
Re: (Score:2)
"I would not like to be the person who authored the bad patch at CrowdStrike."
I would not like to be the person who authored a parser with no input validation,
I would not like to be the person who put code with no error handling in an environment where an error can bring down the entire machine.
There are many things that had to go wrong, and a bad patch is neither necessary nor sufficient to cause this scenario.
Yes, test it - before putting it into service. (Score:2, Insightful)
I'm talking to you, Delta.
Re:Yes, test it - before putting it into service. (Score:5, Informative)
IIRC part of this issue was that Clownstroke does not let you do controlled updates like that.
Re: (Score:2)
https://www.crn.com/news/secur... [crn.com]
Re: (Score:1)
IIRC part of this issue was that Clownstroke does not let you do controlled updates like that.
That just pushes the fault back a step: Delta should never have gotten into such an arrangement.
Re:Yes, test it - before putting it into service. (Score:4, Informative)
I'm talking to you, Delta.
That's not how Crowdstrike channel updates work. Not for Delta not for anyone else. That said there was someone who should have tested something before putting it in service, but you're addressing the wrong side of the court room for this.
Valid if it was just Delta that shit itself. (Score:2)
Re: (Score:2)
Re: (Score:2)
Everyone should be testing before deploying. That fact that it hit so many in modern times shows up the bad practice creeping in.
Re: (Score:3)
https://www.crn.com/news/secur... [crn.com]
Bad practices that vendors force on you.
Re: (Score:2)
EULA (Score:2)
One could guess that the EULA says the software does not have to work and you use it at your own risk.
https://www.crowdstrike.com/en... [crowdstrike.com]
Re: (Score:3)
Re: (Score:2)
A Contact you can't negotiate is called a "contract of adhesion"
"For a contract to be treated as a contract of adhesion, it must be presented on a standard form on a "take it or leave it" basis, and give one party no ability to negotiate because of their unequal bargaining position."
https://en.wikipedia.org/wiki/Standard_form_contract#Contracts_of_adhesion
This is what we see for most consumer contracts.
That said, if two companies are large enough they will negotiate a custom contract. Usually the largest co
Re: (Score:2)
6. No Warranty.
6.1 Disclaimer. THE SOFTWARE AND ALL OTHER CROWDSTRIKE OFFERINGS ARE PROVIDED “AS-IS” AND WITHOUT WARRANTY OF ANY KIND. CROWDSTRIKE AND ITS AFFILIATES DISCLAIM ALL OTHER WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE.
Sounds fairly straight forward. Did Delta not read the agreement?
Re: (Score:2)
> Sounds fairly straight forward
That's not how the law works.
Unconscionable terms and acts of fraud and gross negligence are frequently found to be outside of contract terms.
It's far from straightforward. Partially self-serving on the part of lawyers.
Make Microsoft accountable too. (Score:2)
Re: (Score:2)
Forced reboots are why many people use unsupported operating systems as they no longer reboot for updates disrupting workflows.
Microsoft doesn't force any reboots. Microsoft displays a warning on the screen that during the next reboot or shutdown an update will apply. Microsoft provides an offline timer to apply updates out of hours when the PC isn't use. Microsoft provide APIs that allow state recovery on reboot, as well as APIs to lock out reboots preventing work being lost.
If you are losing work you definitely have someone to blame, but it's not Microsoft. Start with your own IT department (yes ours forces reboots on our machine
Re: (Score:2)
If your systems are so fragile you can't reboot for an update without going offline you need to look more closely at your systems.
That's called a "single point of failure" and is considered a "bad thing".
At early startups, sure, we crossed our fingers and had plenty of outages but as the companies grew and matured we eliminated every spof we knew about or stumbled into the hard way.
Why are mature companies running known spof critical systems?
Re: (Score:2)
Of course, Crowdstrike does not have that kind of money lying around
Crowdstrike is worth $75billion. Not only do they have $4bn in free cash laying around, they could very easily raise more capital.
Re: (Score:2)
Their annual recurring (I.E. subscription) revenue in 2023 was almost 3.5 billion, with a gross profit margin of 75-80% on that income.
They could (maybe with indigestion) stomach a 5.5 billion loss after several years similar to 2023 in recent history.
backups (Score:2)
How did that Clownstrike failure sneak into Delta's backups/snapshots? Asking for a friend.
Problem is lack of transparency? (Score:2)
Maybe the worst thing is that CS's practices came AS A COMPLETE SURPRISE to everyone,
How many other companies out there are you currently relying on? What are their practices?
Delta, like many, took a decision to trust CS and whatever claims CS made about their products. Well, trust works for the good guys. But it's the bad guys that are the problem. And as the old Marx Brothers joke goes, if you can fake that then you've got it made. The bad companies are deceiving you into blindly trusting them.
The only an
Both at fault (Score:2)
CrowdStrike was at fault, but Delta owns the blame here. Delta had no way to recover in a timely manner; Delta didn't demand a Test QA before promoting changes to Prod. A $5 "break glass" USB could have been in a ziplock-type bag attached to each PC to boot to a base image and get back online; there are dozens of remote management recovery solutions available.
Delaware? (Score:2)