Delta Sues CrowdStrike Over Software Update That Prompted Mass Flight Disruptions (reuters.com) 78
An anonymous reader quotes a report from Reuters: Delta Air Lines on Friday sued cybersecurity firm CrowdStrike in a Georgia state court after a global outage in July caused mass flight cancellations, disrupted travel plans of 1.3 million customers and cost the carrier more than $500 million. Delta's lawsuit filed in Fulton County Superior Court called the faulty software update from CrowdStrike "catastrophic" and said the firm "forced untested and faulty updates to its customers, causing more than 8.5 million Microsoft Windows-based computers around the world to crash." [...]
Delta, which has purchased CrowdStrike products since 2022, said the outage forced it to cancel 7,000 flights, impacting 1.3 million passengers over five days. "If CrowdStrike had tested the faulty update on even one computer before deployment, the computer would have crashed," Delta's lawsuit says. "Because the faulty update could not be removed remotely, CrowdStrike crippled Delta's business and created immense delays for Delta customers." Delta said that as part of its IT-planning and infrastructure, it has invested billions of dollars "in licensing and building some of the best technology solutions in the airline industry."
Delta, which has purchased CrowdStrike products since 2022, said the outage forced it to cancel 7,000 flights, impacting 1.3 million passengers over five days. "If CrowdStrike had tested the faulty update on even one computer before deployment, the computer would have crashed," Delta's lawsuit says. "Because the faulty update could not be removed remotely, CrowdStrike crippled Delta's business and created immense delays for Delta customers." Delta said that as part of its IT-planning and infrastructure, it has invested billions of dollars "in licensing and building some of the best technology solutions in the airline industry."
The should have read the fine print (Score:2, Flamebait)
Damages never more than the software costs.
Re: The should have read the fine print (Score:3)
These exclusions donâ(TM)t apply to things like gross negligence or fraud, and whether or not the event was one of those is a question of fact for a jury to decide.
Re: The should have read the fine print (Score:5, Interesting)
Delta is demanding from its contractor reparations for damages that it's refusing its customers. When they reject claims for which they have responsibility, how can they go after their contractors on the same basis?
Delta is not the only airline to use CrowdStrike, and they all had outages on July 19. Delta, however, is the only one that couldn't recover until July 24.
Crowdstrike's test and deployment processes to me look like gross negligence: their business is having companies entrust them with access to the Kernal and deploying timely and safe updates. Everyone else uses a robust testing process including staggered deployment.
But Delta, in going about a lawsuit, will be required to reveal their own IT processes and shortcomings that led to a five-day collapse.
Or they settle for zero dollars.
Re: The should have read the fine print (Score:4, Interesting)
The process that delta uses may not be relevant. What matters is what Crowdstrike said its processes are, what the contract between delta and Crowdstrike says, and what a reasonable person would do based on those assurances. Crowdstrike publicly admitted that it did not test the update. If the contract between delta and Crowdstrike says that Crowdstrike shall test all updates before deployment on infrastructure that duplicates delta infrastructure because delta doesnt want to do that testing itself, Crowdstrike is cooked.
Re: (Score:2)
I didn't follow this one beyond the basics. Did crowdstrike really admit in public or to Delta lawyers that they didn't test before releasing??
Re: (Score:2)
Re: The should have read the fine print (Score:1)
The software was not tested as released.
They have extensive testing, and tested both the software and data file before it was released.
It failed however because they didn't test these as part of the distributed binary package.
The tests running against the software were basically not using the latest data file. Same for the data file, which was tested against software different from the release, and older version.
In testing, because the way they tested - they never actually activated the code path that caus
Re: (Score:3)
> I've been in meetings where the decision to do functional or full end to end testing is deemed not worth while because there are all these wonderful tests for the individual components.
We were in the same meeting.
The process went like this:
1) "we don't need a QA team, the devs test their own work" followed by serious outage and data loss
2) "we don't need end to end testing, QA has 762 unit tests" followed by serious outage and data loss
3) "we are going to skip end to end testing on this release, $impor
Re: (Score:2)
Important systems should have redundant backup/restore options.
Where I worked (now retired) as sysadmin, every system had two boot disks. If a system was patched and failed on reboot for any reason, the system was booted off the unpatched second disk. Now if the second disk failed because lightning struck the system we would boot to the unpatched hot spare system which also had duel boot disks. We also had multiple levels of remote backup to restore every system.
I don't know at what level Delta made their d
Re: (Score:2)
Re: (Score:2)
Does the contract really have to say that Corowdstrite must test updates?
Isn't that just assumed?
I mean, isn't it just basic software development practice to actually test software before you release it?
Re: The should have read the fine print (Score:5, Interesting)
Delta is not the only airline to use CrowdStrike, and they all had outages on July 19. Delta, however, is the only one that couldn't recover until July 24.
That's one of the strongest parts of the defense against the bulk of the lawsuit.
The hours immediately after the issue? Absolutely the company is on the hook there, and their insurance is already paying out a fortune for those claims. No question about the liability there.
But after that? While the core systems were down for hours airlines had to move things around. All the other airlines were out for about an additional day to reposition their flight crews and aircraft. If you want to look at what is reasonable, look to the rest of the industry. The fix for boot-looping computers was available within a couple hours, people were fixing the file, and while it was annoying to manually deploy everybody else had their core systems done in a day or two. The first day CrowdStrike should absolutely be liable. The second day it becomes a sliding window, as the other companies were able to restore operations, it was a race that says more about the IT department of the company and less about CrowdStrike's deployment.
By the end of day two it's hard to see much liability for CrowdStrike.
The flip side of the coin is Delta's own policies for people affected by their shutdown. They're demanding from CrowdStrike what they refuse to give their own customers for Delta's part of the outage. While that's not evidence toward CrowdStrike liability, it is relevant toward actual damages Delta faced and therefore damages CrowdStrike may have to pay.
I would absolutely love it if the damages were issued in the form of a voucher for future CrowdStrike purchases, the same thing Delta offers so many of its customers after Delta damages flyer's property and costs fortunes in changed travel plans. "Yes CrowdStrike is at fault, and we apologize for any plans that may have been affected. Here's a voucher towards future CrowdStrike goods and services that must be used within 365 days. Thank you for using Falcon."
Re: (Score:2)
I don't know why they decided to reject those claims, but I'm pretty sure I know why they're going after Cloudstrike this way. The suit is being put together by lawyers and they're going to start off demanding everything they can think of so that they can use the more outlandish claims as bargaining chips.
Re: (Score:2)
Lawyers dont just come at these things swinging, they are going to have a theory of litigation and theres almost certainly been a trail of backroom negotiation that have fallen apart and now the lawyers are going to start taking skulls. If that theory of litigation is not structured around failing processes than Crowdstrike wont really have any reasonable grounds to start demanding to know about that.
Regardless, an organization as big as Delta is very likely to be following pretty standard ISO change manage
Re:The should have read the fine print (Score:4, Interesting)
Crowdstrike committed negligence that could not be more gross. In that case, all contractual limitations go out the window. You have to at least get the very basics right when you sell a product or service. Crowdstrike did not.
Re: (Score:2)
Major corporations don't sign or agree to standard EULAs when they purchase software. They always negotiate the terms.
The Person I Would Hate to Be (Score:4, Interesting)
Re: (Score:2)
It's possible that the defendant's lawyers may try to cast shadow on the coder, but the court is likely to consider all aspects of the event, including very poor processes by the company at large. After all, any revision and release processes need to take into account problematic or dangerous updates, and so while you might be able to blame a coder or a code group of buggering it up, the update was green lit, suggesting there are severe systemic issues.
also the lack of update control for that part that (Score:2)
also the lack of update control for that part that crashed.
Re:The Person I Would Hate to Be (Score:5, Insightful)
Re: (Score:2)
I imagine their name will come out at some point during the litigation of this case, and at that point their career and any semblance of a decent life will be over.
Only truly stupid companies wouldn't hire the person. A person is incapable of making a mistake like this. A mistake like this requires a fundamental lack of quality control systems and processes. There is a name you can directly blame. George Kurtz, CEO and Founder of a company who demonstrated that they don't take any basic QA/QC precautions.
Re: (Score:2)
"I would not like to be the person who authored the bad patch at CrowdStrike."
I would not like to be the person who authored a parser with no input validation,
I would not like to be the person who put code with no error handling in an environment where an error can bring down the entire machine.
There are many things that had to go wrong, and a bad patch is neither necessary nor sufficient to cause this scenario.
Re: (Score:2)
Re: (Score:2)
This was not the fault of a single person. This was grossly inadequate processes, lack of risk management and general lack of technological skill on the level of the organization. This is typically called organizational failure. Any halfway competent IT process expert will see it that way, as will any halfway competent IT risk manager.
Re: (Score:2)
The best possible outcome of this would be for crowdstrike to be forced into complete bankruptcy and liquidation. Crowdstrike is a total disaster, and was before this - i
Re: (Score:2)
I'm not even in the industry, but all industries and human nature are the same. Comments like "I would not like to be the person who authored the bad patch at CrowdStrike" might or might not be valid. He might indeed get wrung out to dry by corporate execs as the scapegoat for their own screwups, so yes, not fun to be him. But that does not inherently impeach him.
Code is complex. Sure, there might be some people or shops with a habit of incompetence, but even the best programmers and places can have ina
Patch wasn't bad as authored (Score:2)
The problem wasn't someone writing a bad patch:
It's essentially the same problem yo
Yes, test it - before putting it into service. (Score:2, Insightful)
I'm talking to you, Delta.
Re:Yes, test it - before putting it into service. (Score:5, Informative)
IIRC part of this issue was that Clownstroke does not let you do controlled updates like that.
Re: (Score:2)
https://www.crn.com/news/secur... [crn.com]
Re: (Score:2)
IIRC part of this issue was that Clownstroke does not let you do controlled updates like that.
That just pushes the fault back a step: Delta should never have gotten into such an arrangement.
Re:Yes, test it - before putting it into service. (Score:4, Informative)
I'm talking to you, Delta.
That's not how Crowdstrike channel updates work. Not for Delta not for anyone else. That said there was someone who should have tested something before putting it in service, but you're addressing the wrong side of the court room for this.
Re: (Score:2)
There is always one clueless idiot even weeks later. This time it is you. If you had done minimal research, you would FUCKING know that there was no way to test it without doing drastic things like blocking this update on firewall level except for a test system.
Valid if it was just Delta that shit itself. (Score:2)
Re: (Score:2)
Re: (Score:2)
Everyone should be testing before deploying. That fact that it hit so many in modern times shows up the bad practice creeping in.
Re: (Score:3)
https://www.crn.com/news/secur... [crn.com]
Bad practices that vendors force on you.
Re: (Score:2)
Re: (Score:2)
Bad practices that vendors force on you.
counterpoint: Delta chose a vendor that provided direct updates.
It used to be standard practice to employ an admin who's job was to test updates before rolling them out to production machines. This costs money (for staff and a test environment) and slows deployment.
A decision was made that the speed and efficiency of direct updates was more important than the risk of a bad update from the vendor.
Re: (Score:2)
That's not exactly a counterpoint when the argument is that executives are responsible because of executive decisions.
Re: (Score:2)
Re: (Score:3)
Not everyone. Everyone incompetent enough to rely on Microsoft products for critical functions.
Re: (Score:2)
EULA (Score:2)
One could guess that the EULA says the software does not have to work and you use it at your own risk.
https://www.crowdstrike.com/en... [crowdstrike.com]
Re: (Score:3)
Re: (Score:2)
A Contact you can't negotiate is called a "contract of adhesion"
"For a contract to be treated as a contract of adhesion, it must be presented on a standard form on a "take it or leave it" basis, and give one party no ability to negotiate because of their unequal bargaining position."
https://en.wikipedia.org/wiki/Standard_form_contract#Contracts_of_adhesion
This is what we see for most consumer contracts.
That said, if two companies are large enough they will negotiate a custom contract. Usually the largest co
Re: (Score:2)
Re: (Score:2)
I don't know. I'm an engineer, not a lawyer. I worked for a fortune 500 company before retiring so I got familiar with the processes they use.
Re: (Score:2)
Ever heard of gross negligence? You are always responsible for that, no matter what you put into the contract.
Re: (Score:2)
6. No Warranty.
6.1 Disclaimer. THE SOFTWARE AND ALL OTHER CROWDSTRIKE OFFERINGS ARE PROVIDED “AS-IS” AND WITHOUT WARRANTY OF ANY KIND. CROWDSTRIKE AND ITS AFFILIATES DISCLAIM ALL OTHER WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE.
Sounds fairly straight forward. Did Delta not read the agreement?
Re: (Score:2)
> Sounds fairly straight forward
That's not how the law works.
Unconscionable terms and acts of fraud and gross negligence are frequently found to be outside of contract terms.
It's far from straightforward. Partially self-serving on the part of lawyers.
Re: (Score:2)
Re: (Score:2)
Unlike you, Delta knows what gross negligence is.
Re: (Score:2)
There is no way they just clicked "yes" to the EULA. Major corporate software deals are always negotiated, and reviewed by lawyers. The rules that apply to us little people, don't apply to the big fish.
Make Microsoft accountable too. (Score:2)
Re: (Score:3)
Forced reboots are why many people use unsupported operating systems as they no longer reboot for updates disrupting workflows.
Microsoft doesn't force any reboots. Microsoft displays a warning on the screen that during the next reboot or shutdown an update will apply. Microsoft provides an offline timer to apply updates out of hours when the PC isn't use. Microsoft provide APIs that allow state recovery on reboot, as well as APIs to lock out reboots preventing work being lost.
If you are losing work you definitely have someone to blame, but it's not Microsoft. Start with your own IT department (yes ours forces reboots on our machine
Re: (Score:3)
If your systems are so fragile you can't reboot for an update without going offline you need to look more closely at your systems.
That's called a "single point of failure" and is considered a "bad thing".
At early startups, sure, we crossed our fingers and had plenty of outages but as the companies grew and matured we eliminated every spof we knew about or stumbled into the hard way.
Why are mature companies running known spof critical systems?
Re: (Score:2)
Of course, Crowdstrike does not have that kind of money lying around
Crowdstrike is worth $75billion. Not only do they have $4bn in free cash laying around, they could very easily raise more capital.
Re: (Score:2)
As usual, you claim crap and are off by a factor of more than 10: https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
I think he's probably referring to their market cap, listed at $72.43 Billion USD as of today.
Re: (Score:2)
Their annual recurring (I.E. subscription) revenue in 2023 was almost 3.5 billion, with a gross profit margin of 75-80% on that income.
They could (maybe with indigestion) stomach a 5.5 billion loss after several years similar to 2023 in recent history.
backups (Score:2)
How did that Clownstrike failure sneak into Delta's backups/snapshots? Asking for a friend.
Re: (Score:2)
Thel that friend to find out what actually happens and until he does that to stop asking stupid questions.
Problem is lack of transparency? (Score:2)
Maybe the worst thing is that CS's practices came AS A COMPLETE SURPRISE to everyone,
How many other companies out there are you currently relying on? What are their practices?
Delta, like many, took a decision to trust CS and whatever claims CS made about their products. Well, trust works for the good guys. But it's the bad guys that are the problem. And as the old Marx Brothers joke goes, if you can fake that then you've got it made. The bad companies are deceiving you into blindly trusting them.
The only an
Both at fault (Score:2)
CrowdStrike was at fault, but Delta owns the blame here. Delta had no way to recover in a timely manner; Delta didn't demand a Test QA before promoting changes to Prod. A $5 "break glass" USB could have been in a ziplock-type bag attached to each PC to boot to a base image and get back online; there are dozens of remote management recovery solutions available.
Re: (Score:2)
As others have noted, a bunch of old sysadmins appraising things in terms of uptime are not doing the same thing as a group of lawyers examining a contract.
Delaware? (Score:2)
The best possible outcome of this ... (Score:2)
Delta slipping to become Spirit (Score:2)
This airline has gotten cheap. Buying old-ass planes and not updating them, refusing to help customers, and their business is susceptible to a single point in technology failure. It's not the first time it's happened int he past several years. Only reason I still fly with them is status and they don't fly a lot of tiny-ass CRJs (American, ahem) that can't hold a roll-aboard. United is looking more appealing nowadays.
Yeeehawww (Score:2)
Cowboys exist everywhere, the building trades, and even in software development.
Microsoft and silly actions that might have happen (Score:1)