When Ransomware Gets Paid By A City's Insurance Policies (news18.com) 131
Remember when the small town of Lake City, Florida paid $460,000 for a ransomware's decryption key?
As they slowly recover 100 years of encrypted municipal records, the New York Times looks at the lessons learned, arguing that cyberattackers have simply found a juicy target: small governments with weak computer protections -- and strong insurance policies. The city had backup files for all its data, but they were on the same network -- and also inaccessible... The city's insurer, the Florida League of Cities, hired a consultant to handle the negotiations with the hackers via the email addresses that had been posted on the city server. The initial demands were refused outright, and city technicians raced to find a workaround. "We tried a lot of different solutions," said Joseph Helfenberger, the city manager. None of them worked. "We were at the end of the day faced with either re-creating the data from scratch, or paying the ransom," he said.
The insurer's negotiator settled on a payment of 42 Bitcoins, or about $460,000, Helfenberger said, of which the city would pay a $10,000 deductible. After the payment, the hackers provided a decryption key, and recovery efforts began in earnest.
As it turned out, recovery would not be simple. Even with the decryption key, each terabyte has taken about 12 hours to recover. Much of the city's data, nearly a month after the onset of the attack, has still not been unlocked... In Lake City, the information technology director, blamed for both failing to secure the network and taking too long to recover the data, wound up losing his job.
Mark A. Orlando, the chief technology officer for Raytheon Intelligence Information and Services, tells the Times it's unrealistic to expect cities to never pay the ransom. "Anyone who said that has never been in charge of a municipality that has half their services down and no choice."
But does that create an ever-widening problem? The FBI knows of at least 1,500 reported ransomware incidents last year, according to the article, although the Illinois computer programmer offering free decryption help at ID Ransomware says he's receiving 1,500 requests for assistance every day.
As they slowly recover 100 years of encrypted municipal records, the New York Times looks at the lessons learned, arguing that cyberattackers have simply found a juicy target: small governments with weak computer protections -- and strong insurance policies. The city had backup files for all its data, but they were on the same network -- and also inaccessible... The city's insurer, the Florida League of Cities, hired a consultant to handle the negotiations with the hackers via the email addresses that had been posted on the city server. The initial demands were refused outright, and city technicians raced to find a workaround. "We tried a lot of different solutions," said Joseph Helfenberger, the city manager. None of them worked. "We were at the end of the day faced with either re-creating the data from scratch, or paying the ransom," he said.
The insurer's negotiator settled on a payment of 42 Bitcoins, or about $460,000, Helfenberger said, of which the city would pay a $10,000 deductible. After the payment, the hackers provided a decryption key, and recovery efforts began in earnest.
As it turned out, recovery would not be simple. Even with the decryption key, each terabyte has taken about 12 hours to recover. Much of the city's data, nearly a month after the onset of the attack, has still not been unlocked... In Lake City, the information technology director, blamed for both failing to secure the network and taking too long to recover the data, wound up losing his job.
Mark A. Orlando, the chief technology officer for Raytheon Intelligence Information and Services, tells the Times it's unrealistic to expect cities to never pay the ransom. "Anyone who said that has never been in charge of a municipality that has half their services down and no choice."
But does that create an ever-widening problem? The FBI knows of at least 1,500 reported ransomware incidents last year, according to the article, although the Illinois computer programmer offering free decryption help at ID Ransomware says he's receiving 1,500 requests for assistance every day.
Re: (Score:2)
Lessons learned? (Score:5, Insightful)
And now that the hackers know that insurance will pay for their hacks, we have a whole new growth industry.
I wonder who the first billionhackaire will be.
Re:Lessons learned? (Score:5, Insightful)
I really feel for the IT Manager who lost his gig. I guarantee, once the data are finally unencrypted, if someone were to research his requests for funding various lacking parts of his budget, you'd find he was repeatedly denied and ended up doing the best he could within the limits set by the politicians.
This is a common issue within any organization, let alone the public sector, where those who know nothing about IT are put in charge of the purse strings and see it only as a cost center, meant to be limited as much as humanly possible.
Re: (Score:2)
Possibly true but irrelevant. Proper security and backups doesn't have to cost a huge amount of money.
Is that sum greater than zero? 'Cause even if it's small, he's got to convince the City Council to pay for it. And they've got a very important street renaming initiative to pay for....
Re: (Score:2)
It could be zero. I've done backups on zero-dollar budgets. Find an old server someone is chucking or not using anyway and install URBackup or any other number of open source backup programs, with ZFS snapshotting you have immutable backups.
You just have to prioritize what you think is important - is it deploying the latest Slack or other nonsense time-wasting chat app or is it a backup system.
And about finding that old hardware, work for a government-funded office once in a while. When Sun became Oracle, y
Re: (Score:2)
However, if you are dealing with a much more distributed system, like hundreds of desktops with work on them and little servers everywhere, setting up and mai
Re: (Score:2)
Re: (Score:3, Interesting)
Is that sum greater than zero? 'Cause even if it's small, he's got to convince the City Council to pay for it.
Small expenses don't need council approval. A 4TB backup drive costs $79. Buy two so you can hot swap daily, and it is still petty cash.
The IT manager deserved to be fired.
Re: (Score:2)
Have you backed up 4TB of data properly? While your recommendation is better than nothing your recommendation doesn't allocate any time or other resources. A HD is not expensive but that is not the cost of a backup. At least not one that is done properly.
Re:Lessons learned? (Score:4, Insightful)
What's more baffling is how they managed to get ransomware insurance, and get it to pay out, when they had no backups.
When you get commercial insurance against fire, the insurance company requires you to have smoke alarms and fire extinguishers and fire doors and do a fire safety assessment of the premises. I'd have thought that an insurance company offering cyber attack insurance would have wanted to audit their systems too.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Alternatively, what if the term of the contract he was in actually made it impossible to implement a backup system.
Yes, I have seen this. No sorry IT you are not allowed admin rights (and much worse).
Whose fauly is it then? Yes they will still scapegoat IT. But when they cannot do their job within the contract, and there are always people out to use them as a method for promotion if they do, how can anything work.
And yes the larger or more corperate a company then the more likely this will occur! Many large
Re: (Score:2)
I a country with any sort of remotely decent employment rights you would sue the employer for unfair dismissal and win a hansom payout.
Re: (Score:2)
Cost centers basically means the operation doesn't see a direct benefit from it. It may be necessary to the operation but it doesn't bring in customers.
Re: (Score:2)
I really feel for the IT Manager who lost his gig. I guarantee, once the data are finally unencrypted, if someone were to research his requests for funding various lacking parts of his budget, you'd find he was repeatedly denied and ended up doing the best he could within the limits set by the politicians.
And don't forget the bean counters, who want the IT department's budget for themselves. My experience is that the bean counters metastasize, eventually demanding to hire 6 figure accountants to keep track of things like the 600 dollar pencil budget. You don't get to make smart financial moves like that with having IT people making money.... or existing.
Re: (Score:2)
I really feel for the IT Manager who lost his gig. I guarantee, once the data are finally unencrypted, if someone were to research his requests for funding various lacking parts of his budget, you'd find he was repeatedly denied and ended up doing the best he could within the limits set by the politicians.
While that might be true, it could also be totally not what happened at all. I can tell from this kind of post you've never worked for any government in the USA. I did early in my career. I'm long gone from that job, but I really learned a lot. Here's a rough guide to how things were and probably still are.
1) Federal government (where I worked) - There are really smart IT people there and some very good managers. People that take these jobs usually do for job security as pay is always going to
Re: (Score:2)
You are being too kind. If he had any evidence of requests and denies, he would have been able to fight to keep his job. Municipal emails are kept for 7 years so there should be "some" record of requests, even if they were denied.
You are correct though. The data owners are very poor at making data security decisions.
The new rise of Bitcoin (Score:2)
I am seriously wondering how much the rise of bitcoin can be attributed to the market for cities (and companies) needing to buy bitcoin to pay off ransoms!
Re: (Score:2)
I am seriously wondering how much the rise of bitcoin can be attributed to the market for cities (and companies) needing to buy bitcoin to pay off ransoms!
Good question. Kinda scary as well, because it's a positive feedback loop.
Re: (Score:2)
I am seriously wondering how much the rise of bitcoin can be attributed to the market for cities (and companies) needing to buy bitcoin to pay off ransoms!
It shouldn't have much effect if the crooks are cashing out. The price will only stay high if they are holding them.
Cities have to go public. Companies can keep it under the rug. So the problem may be bigger than it appears.
At least the hackers are driving incentives for better security.
Re: (Score:2)
I wonder how long our three letter agencies, with all their resources, would have taken to crack the key? I guess it depends on the key size. If ever realistically feasible, why wouldn't they offer such a service to tax payers?
Re: (Score:2)
They wouldn't have to reveal their full strength. Compare it to military technologies that are made public once they have moved far ahead. Also, if they are able to crack it in 6 hours on average, just take 2 days etc.
At some point, if the attackers test ridiculously long keys, the target hardware won't be able to encrypt everything before weeks which raises the possibility of being detected and make the attack less efficient.
Re: (Score:2)
I wonder how long our three letter agencies, with all their resources, would have taken to crack the key?
Many times the age of the universe.
If ever realistically feasible, why wouldn't they offer such a service to tax payers?
It is completely infeasible.
fire suppression systems -- city code required (Score:2)
Actually, insurance companies do not really care about smoke detectors and sprinklers. City building codes require that stuff in commercial buildings and are responsible for policing it. A fire that triggers an alarm and sprinklers means typically almost as large an insurance payout as if the building had burnt to the ground. The sprinklers and alarm are only there to mitigate loss of life. The water
Re: (Score:2)
A fire that triggers an alarm and sprinklers means typically almost as large an insurance payout as if the building had burnt to the ground. The sprinklers and alarm are only there to mitigate loss of life. The water causes essentially the same amount of damage to the building as the fire would.
Do you have a source for that? Sprinklers are triggered by local heat (which melts a plug in the nozzle), not by a central valve which floods an entire building if triggered. Sprinklers act only where the fire is and keep the fire/water damage local; preventing the fire from spreading and destroying the entire building.
Re: (Score:2)
Actually they do care, and insurance your rates will reflect that. For example FM puts out a load of white papers and standards on fire prevention (among other things affecting loss rates) and FM can, and will, require some customers to conform to them in addition to the code requirements. I have worked on construction projects where the design had to be reviewed and was modified by FM.
Re: (Score:2)
The lesson learned, I think, will be that insurance companies will stop writing policies that cover ransoms without proof that systems can be restored from backup or just drop that kind of coverage altogether.
Eventually, yes.
Re: (Score:2)
Why on earth do you think such extortionists just learned this?
There's a reason they do this: some targets pay. Enough targets pay enough money to make it worthwhile to the criminals.
Re: (Score:2)
Why on earth do you think such extortionists just learned this?
You are way overparsing my sentence. Feel free to re-write it, I suppose. It was merely a way to segue to the third sentence
My point was in the first sentence. The cities or businesses won't learn anything.
There's a reason they do this: some targets pay. Enough targets pay enough money to make it worthwhile to the criminals.
Right, which was the takeaway
Re: (Score:3)
And now that the hackers know that insurance will pay for their hacks, we have a whole new growth industry.
I'm guessing that the ransomers are in cahoots with someone who works for the city, and gets the city employee gets a cut of the ransom for intentionally spreading the ransomware.
But to hire some private detectives, to check out employees, would probably cost more than paying the ransom.
Re: (Score:2)
And now that the hackers know that insurance will pay for their hacks, we have a whole new growth industry.
I'm guessing that the ransomers are in cahoots with someone who works for the city, and gets the city employee gets a cut of the ransom for intentionally spreading the ransomware.
Highly possible. I think that local governments being only too happy to pay off people isn't going to end well when their insurance rates skyrocket.
But to hire some private detectives, to check out employees, would probably cost more than paying the ransom.
Kinda a "let's keep this as quiet as possible" instinct as well.
So their insurance gets to pay out almost half a million dollars. To criminals, as ransom.
Almost a dead lock that the insurance company will hire some forensic investigators. And I'd be really curious what the city's next insurance bill will be.
And if they do insure the city, there's anoth
Re: (Score:2)
Almost a dead lock that the insurance company will hire some forensic investigators.
. . . now to test how really "Ol" you are . . . did you ever watch Banacek [wikipedia.org] when you were younger than "Ol" . . . ?
This case would have been perfect for him!
Re: (Score:2)
Almost a dead lock that the insurance company will hire some forensic investigators.
. . . now to test how really "Ol" you are . . . did you ever watch Banacek [wikipedia.org] when you were younger than "Ol" . . . ?
No- I never saw that show. Sounds like it might have been amusing, what with all of Peppard's strange sayings.
I do watch "Mike Tyson Mysteries" on occasion, I'll bet he could straighten them out.
Re: (Score:1)
I think we already know that. It's Bill Gates.
In his day he was a hacker of the first order. A genius.
Re: (Score:2)
I wonder who the first billionhackaire will be.
You are a bit late for that, like 50+ years. All of you were already hacked in your heads into exploatative work practices, outsourcing and other work laws suited for their purposes, while the billionhackaires laughed all the way to the bank. And the sums we are talking about range in 10's of trillions if you sum it all up through all the years.
Worst.
Analogy.
Ever.
Re: (Score:2)
The learned lesson??? the hackers know that insurance will pay for their hacks
For the time being, exactly.
So where's the data center (Score:5, Insightful)
'Even with the decryption key, each terabyte has taken about 12 hours to recover.'
And who was blind enough not to see resources needed to encrypt their data? How could someone be so blind not to notice during the attack?
Re: (Score:2)
To be fair, the hacker would have optimized the encryption process to proceed as fast as possible to encrypt as many files as possible before it was detected and stopped.
But, it's likely the hacker either did not provide a decryption tool at all, or provided only a bare minimum one, making the decryption process much slower. Why bother, since at that point the hacker has your money and no longer cares.
Re: (Score:2)
Why bother, since at that point the hacker has your money and no longer cares.
They care because they want to maintain their reputation for good customer service. Once you pay, you are treated well. This means more people will pay in the future. They might even get a few repeat customers.
Re: (Score:2)
My guess is that different client PCs had different keys, so they have to figure out which key was used to encrypt each file which makes the decryption process a lot slower than the encryption process.
Also from TFA it sounds like it happened at night, so managed to hit a large amount of data before anyone noticed. Apparently as well as having no backups they didn't have any intrusion detection either.
Proper backups or insurance for lost data? (Score:1)
Only one question is asked. Which one is cheaper?
Re: (Score:2)
Also consider: the next group of hackers might not be after Bitco
Re: (Score:2)
Yes, all those things are true, but the bean counters will take the path of least resistance. They decide how well the IT department is managed.
Re:Proper backups or insurance for lost data? (Score:5, Informative)
Re: (Score:2)
How do you expect this to happen when the recovery cost would bankrupt the city? Someone is going to have to bail them out.
Re: (Score:2)
Re:Consider the data erased, not encrypted. (Score:5, Insightful)
This is not rocket science, and with storage being so cheap, there’s no excuse. Same as there’s no excuse for not maintaining multiple backups of your personal data, not when a 128 gig Kingston USB stick retails for under $30 at Staples.
Re: Consider the data erased, not encrypted. (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
To be fair they did have at least 16TB of data, probably more, to back up. That's not trivial. A hard drive capable of sustaining 150MB/sec write speeds would take 31 hours to copy all of it.
So you need fast storage media, you need to be doing differential backups, and you need to do it automated overnight for performance reasons. And some data isn't trivial to do such backups on, e.g. if you have a database or email server store it needs to be saved in a state where it is consistent so you can reload it, o
New insurance policies will change this: (Score:2)
They will insist on a best-practices backup mechanism and will pay reasonable costs associated with downtime and restoration from those backups, but they will explicitly declare the entire policy void if any payment is made as a ransom or to any "recovery company" that isn't KNOWN to never pay ransoms.
Backups (Score:1, Insightful)
So they don't have offsite backups, no rotating backups or maybe just mirrored everything somewhere.
Even no archives - old files could be manipulated as well.
This is total incompetence, you can do DIY backups with second hand PC and free software like home system, storage is dirt cheap nowadays.
Microsoft mono culture is also to blame- single system types, problems spread like a fire everywhere.
Re: (Score:1)
So they don't have offsite backups, no rotating backups or maybe just mirrored everything somewhere.
Even no archives - old files could be manipulated as well.
This is total incompetence, you can do DIY backups with second hand PC and free software like home system, storage is dirt cheap nowadays.
Microsoft mono culture is also to blame- single system types, problems spread like a fire everywhere.
Right. Because a Linux mono culture, with no backups, or backups on the same network, would be so much better.
Dumbass.
Re: (Score:3, Insightful)
Re: (Score:1)
So "MS_certified_double_clicker" , who said Linux, and how the hackers will encrypt a filesystem which is kept on different file system separated by say rsync of FTP, not accessible to Windows workstations if you do _version_ backups or snapshots? Even Macs have version backups with hard links.
But you don't know about these because MS schools don't teach them.
Re: (Score:2)
Yeah, monoculture = a good chunk of the problem. A snapshotting file server could have mitigated a good chunk of this. Good luck getting one of those from Microsoft though :-(
Need to outlaw paying ransom. (Score:5, Interesting)
We need strong laws against EVER paying ransom, combined with technical and financial assistance to help cities that are attacked.
This market needs to be crushed ASAP or the cost of doing business will go way up for all governments. Its easy to say that they "should" have had better security, but that security requires hiring talented people and that cost comes out of taxes.
I'm also worried that these attacks are a proving ground for large scale cyber attacks by foreign governments.
From: Your Friendly Neighborhood Hacker (Score:3)
Hi! Thanks for your recent transaction, I'm so glad we could be of help to you. I hear you're having a time-crunch problem with the key we provided you. We could easily provide you with a shorter key that would run much faster.
All we'd need is you to forward THIS email to at least one of your friends. But don't worry, even after opening this email, _your_ systems are still safe -- after all, what are friends for?
P.S. - do we need to pay state taxes on our recent transaction? No, I didn't think so.
Responsible backups are unrealistic? (Score:2)
Why would you ever have to pay a ransom if you had any kind of even half-robust backup policy? You might have some down time while you recover your data from a backup after an attack, but if you paid the ransom, you'd be having the same sort of downtime just decrypting your data anyways
Re: (Score:2)
Honest question: what does a half-robust backup policy look like to you? Half-robust to me sounds a lot like using Veeam to snapshot VMs to a NAS that is isolated from the workstation network, but obviously network accessible to all the servers, and then replicating the backups weekly to secondary storage. Robust would be getting it offsite.
That process can still be corrupted due to the significant storage over-provisioning required to handle every file changing. If you have 100TB, and 2TB changes per day
Re: (Score:2)
Re: (Score:2)
I would call that about 25% of a solution, and the issue is still having sufficient disk capacity and monitoring on your backup system to detect a problem in a day or two rather than two weeks later.
We used to use the Synology BTRFS snapshots on our backup system, but it really wasn’t as bullet proof as I would hope, but there is at last a redundant archive of (file) data. But even with off-site storage you usually are going to have a cost-controlled data retention policy that could do you in.
But havi
Re: (Score:2)
Having sufficient disk capacity should not be a problem... storage is cheap. You need twice the storage just to maintain the mirror, plus whatever additional space is required to keep track of diffs on the data files from day to day.
Especially when you compare it to the costs of not keeping a backup when things go south.
The malware dormancy can't be too long, or else it runs a risk of being detected before it can execute its primary payload, so even a very mundane scanning policy of checking for malwa
CTO of Raytheon IIS (Score:3)
Storks and Victims (Score:2)
The observed ratio of encryption to decryption speed is about 14:1. https://crypto.stackexchange.c... [stackexchange.com]
This suggests if it takes 12 hours to decrypt a TByte, and they've been going at it a month (let's call that 30 days), then at the same ratio, assuming asymmetric cryptography using RSA e/d it means it took over 30 hours to encrypt the data in the first place. Assuming the "networked backups" they had contained the same data, but possibly taking up less space depending on file contents, I'm going to make u
Re: (Score:2)
The large difference between encryption and decryption is for public-key encryption such as RSA. Normally, you only use RSA to encrypt a key for a fast, symmetric cipher such as AES. At least, that's what happens with SSL, SSH, and various legit full-disk encryption schemes. Not sure what this particular ransomware does.
Note that 1 TB in 12 hours is 23 MB/s, which requires 46 MB/s bandwidth at the disk if the decryption is to be done in-place. Doesn't seem unrealistic if there is a network in between and pe
Not a backup... (Score:4, Insightful)
"The city had backup files for all its data, but they were on the same network"
If it's online (spinning rust or not), then it isn't a backup, it's a copy.
A small town has 60TB of municipal records? (Score:1)
Seriously? People are expected to believe that?
Re: (Score:1)
What? That's too small to believe? /. people have a petabyte spinning in their basement and that's just for their porn collection.
I bet a number of
Giving Money To Criminals? (Score:3)
Re: (Score:3)
Of course it is. I do so every April 15.
Price tag (Score:1)
Now they paid criminals,
they will pay for decryption labour and hardware,
they will pay for cleanup the malware,
will pay for consultants to design proper backup and
pay for software and hardware for that implementation.
They will pay for downtime, delayed payments, contracts, maybe salaries, projects, penalties...
Eventually will pay for some litigation or class action from victims of the leaked data.
Insurance from now on will go up as well.
I don't know how they gonna trust that "recovered" data.