Police Department Loses Years Worth of Evidence In Ransomware Incident (bleepingcomputer.com) 131
"Police in Cockrell Hill, Texas admitted Wednesday in a press release that they lost years worth of evidence after the department's server was infected with ransomware," reports BleepingComputer. "Lost evidence includes all body camera video, some in-car video, some in-house surveillance video, some photographs, and all Microsoft Office documents." An anonymous reader writes:
Most of the data was from solved cases, but some of the evidence was from active investigations. The infection appears to be from the Locky ransomware family, one of the most active today, and took root last December, after an employee opened a document he received via via a spam email. The police department backup system apparently kicked in right after the infection took root, and created copies of the already encrypted data. The department did not pay the $4,000 ransom demand and decided to wipe all its systems.
Would be a shame if this happened to the IRS (Score:4, Funny)
Rubs chin, thinks it would be a shame if the IRS similarly lost their records. Also the student loan people. I would be sad if nobody had a record of the $60,000 in student loans that my wife owes.
Re: (Score:2)
Elliott, is that you?
Re:mmmmmm... (Score:5, Insightful)
They lost 8 years of files... Because it did a backup right after the encryption...
THE MORONS ONLY HAD ONE BACKUP!!!!
There is so much wrong with this from a security standpoint that whatever fool made that decision needs to either be fired, or at least removed from any influence over IT.
As the old saying goes:
So when did your data become important to you, before or after you lost it?
Backups? (Score:4, Informative)
It sounds like they only had one backup, and that promptly got overwritten. It should be standard procedure to have an offsite backup as well. I always did.
Re: (Score:2)
This.
Also, didn't they think of properly set file permissions?
Re: (Score:2)
It sounds like they only had one backup, and that promptly got overwritten. It should be standard procedure to have an offsite backup as well. I always did.
A backup implies exactly that regardless of medium or location, and if the backup runs after the infection, then you're doing nothing but backing up (ransomware) encrypted data.
The end result is you're still fucked.
Re: Backups? (Score:3)
Re: (Score:2)
What's the point of even doing a backup if you overwrite the only copy every time?
This is among the least expensive in terms of storage and in terms of time. You can do an rsync between a local and offsite storage, get a couple redundant copies easily and simply.
It prevents against a non-malicious system failure... e.g. a hard drive going bad.
Yes, its wholly inadequate vs ransomware, or malicious file modification, or gradually failing hardware that is corrupting data.
But even so it remains one of the most common backup strategies because it is simple an inexpensive. And police IT budget
Re: (Score:2)
Re: Backups? (Score:4, Interesting)
Retired IT here, after 34 years.
It's not easy being a cost center.
I was always on the wrong side of the ledger.
All of my meetings with management were about spending money that they had to recover.
Sometimes a new implementation would be an instant money-saver, but that was not very often.
I insisted on one of two (2) things:
1.) Acceptance of my recommendations or
2.) An official email quoting my recommendation, along with the rejection of same.
2.) was, on occasion, the answer to the question, "How in the hell could you let this happen?"
Re: (Score:2)
The whole cost center thing is a silly over-simplification anyway. If it enables the company to have revenue it is a profit center. The question they should be asking is if we replaced all of this IT stuff with a building full of people with spindles and adding machines, a big warehouse full of documents and a bunch of people to run documents back and forth, how much extra would that cost and how much would it slow us down.
IT is a cost center like sales and marketing is a cost center. You have to pay sales
Re: (Score:2)
TL;DR but a scan informs that IT is a cost center.
Re: (Score:2)
In the sense that EVERYTHING is a cost center yes. It is not as the management over-simplification where soem departments are macically profit centers even though they also cost money.
Re: (Score:2)
My wheelhouse is small in that I was the systems department.
I'm not qualified to fantasize about the cost of light bulb changers and shit.
I am an eye-witness to my own lengthy career and it's my call, not yours, regarding my position on the ledger that put me in the category of cost center.
So it is written, so let it be done.
Re: (Score:2)
It it is not sales, then it is overhead.
Re: (Score:2)
I didn't claim management didn't do that, just that it is wrong headed and results in problems.
Re: (Score:2)
It is wrong-headed for business managers to do risk analysis of data security.
In my world of IT management, we all knew, from firm to firm, what best practices would allow us to be top-notch gatekeepers.
Business made the decisions because they were the owners of the data.
IT managers and staff were simply the custodians.
Re: (Score:2)
No, wrong headed to decide one department is a profit center and gets gold toilets and another gets scraps because they are a cost center.
Risk analysis is fine, but only when that analysis is done by people smart enough to know that every center has a cost and every center brings profits. Those odd ideas about cost centers and profit centers lead to poor decisions which lead to bad results, like a police department that's loaded for bear but then loses all the digital evidence.
The fact that you had to keep
Re: (Score:2)
We didn't have gold toilets.
Perhaps you're thinking of this case [youtube.com].
Re: (Score:2)
Of course not, you were a "cost center". Also apparently bad at metaphor.
Re: (Score:2)
I never met a 4 I didn't like.
Re: (Score:2)
That seems to be the misguided belief. Naturally, they would like for the product to just appear in the warehouse for free, but likewise as long as they're wishing, they would like the potential customers to spontaneously wire money to them. Thus, in reality sales is also overhead.
Even janitorial services should be counted as savings since otherwise they would have >100K/year engineers or multi-million a year CEOs spending their expensive time waxing the floor in the lobby. Alternatively, they would lose
Re: (Score:2)
I do not disagree with anything you said. I just have run across this attitude before.
Re: (Score:2)
"This is among the least expensive in terms of storage and in terms of time."
No, it isn't. In terms of time is much quicker to backup to /dev/null, and even backups to /dev/null get surpassed both in time and storage by not doing backup at all.
And, in this case, it seems they offer exactly the same result so, why don't they make it clear -and cheaper, what are they really acomplishing?
Re: (Score:2)
And, in this case, it seems they offer exactly the same result
Yes *In this case*.
But in other cases -- such as the building it is in being destroyed in a fire or the hard drive/raid array getting fried by lighting or a power surge etc -- a simple rsync job offers an actual offsite backup copy that they can restore data from.
Its a legitimate backup strategy for a lot of use cases. In terms of risk management guarding against hardware loss or failure was historically the big one. Only recently has the 'malicious modification of files' rocketed to the top of the list bo
Re: (Score:2)
"a simple rsync job offers an actual offsite backup"
You don't understand what a back up is.
Hint: if it is not fully decoupled from the original source (as in "air gap") is not a backup. So an off-site rsync is not a backup; an off-site rsync and tarring the result from time to time to an external device, *may* be a backup.
You are probably in the league of those that think RAID5 is also a backup strategy ("sure, not always, not perfect, but in some simple cases...").
"Only recently has the 'malicious modifica
Re: (Score:2)
You don't understand what a back up is.
Give it a rest. I know full well what a back up is.
Hint: if it is not fully decoupled from the original source (as in "air gap") is not a backup.
Someone running crashplan or carbonite etc has a backup that is resilient to ransomeware, hardware failure, malicious tampering, etc. That is enough of a backup to mitigate most modern threats. It has incremental backups and versioning, and preserves deleted files. And it's not a mounted remote folder so a malicious process/user running your computer can't run amok and delete the backups. By itself, its not perfect, but its sufficient for a lot of entities.
Re: (Score:2)
If I have a tape drive, and 7 tapes one tape labelled for each day of the week, and each day I rotate the tape.
This gives me about a week of backups. Lots of entities uses stategies like this. It is certainly a backup strategy... it has a few flaws. It's vulnerable to the building catching fire. Its vulnerable to a malicious file deletion or modification if it goes undetected for more than a week.
Likewise, have 2 scheduled rsync jobs to 2 remote sites is a backup strategy. It has some advantages over the th
Re: (Score:3)
Like many, you don't understand the difference between a backup and an archive. A backup is meant to preserve data in the event of a hardware or other failure. An archive is supposed to preserve the data as it was at some point in history.
Re: (Score:2)
Your assumption of whether GP understands the difference is irrelevant to their argument: that during the period where your system is overwriting the only copy, you don't have a backup* to "preserve data in the event of a hardware or other failure"...
Oh, and also? While an archive is not necessarily a backup, a backup is inherently an archive.
* (you may have part of a backup, maybe even parts of two backups, depending on how your backup process overwrites the old one with the new one, but I certainly wouldn
Re: Backups? (!= archives) (Score:2)
By this rule 'live backups' that are (semi) online and available for users without other human interaction are actually archives. T
Re: (Score:2)
A backup implies exactly that regardless of medium or location, and if the backup runs after the infection, then you're doing nothing but backing up (ransomware) encrypted data.
The end result is you're still fucked.
Only if you foolishly overwrite all previous backups so that only the last version remains. If that's how their backup works, then it's severely lacking given the importance of the data in question. What if you need a file and discover it got corrupted, and it might have been corrupted months ago?
Re: (Score:2)
Doesn't excuse not having offline backups, but in the post-tape world that gets harder and harder.
Re: (Score:2)
Not what backup means, you're describing a RAID or other sort of mirror (even if it is delayed). Redundancy is not the same as a backup.
A backup has history, you could use snapshots or tape rotations or whatever, but older versions cannot be overwritten by newer versions, in most (best) cases, older versions cannot be written to period (the tape could have a physical tab or the storage system does not allow writing). When things change (eg. they are encrypted), you would see a very large backup if you're do
Re: (Score:2)
A single copy that's overwritten ever time it's run is not a backup of any nature it's a copy.
Why would evidence be stored on an internet accessible or even online thing. Computer forensics 101 is get the drive cloned, bagged, tagged, and stored all other digital evidence should be the same. How they can fail as such basic levels of evidence preservation is astounding. Realy anything not on a write-once medium since the time it was collected should be suspect.
Re: (Score:2)
https://www.google.com/search?... [google.com]
By definition, backups are just copies. A single copy that is overwritten daily is still a backup. It isn't a good backup plan, but it is not less of a backup by being only a copy.
Re: (Score:2)
From your own cited definition copies plural a single is just a copy.
Re: (Score:2)
Yeah, but what about the day BEFORE the infection?
And the day before that, going back in time.
I worked two law firms -- half day each -- and one wanted 7 days of rolling backup. The other wanted 30 days of rolling backup.
For the first site, 6 external hard drives (EHD) were always offsite, and for the second, 29 EHD were always offsite.
In both cases, the firms' management made the call regarding:
1.) Cost of backup hardware and offsite storage
2.) Risk of record retention -- especially email.
The law firms wer
Re: (Score:2)
Unless they routinely deleted email on reading, the backups might contain inboxes containging years of emails.
OTOH, unless there is a specific data retention requirement, discovery is for things you have. If things missing from discovery are consistent with your retention policy (showing that you didn't make a mad rush to delete things to avoid discovery), you're fine.
Re: (Score:2)
Your point is well-taken.
The only real protection from discovery would be the non-existence of emails (or other data) deliberately deleted prior to any overwrites with the backup.
Re: (Score:2)
So you restore from the day before that instead of the most recent one.
The whole idea of backups is you can restore to some point in the past - with the size of that window and the granularity within it depending on how much you want to spend...
If you just have one copy then you don't have a backup, you have a really slow RAID.
Re: (Score:2)
It's best to be able to get up to the minute backups, or roll back file versions. But the reality is, you might be so screwed that you have to go bac
Re: (Score:2)
Re: (Score:2)
For 40k/year I can easily set up a 200TB storage system, host it in a datacenter and professionally maintain it. $4000 buys you about 20TB which would probably last about 3-5 years without maintenance and that should be sufficient to back up pretty much everything in that police department with 3 months of retention.
Re: (Score:1)
" $4000 buys you about 20TB"
One could build an 8 bay, 32 TB, Raid 6 NAS for around $2000.
Re: (Score:2)
Re: (Score:2)
How is it able to overwrite the snapshots?
Re: (Score:2)
The phrase that comes to mind is, "An automatically mirrored copy is not a backup."
Any real backup strategy requires versioning. For example, my personal data backups involve a NAS providing storage for Time Machine. If a ransomware attack screwed up my Mac, I would have multiple backups that I could restore from, and if the ransomware attacked while the backup was running and corrupted the entire backup volume, I could still roll back the NAS volume to its most recent daily snapshot and restore the Time
Re: (Score:2)
Hell no, a federal government IT department? You mean, a bunch of bureaucrats charging $100k/y for a 10TB storage unit because 'vendors' from the Gartner Triangle recommended it to them and attached a huge IBM and Oracle contract to it.
What these small departments need is to find and hire a local IT person or if they can't afford an IT person (if you have less than 200 devices, you don't need a dedicated IT person), contract with a local company, there are plenty everywhere, they will take care of these sor
Re: (Score:2)
Re: (Score:2)
Something is REALLY fishy about this one.
Hopefully, the FBI looks into it, but with this admin, I doubt it.
Re: (Score:2)
Meh, it's only evidence. We all know that police don't care about that.
"backup" (Score:3)
The police department backup system apparently kicked in right after the infection took root, and created copies of the already encrypted data.
Backup. You keep using that word. I don't think it means what you think.
If you automatically overwrite previous data with no way to restore some older state, meaning that at a given moment you may only have a copy a few minutes old and no older state - it's not backup. It's just a secondary remote copy. Useful against heavy physical damage to the primary storage (or the whole machine), but nothing else. If it's not even remote, it's not useful for anything.
Re:"backup" (Score:5, Interesting)
Re: (Score:2)
https://www.google.com/search?... [google.com]
Or they are using the word exactly correctly? Just because it isn't a well thought out backup scheme, does not mean it is not a backup.
A link to scribd? That unreadable mess? (Score:2)
Here's a better link [theregister.co.uk].
Intentional infection? This doesn't add up. (Score:5, Interesting)
"Most of the data was from solved cases, but some of the evidence was from active investigations...the department did not pay the $4,000 ransom demand and decided to wipe all its systems."
I'm sorry, but one legal firm can rack up more than $4000 in legal fees in a single day.
You're going to tell me that the active investigations along with the potential liability of not holding data for years worth of solved cases was somehow not worth $4000?
The numbers just don't add up here. At all. Hate to go all conspiracy theory, but this sounds more like an intentional infection and a premature decision to wipe data that might have shown a bad light on a certain law enforcement actions.
Re:Intentional infection? This doesn't add up. (Score:5, Interesting)
Any evidence that was altered by ransomware would get challenged by a defense attorney. Maybe they decided they didn't need to pay ransom for evidence that had built-in reasonable doubt.
Re: (Score:3, Insightful)
Maybe they decided to do the right thing and not fund criminals. We need more people to do the same thing. If nobody payed, ransomeware would stop being a thing. Plus, the evidence should now be considered compromised anyway.
Re:Intentional infection? This doesn't add up. (Score:5, Insightful)
Re:Intentional infection? This doesn't add up. (Score:4, Insightful)
Re: (Score:2)
Re:Intentional infection? (Score:2)
How could you trust any evidence once you know the system is open to change? More than one person should be fired for this.
Re: (Score:2)
Re:Intentional infection? This doesn't add up. (Score:4, Interesting)
It is $4000 to a criminal organization, it's illegal (especially for government agencies like a POLICE department) to make any payment and become complicit in the criminal activity.
On the other hand, $4000 is what they start off with, I heard of a company that got hit with $10k in ransom demands, a few days later they realized their backups weren't working well so they gave them the $10k, by then the criminals realized they were attempting and failed to restore from backup so they quadrupled the demand so the company got the FBI involved, when the criminals realized the FBI got involved, they wiped EVERYTHING. It took them about 3 weeks and about $100k to recover the broken backups by a professional recovery company.
Re: (Score:2)
Sucks to be that company, but on the upside the criminals also apparently got nothing out of it.
Re: (Score:2)
Re: (Score:2)
You're going to tell me that the active investigations along with the potential liability of not holding data for years worth of solved cases was somehow not worth $4000?
What legal liability? Some cases might get dismissed but why would that matter for the police department?
Re: (Score:2)
You're going to tell me that the active investigations along with the potential liability of not holding data for years worth of solved cases was somehow not worth $4000?
What legal liability? Some cases might get dismissed but why would that matter for the police department?
And some cases might get re-opened because of new evidence brought to light that might benefit the wrongly accused, which would be essentially impossible to further such an investigation because of evidence being destroyed.
Regardless, the chain of custody issue has to be validated with such an intrusion anyway, which even furthers my point regarding this being used as a scapegoat excuse for evidence being destroyed deliberately by those holding it.
Re: (Score:2)
And some cases might get re-opened because of new evidence brought to light that might benefit the wrongly accused, which would be essentially impossible to further such an investigation because of evidence being destroyed.
Why would the police care about that either? Most prosecutors certainly do not and go to great lengths to prevent review.
Re: (Score:2)
And some cases might get re-opened because of new evidence brought to light that might benefit the wrongly accused, which would be essentially impossible to further such an investigation because of evidence being destroyed.
Why would the police care about that either? Most prosecutors certainly do not and go to great lengths to prevent review.
If years of evidence is truly worthless to the organization holding it, then why the hell did they even save it.
Re: (Score:2)
And some cases might get re-opened because of new evidence brought to light that might benefit the wrongly accused, which would be essentially impossible to further such an investigation because of evidence being destroyed.
Why would the police care about that either? Most prosecutors certainly do not and go to great lengths to prevent review.
If years of evidence is truly worthless to the organization holding it, then why the hell did they even save it.
Often they do not. There may be statutory requirements.
Feels like we're heading backwards (Score:2)
After all, Computers have complicated lives very greatly.
Re: (Score:2)
Bad practices are bad practices, be it with physical copies or digital.
How the fuck? (Score:1)
How do they not have a proper storage array with snapshot and remote copy functionality to provide both point in time "backups" as well as offsite replication?
Even if they couldn't afford that why weren't they doing disk to disk backups using removable drives or backing up to a local tape drive and rotating the tapes daily?
I sure hope somebody got fired for this...
Re: (Score:2)
Well, clearly their spending priorities are wrong. Police are not a paramilitary organization; they don't exist to fight battles although that happens sometimes. Their primary function is to bring miscreants to justice, along with the evidence needed to obtain a conviction. If they can't do that there's no reason to spend money on police at all; you could just put the money into the National Guard instead.
Re: (Score:1)
Because most smart folks don't want to be cops, and most cops aren't all that smart...
Statement says they did NOT lose evidence (Score:3, Informative)
"...hard copies of ALL documents and the vast majority of the videos and photographs are still in the possession of the Police Department on CD or DVD".
They only lost digital copies of evidence...probably why they chose to wipe rather than pay ransom.
Re: (Score:1)
Or anything that hasn't been converted to hard-copy. I doubt the devices record on ROM media directly.
Windows I presume (Score:1)
Presumably, Windows. Balance of probability both by numeric prevalence and vulnerability. How is it responsible for police to store valuable data on a vulnerable system? Without backup no less?
Re: (Score:2)
Presumably, Windows. Balance of probability both by numeric prevalence and vulnerability. How is it responsible for police to store valuable data on a vulnerable system? Without backup no less?
What's this, a visit from a Microsoft astroturder with mod points? Confirming that Microsoft is, well, the same old Microsoft.
BTW, it is not in doubt that the police had their stuff on a Windows computer because Locky, like the vast majority of ransomware, is Windows malware.
well, prop for not paying the ransom, but... (Score:1)
It's good they decided not to perpetuate the randomware industry by paying the ransom. That was the right choice on their part.
However: "...after an employee opened a document he received via via a spam email...."
There are all kinds of problems with this, starting with the general lack of technical awareness of the whole population (I won't blame it specifically on that one lady or gent: they have a billion or so to keep them company, and if you only try to hire technically literate people, you won't have
Re: (Score:2)
In a group of people who aren't computer security professionals, somebody's going to open the document, or at least, you have to figure someone will. The exact person doesn't matter. If your computer system is such that opening a document can encrypt the storage, somebody's screwed up the system very thoroughly.
Amazing (Score:2)
It's disheartening to realize that I, just an average Joe, have better, more secure backup procedures than the police department in Cockrell Hill, Texas.
Yeah, maybe they're just a podunk little town in the middle of nowhere, but still...
Yay to "no backups"! (Score:2)
Maybe the police department will now move on a backup system with multiple iterated backups so that if this ever happens in the future they can recover everything.
Re: (Score:2)
Nah, this is a feature. It takes the pressure off from having to do actual work.
I am sure they probably feel that more evidence needs to be digital and on systems that they let the intern surf the web on.
Right to be forgotten (Score:2)
EU's right to be forgotten has been criticized as impossible to enforce, but here is its implementation: get infected, refuse to pay, wipe data.
Right to be enforced is enforced with the help of ransomware, though the citizen cannot choose when it happens.
Incompetent and Irresponsible. (Score:2)
Whether it's called backup or archive, their system was inadequate. Multiple full copies taken at various points in time - of all data - should be in more than one offsite location.
Just a single backup? (Score:2)
So are we to understand there is just a single backup which, when running, overwrites the previous backup? So if you backup at the wrong moment, everything is gone? That is extremely, extremely incompetent...
I can understand losing maybe a few days of work, but beyond that point, an older backup should be recoverable. Why wasn't it?
Protecting backups from ransomware/infections (Score:1)
That said, regardless of how many backups a location maintains, there should be a standard mechanism that analyzes key files BEFORE starting a backup, verifies that they have not been modified or deleted, i.e. by ransomware, and if it detects that they have been modified or deleted, displays an alert and stops the automatic backup before it even begins, thereby protecting the integrity of the existin
Small wonder (Score:2)
What did you expect from a Texas Police Department, after all they got a president killed on their watch.
$4000 was all? Sounds weird (Score:2)
And the back-up only went with 1 level? Seriously?
Normally, I stick up for the police, but Gut feeling says that there was a case that they did not want tried and the police were in on that.
Re: (Score:2)
They could have paid it... but that would have blown the budget for the PD sponsored BBQ....