Angry IT Admin Wipes Employer's Databases, Gets 7 Years In Prison (bleepingcomputer.com) 83
Han Bing, a former database administrator for Lianjia, a Chinese real-estate brokerage giant, has been sentenced to 7 years in prison for logging into corporate systems and deleting the company's data. BleepingComputer reports: Bing allegedly performed the act in June 2018, when he used his administrative privileges and "root" account to access the company's financial system and delete all stored data from two database servers and two application servers. This has resulted in the immediate crippling of large portions of Lianjia's operations, leaving tens of thousands of its employees without salaries for an extended period and forcing a data restoration effort that cost roughly $30,000. The indirect damages from the disruption of the firm's business, though, were far more damaging, as Lianjia operates thousands of offices, employs over 120,000 brokers, owns 51 subsidiaries, and its market value is estimated to be $6 billion.
Backups? (Score:3, Insightful)
This scenario plays out every so often, and you have to ask; Where are the database backups? I mean, it's understandable this would disrupt business temporarily and have a cost involved in restoring everything. But leaving employees "without salaries for an extended period" sounds like regular backups weren't being done here.
Re:Backups? (Score:5, Insightful)
What I've found time and time again is that when people say they have backups, no one is ever actually testing the backups. I've seen backup systems that stopped working months before, and it was only discovered because someone deleted a directory, and when the IT team went to restore the folder from backup, lo and behold, the most recent backup is two months old. When I was directly running an IT department, weekly testing of the backup was required, because looking at a status box on a screen that shows green often doesn't mean a damned thing.
Re: (Score:1)
Lets just assume for a moment that you are lucky enough that your backup actually worked and that you can get back the data that you saved. Was the whole computer backed up, or was it just the data in the database? Does anyone know how to build the computer from scratch and configure everything that is needed? Do you even know what is needed?
The only sure fire way to know that your backup is working is to s
Re: (Score:2)
I simply use RAID 10 as backup strategy, no backup testing needed, much simpler this way /s
Re: (Score:2)
I simply use RAID 10 as backup strategy, no backup testing needed, much simpler this way /s
lol, my heart rate increased as I read that. Thank goodness for the /s!
Re: (Score:2)
I use RAID 0 for this. /s
Re: (Score:2)
But, but, but... RAID 0 means no (0) backups! RAID 10 has 2 levels of backups so you are guaranteed to never lose data in case of a ransomware attack ! /s
Re: (Score:2)
Re: (Score:2)
Re: Backups? (Score:5, Interesting)
I wanted to do that, but I was far too busy because today was the day that I had to change my furnace filters, my car also exactly hit the long-term service interval mileage so I've been busy inspecting all of my rubber grommets like the good book said.
This also was the once a month I had to go around pressing the test button on all my smoke alarms, I also had to change that little battery inside my furnace backup board because of the book keenly warned me to do this exactly Ten Years After purchase. Same day I had to go around and test all the HVAC events for balanced air flow prior to me checking all rotating fans to be rotating in the most efficient direction for this season. It also is exactly 3 years after my last chimney sweeping so I had to get that done, but not before taking the grease gun to lubricate all of the grease points on my lawn mower as it had asked for and it's manual. But then I was too busy replacing seat belts because they were older than 25 years which technically means they can expire. But then I realized my tire rubber was older than 5 years which also means it technically isn't good either so it too had to go...
But I couldn't do that because I was balancing my checkbook, before performing a bunch of income calculation make sure this year dependent filings are still correct-o-mondo. But the city council meetings I was urged to attend are happening so you know.
Re: (Score:3)
I see your point, I think, and you are right. For a private entity.
But this is a six billion corporation. They should absolutely be able to pay someone to see this task as his primary raison d'etre.
I am a storage, virtualization and compute engineer. Call it cloud if you want. I checked compliance almost daily and did restore tests once a month.
But granted, it was random testing, not all of it. Because that would have made my brain ooze out of my ears.
Believe it or not, everybody talks about testing backups
Re: (Score:3)
I see your point, I think, and you are right. For a private entity.
But this is a six billion corporation. They should absolutely be able to pay someone to see this task as his primary raison d'etre.
What if that person is the one who decides to delete everything?
Setting things up to deal with an accidental issue is one thing- though far fewer places clear that bar than they should. Making sure that the people dealing with that can't also break things is another level of work, which is non-technical folks are going to have a very hard time determining if is being done correctly. Security is hard.
Re: (Score:2)
Backups should be shipped off-site and be on offline media. That way even if the sysadmin goes nuts one day, there's still an archive they can't touch easily.
Re: (Score:3)
That's a cry for help if I've ever heard one. Have you considered outsourcing?
I know you are going for funny, but that's precisely the reason a service industry exists in the world. Inspecting the car? Never done it, pay someone to do it yearly. Furnace filters, gas fittings, and testing? Never done it, some guy comes and does it every 2 years.
In both cases I don't even need to keep track. They send reminders. I also don't grow my own food, sew my own cloths, or teach my own kids.
The service industry exists
Re: (Score:2)
Re: (Score:3)
Also, companies don't always do what they're supposed to. As long as the profits keep coming in then they're not too worried about the fire insurance that they forgot to pay for. Sure, you may think it is unthinkable, but it happens. It's also somewhat invisible, everyone assumes someone else does it.
I used to do backups back in the 80s for a major defense contractor. I did not get a lot of instructions, there were no written procedures. You just emulated what someone else did. There's a label on the
Re: (Score:2)
Re: (Score:2)
I did some work for a school district in the mid-90's. Their IT group put out an RFP for a backup solution, and bought the absolute cheapest one they could find. Things were going fine until one day they did get a virus on the network that wiped out all their common storage (pretty much their grading system, attendance records, etc.) They later found out that while they were backing data up each night and rotating tapes each night offsite, their RFP never covered the restore function of the software. Th
Re:Backups? (Score:5, Interesting)
Re: (Score:2)
> Its not exactly clear whether they restored from backup or not. But given the restoration cost was only $30k I think it sounds like they did have backups, the $30k was probably mostly spent in recovering data between the last backup and time of deletion. So to me it sounds like backups were being done.
That's probably true, but in a recent document dump from NAIDA it showed that Shezheng Li is making $23K a year, and she's the top virologist in the country. $30K probably covered a lot of labor. I'd gu
Re: (Score:2)
This then goes into a deeper level of distributed and restricted access, and offsite backups.
Re: (Score:2)
Should not be able to. In a well-run IT organization of this size no single employee should be able to erase life system and backups. Backups should either be write-once with access by a different team only or offline again with access only by a different team. But yes, doing IT right costs money and many moron managers still do not realize that doing it wrong costs far more money in the long run.
Re: (Score:2)
I think they had backups. That "extended period" does sound like bad negligence though, I agree. They probably never ran any restore tests and that is just wrong. The reason you do these tests is because unless you do them regularly (e.g. every 2 or 3 years), you are basically assured to run into some serious, unexpected problems.
Re: (Score:2)
Interestingly, it's starting to become common for staff to be put on gardening leave immediately are made redundant or resign.
Guess who was probably in charge of backups (Score:2)
This scenario plays out every so often, and you have to ask; Where are the database backups?
Remember, this is a database administrator, deleting the databases he administered. Even if he didn't make them himself, he likely knows exactly where the backups are and how to delete or corrupt them if it can be done remotely.
Re: (Score:2)
This is a head admin with root access. Of course he deleted any backups as well.
Backups (Score:3)
Needless to say, that's why you have a rigorous and frequently tested backup system. Such an extraordinary internal attack is almost certainly going to create downtime and chaos, but you can at least partially mitigate that. At the same time, with such a large firm, one has to ask why one person has sufficient privileges to do that much damage. Reality is that if someone has that level of access, right up to root access on servers, they probably have the ability to muck up backups as well. So lots of things have gone wrong here. This isn't a defense of the asshole who did it, who sounds like he got what he deserved, but that he was able to do it and cause such widespread damage indicates some seriously sloppy security on his employer's part. This isn't the first time we've seen a disgruntled IT worker smash systems to pieces.
Re:Backups (Score:4, Interesting)
As you say if you know what you are doing the backup would be toast too. If you are not stupid, in 2022 it looks like a ransomware attack coming from outside any machine you have legitimate control of and you are in the clear.
Look if you are going to bad things first hack a computer elsewhere, then hack your target proxying through that machine. When the hack is over nuke your proxy machine. For even greater security make your launch machine say a Raspberry Pi and after the hack physically destroy the microSD card.
None of this is fricking rocket science.
Re: (Score:2)
If that IT organization is competent, then in 2022 there are write-once backups and/or offline backups (and also off-site) and these are managed by a different team, specifically because of ransomware and insider threats. Those people did it on the cheap and it became hugely expensive.
Re: (Score:3)
Disasters serve the purpose of motivating a company to pay more attention to how things are done. Hindsight is common, foresight is rare.
Backups? (Score:3)
Was there prison time for the bloke who said there were backups and they had been tested? :P
Re: (Score:1)
Re: (Score:2)
Don't know, but before you ask that question shouldn't you ask if the backups existed and worked? No where does it say they didn't. Just that some guy trashed a server and was sent to prison for it.
Stupid move (Score:4, Insightful)
It's okay to hate your job, or your supervisor, or your coworkers. But if that's the case, even if they're all dicks... you should act like a professional and just quit. Don't throw a tantrum, and don't destroy your own future - even if you think it'll feel good in the very short term.
Be the adult in the room.
Re: (Score:1)
Re: Unfortunate maybe.. .. but stupid? No (Score:3)
Re: (Score:1)
Re: (Score:3)
Some people end up in shitty jobs because they're shitty employees, and that's all they can get.
Re: (Score:1)
I'm pretty sure I could kill a man (Score:1)
Re: (Score:2)
Testing backups... (Score:1)
This is why you should integrate your production backups into your QA environment by baking in, and automating the obfuscation/munging operation, so you're constantly testing your production backups... even beyond the once a year when the auditors force you to do it and show them a screenshot...
Re: (Score:2)
Basic steps are missed all the time. "We can't afford that!" is a typical response I get.
I hope the company learned its lesson (Score:3)
Have better offboarding protocols.
Re: I hope the company learned its lesson (Score:4, Funny)
Re: (Score:2)
Better offboarding, as in, "walk the plank"?
No better offboarding of employees when they leave the organization. (ie, disabling their accounts, changing passwords they had access to before they walk out the door). It sounds like this guy left and they didn't change the root password to the database server. That was dumb, but then again there are many companies that have this problem because they are too lazy to fix it.
Quit bitching about backups (Score:3)
Anyone bitching about backups obviously did not look at the scale of the company
Its going to take some time and money to rebuild from backup media which is usually large but slow
Re: (Score:2)
Actually, that is why a company this size should run regular recovery tests. I guess they are not regulated or that would be a requirement. Because you always find something does not work when you try to restore for the first time or after not having done a real test for a while. My guess is that they had to replay transaction logs as well, 30k is more than just getting a DB dump back in. And they probably had never done that and it went wrong in the first few attempts.
But yes, companies are sloppy. Some ti
Re: (Score:2)
Backups that take more than, say, a week to restore are not fit for purpose. If you have a very large database then your backup system needs to have the bandwidth to restore it completely in a reasonable amount of time. The definition of "reasonable" doesn't change with the size of the database. Disaster recovery plans should consider this issue.
Re: (Score:1)
What if angry IT guy delete backups ? (Score:2)
Re: (Score:3)
The right backup is. Offline or write-once and off-site with multiple copies. Managed by a team only responsible for that with 4-eye principle for access. Yes, that is more expensive. But it is not rocket-science.
Immutable cloud backups taken hourly solves this (Score:2)
Re: (Score:1)
Re: (Score:2)
Indeed. My guess is that the 30k were because they had to replay transactions logs because the backups they could get were older. And they may have never done that and needed assistance.
Idiots abound! (Score:5, Insightful)
The real mystery isn't how a single rogue employee crippled the company, but how a 6 billion dollar company ended up with a ten cent disaster recovery plan.
It seems that even had this guy not been malicious, a clumsy employee with a cup of coffee could have accidentally done the same thing.
Re: (Score:3)
If this only cost them $30k to restore, they probably had a pretty good backup plan in place.
It's rarely just a case of restoring a database snapshot. They need to figure out which is the most recent good snapshot and what data between then and when he started deleting stuff has been lost. Often drives fail when restoring data, it's a stressful process for them and ZFS/RAID rebuilds take time.
They might not have wanted to overwrite the production servers either, to preserve evidence or perhaps do data recov
Re: (Score:3)
What's wrong with the plan? The news here is that someone went to prison. A company mistake costing $30k directly to recover for a 6 billion dollar company isn't news. It's actually evidence that a disaster recovery plan is in place.
What? (Score:2)
"It seems that even had this guy not been malicious, a clumsy employee with a cup of coffee could have accidentally done the same thing."
I reject the premise of your use case. Nobody builds things in a way that shorting out a single console will lead to catastrophic failure.
Well... okay. If you're in Star Wars you engineer a platform built on lava flows so that nudging a panel causes complete structural failure... but nobody else does that!
Lucky he wasn't shot. (Score:2)
There have been serious crimes in China where the results aren't so nice. [theatlantic.com]
Still, he can have years to perfect his skills in making Christmas lights [independent.co.uk] to keep us all happy in the coming years.
Wrong headline! (Score:5, Funny)
Re: Wrong headline! (Score:2)
Reading this, I heard Dana Gould saying an adapted version of his classic Black Dahlia joke punchline.
And to this dayâ¦police still donâ(TM)t knowâ¦what it wasâ¦the company did to deserve that.
Re: (Score:2)
Who would believe Bing had such capabilities when it can't even return a simple search result.
Or did they Bing "DELETE FROM Employees WHERE Name='Al';
and Bing helpfully replied:
Showing results for "DELETE FROM Employees WHERE Name=All;
Re: (Score:1)
I simply don't understand. (Score:2)
Re: (Score:2)
You used to use DAT? I use DIS because DIS is better dan DAT.
/sarcasm
Also Indian consulgint companies (Score:1)
I've worked with quite a few in the last decade. "We can do that, no problem!" does not include ever restoring a production system from scratch. It includes a lot of "I pushed the button on the GUI!!!"
Backups are missing the point... (Score:2)
Root accounts for organizations really need to require multiple authorizations to login, and every action/program/script that is run should require the other authorizers to confirm, so that there is always supervision.
Re: (Score:3)
It sounds like he was basically the only one at the top in the IT there? And his complaints and suggestions were being ignored. Many of his complaints were specifically regarding security problems that the management refused to take action on.
And it may exist somewhere somehow, but I've never seen two-person root login anywhere before.
It sounds like the fairly typical case of overworked, understaffed IT getting sick of
The Secret (Score:1)
The secret to this kind of thing is how you set it up. I won't delete anything at work. However, if I die or am fired I am quite sure the system will fall apart on its own. I'm the only one that does maintenance.
Two-pronged approach (Score:1)
Hope the company deletes my accounts when I leave (Score:1)
Because if they don't and someone breaks in using those accounts somehow, they'll be looking for me.
I started at one place where the previous admin had left so there was no overlap. The one before that had been fired 1-2 years prior. The fired employee's login and SSH keys were used to access all the customer facing systems in the cloud.
It was hard coded into all kinds of things and I had to follow the spaghetti strands to change it. It took awhile but it was better protected against any ex-employee and