Internal 'Set Of Blunders' Crashed Australia's Census Site (cso.com.au) 92
Slashdot reader River Tam explains the crash of Australia's online census site, citing the account of a security researcher who says IBM and the Australian Bureau of Statistics "were offered DDoS prevention services from their upstream provider...and said they didn't need it." From an article on CSO:
The ABS and IBM gambled on a plan to ask its upstream network provider to block traffic from outside Australia in the event that a denial-of-service attack was detected... Offshore traffic to the site was blocked in line with the plan, however, another attack, for which the ABS had no contingency to repel, was directed at it from within Australia. The attack crippled the firewall and the census site's operators opted to restart it and fall back to a secondary firewall. However, they forgot to check that it had the same configuration as the primary firewall. That crippled the census site.
In an unfortunate confluence of events, IBM's security warning systems started flagging some unusual activity, which indicated that information on the ABS servers was heading offshore. The site's operators, thinking the DDoS activity was a distraction, interpreted the alarms as a successful hack...these were little more than benign system logs and the technical staff monitoring the situation poorly understood it. Amid the confusion they naturally erred on the side of caution, [and] decided to pull the plug on the site...
In an unfortunate confluence of events, IBM's security warning systems started flagging some unusual activity, which indicated that information on the ABS servers was heading offshore. The site's operators, thinking the DDoS activity was a distraction, interpreted the alarms as a successful hack...these were little more than benign system logs and the technical staff monitoring the situation poorly understood it. Amid the confusion they naturally erred on the side of caution, [and] decided to pull the plug on the site...
That would be (Score:1)
blunders from down under.
Re: (Score:1, Funny)
Can't you hear, can't you hear the blunder?
IBM you say? (Score:1)
Let me guess, "the technical staff monitoring the situation poorly understood it" were needful-doers from IBM.
Re: IBM you say? (Score:1, Insightful)
Just think of the money they'll make from fixing the problem and consulting fees for handing it off to another vendor.
Re: (Score:3)
The part that was hosted by IBM (static files, etc) is the only part that didn't go down in flames. Why did they host only the static files? Because they didn't have the Australian-specific certification for cloud computing. So the Australian government opted to host this thing on their own servers. A piece of shit solution, but a certified one.
Re: (Score:3)
Where did you get that from?
Everything else I've read disagrees with that and says that IBM was hosting the VMs for the ABS.
Re: (Score:3)
Where did you get that from?
You must be new here. On Slashdot, you don't need to be right, you just have to sound right to get mod points.
Re: (Score:2)
Where did you get that from?
You must be new here. On Slashdot, you don't need to be right, you just have to sound right to get mod points.
I see you went for the other strategy, which is to accuse other people of making stuff up because you are yourself too lazy to even read TFA.
Re: (Score:2)
Where did you get that from?
You must be new here. On Slashdot, you don't need to be right, you just have to sound right to get mod points.
I see you went for the other strategy, which is to accuse other people of making stuff up because you are yourself too lazy to even read TFA.
You must be new here. On Slashdot we don't read TFA before posting ;-)
Re: (Score:2)
Where did you get that from?
You must be new here. On Slashdot, you don't need to be right, you just have to sound right to get mod points.
I see you went for the other strategy, which is to accuse other people of making stuff up because you are yourself too lazy to even read TFA.
You must be new here. On Slashdot we don't read TFA before posting ;-)
Yeah I usually do that but this time I wanted to see if it was another debacle caused by low-cost offshore rent-a-sysadmins. Turns out it's not, it was caused by expensive IBM rent-a-suits and/or somewhat expensive local civil servants.
Re: (Score:2)
A bit over five years ago IBM Australia had the majority of their sysadmins spend a couple of weeks in China to train their replacements and then laid the Australian sysadmins off.
Re: (Score:2)
Also I did not accuse you of "making stuff up" - it's interesting that you are suggesting that I did instead of what I did do which was ask a question. Why do you think I was accusing you of making things up? Should I be assuming you are instead of just having information I have not seen?
Re: (Score:2)
Where did you get that from?
Everything else I've read disagrees with that and says that IBM was hosting the VMs for the ABS.
That "everything else" can't be much because this comes from one of the linked articles in the summary.
The clarification here is that many people have been saying IBM was hosting the e-Census website. According to our source, this is not strictly correct. IBM provided a content distribution network (CDN), running on SoftLayer, for static content such as fixed text and images. This is similar to the services Akamai provides with clustered nodes distributed across the world.
But the actual e-Census application, which operates dynamically is not hosted by IBM. Our source suspects the application is being hosted on ABS' own systems.
http://www.cso.com.au/article/... [cso.com.au]
SUSPECTS? (Score:2)
How is "Our source suspects" proof? Other articles have been referring to IBM hosting that lot because the ABS just does not have anything close to the infrastructure to do it in-house and a proposal to acquire more servers was denied last year.
Re: (Score:1)
No. People have until the 23rd of September [abc.net.au] to complete the census and you won't be fined [abc.net.au] if you didn't complete the census on the night.
Re: (Score:2)
Give yourself to the dark side. It is the only way you can save your privacy.
Re: (Score:2)
It's okay, it isn't like anyone ever tells the truth on the damn thing.
According to it, this time, I have changed my religion from Jedi to Sith.
I don't know about Australia. In the UK, the whole point of the Jedi thing was to point out that the only part of the census that was optional and didn't need to be truthful was the religion question
What DDOS? (Score:4, Insightful)
I still haven't seen any mention of evidence that there was any attack at all. Well, except in the negative sense, as in "Global DDOS sensors failed to register any attack [cso.com.au]".
From the server's point of view, what exactly is the difference between "a DDOS attack from within the country" and "ten million users trying to log on to the site within one hour"?
Re:What DDOS? (Score:5, Insightful)
Arguably if the census servers where nullrouting traffic from uoff-site, that might well explain why nothing showed up on those maps.
Regardless, a DDOS seemed like it was innevitable. The stupid and anti privacy decision to store identifying info (Names, etc) with this census despite widespread condemnation from academics, activists and security researchers (at least 9 senators from across the political spectrum are refusing to fill it in citing the leaked papers from the bureau stating they want names and addresses to create "saleable products", ie selling peoples personal info.
Of course Anonymous or someone of their ilk was going to take umbrage and attempt to sabotage the whole thing.
DDOS? More like a self inflicted slashdotting. (Score:5, Informative)
In previous years, they had been quite careful to inform people to pre-fill their form before census night, and submit after. This year they were expecting only a minor increase in peak traffic.
Then they go and blast the message, "Fill in your form online, ontime or face massive fines", all over the media.
So what did we all do? When the majority of 9-5 workers got home, we all tried to login and submit at about the same time.
Sure they screwed up their network config, but it was a combination of poor planning and poor communication that triggered the whole mess.
Re: (Score:2)
It was also very concerted "census night". Tell everyone to do it on that night and they'll do it on that night. Doesn't matter though it shouldn't have been hard to handle that in this day and age. ABS SES Executives were just idiots.
Re: (Score:1)
Re: (Score:2)
Wouldn't an address be a really bad choice for an identifier?
Re: What DDOS? (Score:2)
It does not use the address. It uses the 2nd, 3rd and 5th letters of the last name, 2nd and 3rd letters of the first name, date of birth and sex. For shorter names it pads with "2". You can fake it into reporting "name withheld" by putting a "9" in each of those positions in the name.
Re: (Score:1)
I'm quite certain that the ABS (you now, professional statisticians) could have and should have predicted the load.
Re: (Score:2)
Well, technically it is a DDoS. And they brought it onto themselves by pretty much demanding that people participate in it. So ... who's to blame for it?
Re: (Score:1)
Re: (Score:2)
Damned if you do, damned if you don't...
Online voting (Score:5, Insightful)
There's some good news here. This ABS blunder sets the likelihood of paperless and/or online voting happening in Australia back another decade or so.
It's probably weird that as a technology geek I'd be a fan of paper voting, but paper forms are a lot harder to hack or manipulate without a trace.
Re: (Score:2)
And it's far harder to do it in a way that nobody can find out. Plus, it is far harder to cry foul play without having to prove it.
Re: (Score:2)
It's bad news really.
It should be trivial to do a good electronic voting system. Electronic touch screen to do your vote and count it, with a two paper receipts printed for you to check. One you keep for yourself and the other you put in a ballot box for recounting. The electronic count would provide the main count on the night to indicate the result, the paper receipt is available for recounting with scrutineers and the voter can verify that the receipt matches their intentions.
However if they did do it, t
Consequence of not having a Social License (Score:3, Insightful)
In Australia the phrase 'Social License' is starting to register with the wider community. Issues such as the coal seam gas mining and a range of unpopular but otherwise legally compliant initiatives are feeling the backlash from ordinary people.
People may think that the 'Brexit' phenomenon is new, however there is a growing discontent among the wider population with the small but influential groups that ignore the views of the community affected by these schemes.
I wouldn't support the alleged DDOS attacks on the ABS web site, however the ABS has moved ahead with changes to its data retention policies without considering the associated risks, and even well known politicians are refusing to cooperate with the Census.
You can imagine the executives at the ABS discussing their planned changes and asking "what will people do if they don't like the changes" - well now they have seen what could happen.
It's more than likely that the Chief Statistician (on over $700,000 a year) will be asked to resign. It's difficult to sack him (a quirk of the legislation that created the ABS) however you would not expect that a person on such a salary would show such poor judgement.
The 'Brexit' phenomenon has only begun to unfold, and you can only hope that people look past the technology issues surrounding the ABS Census debacle and start asking the question - if you don't have community support is your idea actually any good?
Re: (Score:3)
"Denial of service attack" by means of cutting resources and by people in politics pushing a scare campaign to get people to all log in on the same night in fear of being fined for doing it a day late.
Re: (Score:2)
"Meaningless decisions" like not employing enough people to do a task you mean? Why yes, indeed they are incapable just as you would be when confronted with a task you do not have the resources to attempt.
Re: (Score:2)
You can imagine the executives at the ABS discussing their planned changes and asking "what will people do if they don't like the changes" - well now they have seen what could happen.
Actually I would really expect such a question to not be asked at all. Rather, a question like "what if people don't like it" would be filed in the "doesn't matter" pile.
Re: (Score:2)
It's difficult to sack him (a quirk of the legislation that created the ABS) however you would not expect that a person on such a salary would show such poor judgement.
What poor judgement?
The poor judgement of a person who is completely new to a role that has been vacant for long enough that the entire division falls into leaderless disarray?
Or the poor judgement of someone desperately treading financial water after successive governments collectively have managed to slash $200million off the budget for a division that originally had it's budget increased because it couldn't actually afford to hold the last census?
I'm sure he'll get fired. But it will be because he's a ni
Re: (Score:1)
I'm guessing you're one of these people that still hasn't filled out the actual census and realised that all along there was a checkbox at the end ASKING FOR YOUR CONSENT to store your name. JUST LIKE IT HAS ALWAYS DONE.
But sure, keep claiming that the thing you've made up in your head somehow has something to do with the site falling over.
anti-DDOS is like real estate (Score:5, Insightful)
It is all about location, location, location...
My employer is on a state-wide network that connects, among other things, a ton of colleges and universities. After some recent BLM events, there were sympathy DDOS attacks from anonymous or whoever, so the state just spent millions on fancy new anti-DDOS gear on the external side of all of their POPs.
A few weeks ago, I had an opportunity to ask the state's Chief Information Security Officer what their plan was to handle internal attacks coming from the colleges, which are inside the perimeter, and typically have incredible switching and routing capacities (as part of I2), far in excess of anything our rural fiber rings could handle. It took him a few seconds to review the topology of the network in his head before he realized that we'd be screwed.
I have some sympathy for Australia. DDOS is a hard problem to solve, even if you've got millions to spend on the newest, shiniest gear.
Re: (Score:3)
30+ hours to get your password / receipt emailed to you. 20 minutes, maybe 1 hour is acceptable.
Started my census return on the 12th at 17:05 and requested the password be emailed to me.
The password email was sent by the ABS servers in the 14th at 03:50.
If I was depending upon the password to resume doing the census I would have had to wait an additional day.
Additionally the forms really didn't handle doing "Father", "Daughter", "Wife". Had to go and delete all the data entry for my daughter. Add my wife
Re: (Score:2)
Why are you entering anything more than: 1 female, under-18?
Does the form record full contact information about your daughter? And you voluntarily supplied it?
Re: (Score:2)
No the form does not record her contact information. The only contact information was mine.
The relationship information however is not collectable unless you enter the people in the household in specific orders. i.e. the form was poorly defined as there were no instructions as far as I could see about entering people in specific orders.
Re: (Score:2)
I see, cheers for the explanation.
I've filled in many Government issued forms where the instructions are not very clear. Unsurprising to hear the online ones are no better.
Re: (Score:1)
Off with his head! (Score:3)
The prime minister Malcolm Turnbull went on the record to say that he will punish those responsible.
Yet it was the coalition government that cut the ABS budget by $68m, left the department leaderless for a year, and also poked the bear with talk of selling citizen information to make money which may have prompted the attack in the first place.
The only question is who will be the scapegoat.
Re:Off with his head! (Score:4, Insightful)
So you believe this census debacle happened entirely in the last 3 years? even though ABS get specific budget for census and they have been planning current systems for way longer than 3 years.
not saying current government is good, but you are completely full of shit and making excuses for bad decisions made by ABS and IBM.
Re-read my post. The budget cuts were a direct driver for the ABS to attempt to make money by selling data, which is exactly what people are blaming the DDoS for.
As for bad decisions, they are very easy to do when you have no oversight or leadership, a role which was left vacant for over a year by the government.
As for planning the current systems and funding, it may be worth looking into the time line. IBM was a contractor selected for outsourcing a good 8 years ago. That project was then put on hold indefinitely. It was restarted after the announced budget cuts as a way of reducing costs by the ABS by not having to manage their own servers.
ABS is a government entity. I didn't make excuses for bad decisions, I laid blame. When you fuck up something this glorious and affect every person in the country, the buck rests at the very top. Systems were in place that let this happen, e.g. why would the federal government partner with a company that has been blacklisted is a government supplier by two Australian states without some form of inquiry?
Re: (Score:2)
Regardless of any politics, the large, highly promoted public census was always going to be a huge DDoS target.
Re: (Score:2)
Just like the DDoSes that didn't happen 5 years ago when we also were able to fill it out online? Or the yearly tax return? Or the censuses that work online the world over without a hitch?
Please don't make excuses for them, they come up with their own lame ones already.
Re: (Score:3)
Labor had also done cuts in the previous years to the tune of $45 million and had plans for more cuts had it won the election
Oh I'm sorry. You seem to be under the impression that my comment was partisan. That doesn't change what happened. A series of budget cuts, lack of management oversight, decision to outsource to save costs, and then entice the DDoS by making the unpopular move to sell data of citizens is a bad call that could have been made by any government.
The only reason I mentioned the libs at all was that it was the current lib leader looking for a scape goat despite his hands being covered with digital blood from the
Re: (Score:2)
The prime minister Malcolm Turnbull went on the record to say that he will punish those responsible.
And then those responsible for the sacking, will be sacked.
Dutch police (Score:1)
This sounds similar to Dutch police, who put out a press release that there website was having trouble because they where being hacked.
In about half a day they found out that they added a 40 MB JPG on there front-page and scaled it to a thumbnail using CSS....
Retrospective self serving techno excuses (Score:2)
The Force was not with them. (Score:2)
Re: (Score:2)
I don't get what the fuss is about. To be honest, as I don't watch free to air or listen to it, I missed the whole must fill it in on the night scare tactic, I was expecting the book to come and as it didn't, missed the whole thing, until it all fell over. They asked bugger all questions and let's be real about this, if you make a mistake filling it in, they can not fine you and yes, I am still a Jedi and will be for as long as they ask that particular question in a secular state.
Nobody ever got fired for buying IBM (Score:1)