White House CIO Describes His 'Worst Day' Ever 333
dcblogs writes "In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago. There were no redundant email servers."
Floppy... (Score:1, Insightful)
All my machines have a floppy.
I don't understand how adding a peripheral can make the machine "worse"?
No surprise (Score:3, Insightful)
They inherited a system that "lost" months/years worth of emails during the Bush administration. Of course it all sucked, it was designed to.
Still blaming Bush? (Score:1, Insightful)
I have to say I got a chuckle when I got to the part about "inheriting" their IT problems. Obama "inherited" all his problems after all!
Similar situation... (Score:5, Insightful)
I recently took over for a staff which had been interned in their positions for the better part of a decade. Out with the old in-house staff, in with the new outsourced IT 'team'.
I can easily see how this happens, outside procurement and ineptitude problems on the part of the previous WH IT staff. When you've got what amounts to 'institutional knowledge death', with the institution carrying on, you've got to over-staff for some time or things fall apart completely while you play catch up. With a situation where you don't understand it all, are under staffed or under skilled, you're faced with only a couple options when you come in behind the curve, with aging equipment and software: you either start replacing everything you can, as you are able, as quick as you can, or you start suffering outages. It's even worse if things are mismanaged and things are failing all around you.
As for the claims of the article? Meh. I'm actually not that impressed by his claims to the poitn where I think 'this is bad':
In 2008, "floppy drives" weren't all that uncommon. I remember servicing Core machines which had floppy drives, still. We're not talking biege boxes with ISA slots here, necessarily - with a 4 year replacement schedule for desktops, floppy drives don't speak of ineptitude.
The 80-hour-week thing means nothing. It might mean he was understaffed, or that he's a workaholic. To me, it sounds like the meaningless words of a political appointee.
"Over 82% of the White House technology had reached end of life" means nothing. If they were on a 3-year replacement schedule for desktops and they had 10/100 switching, I can easily see where you'd come to that number.
He had one "data center", with no redundancy. A bit of a contradiction, yeah? This is made somewhat less impressive by the fact that this administration, in particular, was a bunch of Nancys when they came in with "oh woes, look at this mess", quite obviously overstating things for dramatic media effect.
"Our email servers went down for 21 hours" isn't a statement of disaster, it's a statement of ineptitude. If they got the mail servers back up, with the data intact, the problem wasn't with the environment but the people involved (or the lack of staffing). His BB starting to have mail incoming suggests a reinstall wasn't required, so safe to say BES was OK, so who knows what the real 'problem' was which caused a day of outage...
Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%. Realize that a) this is a political appointee talking, b) it's a seemingly non-technical manager (he's up in his datacenter, lookin' for redundancy!), and c) this is the government we're talking about, after all. Anyone who's had any dealings with them on a technical level realizes that 'setbacks' and 'shortcomings' or 'difficult problems' or the like are (probably!) due to ineptitude. Yes, sadly, even amongst the elite (though not necessarily of their own doing - thank you bureaucratic bullshit).
Granted, this may not have been the case when BO came to the WH and took over. They may have had previous IT staffers who stayed through the transition, but I'm guessing they did not (due to political mistrust issues). It could've been a genuine clusterfuck. Sometimes it's nothing and people cry about the sky falling as they pull down the curtain; sometimes, it really is bad. (If you understand weather patterns, you may recognize a summer storm to not be the disaster that chicken little claims...)
Re:No surprise (Score:5, Insightful)
Of course it all sucked, it was designed to.
It wasn't originally designed to suck, but when you refuse to spend money on infrastructure improvements,
you end up spending your time putting out fires instead of making improvements.
This applies equally to computer hardware/networks as it does to our highway/bridge, electrical, and water infrastructures.
FFS, there are critical metal pipes in DC's water distribution network that date back 150 years to Lincoln.
Re:No surprise (Score:5, Insightful)
Well and as I have learned the hard way lately, if it's going to cost 500k per year to run IT for a couple of hundred employee outfit when it's government money, someone will complain. When I did private sector stuff the biggest issue was downtime, a million dollars, no problem if that means good uptime. I used to go into insurance companies and banks at 4pm, the regular staff left at 5 -5:30, if it wasn't ready to go the next day by 8 or 9am you were in seriously trouble. In government it's all about how much money they have to explain to some jackass who wants to make political hay out of it.
The way I count it from http://www.washingtonpost.com/wp-srv/opinions/graphics/2006stafflistsalary.html the white house has about 400 employees. Figure 350k a year in desktop computers alone, for IT staff, another couple of hundred K in 'mobile' and accessory devices, ancillary office equipment you could easily be looking at 1.5 million or for just the non classified IT stuff. That isn't, in the grand scheme of things, a lot of money, but you have to know that whomever isn't in charge is going to want to curtail that spending, because it's 'wasteful'.
(how you count IT spending can vary wildly. When you're up into that many people you have a lot of dedicated IT staff in various sub groups who may or may not count towards the total and so on). On top of the mess that would be trying to deal with 400 spoiled brats who want everything their way (I'm sorry, executives who want to maximize their productivity), you have to try and plug into everything else in government and have the secured computers/networks as well. That isn't cheap.
Re:Not a bad number (Score:5, Insightful)
23% down sounds about average for MSExchange servers.
Only on slashdot could such ignorance get modded up.
On a bad bad day as a consultant, I have to fix scenarios with Exchange where everything blew up and theyre down for a single day-- MAYBE 2-- out of several years uptime.
Thats with the clients who have no full time IT staff whatsoever and a shoestring budget.
Possibly if you have no idea what youre doing, or dont know anything about exchange, then yea 23% might be an OK guess.
Re:Not a bad number (Score:2, Insightful)
Youre free to pretend we still live in a day where it is unnecessary to do group scheduling through email, but you would be wrong.
Re:No surprise (Score:5, Insightful)
Just a tip, if you ever want people (outside of a small echochamber) to take you seriously, you may want to grow up and stop referring to GW Bush as "Dubyah"-- its about as mature as calling Microsoft M$, or someone you dont like a doo-doo head.
Re:Not a bad number (Score:1, Insightful)
Group scheduling and email are different applications. Combining them in one backend is shortsighted.
Re:Indication of Government Ability? (Score:5, Insightful)
Oh SHUT up.
The reason government can't get anything done, generally, is there's always some jackass out there questioning whether a thing is needed because it happens not to be exactly what they want, or why workers cost anything at all since their life is in the shitter so why should a government employee make money either?
There is a significant interest in this country in starving government, and then mocking it for under-performing. That's a combination of arguments only an imbecile would make.
White House IT: thumbs down (Score:4, Insightful)
From what I've read, there's only one firm that does White House transitions. I think it's Bechtel, but it's been so long that I've read anything about transitions that I have around a 15% confidence level in that piece of data.
Google "white house transition" and you'll see that it's a total mess. If you want to read about it, there's info here:
http://whitehousetransitionproject.org/ [whitehouse...roject.org]
From what little I've read, you basically get a mostly empty building (the White House). It's up to the team to build/rebuild the infrastructure...but as any operations person knows, IT infrastructure is usually way behind everything else. The general executive branch IT has been a low priority for decades. What's more important, email or setting up the phone so the president can call someone (or someone can call the president)?
At that point, the team is probably so far behind that they're screwed continuously for the one or two terms.
Are the guys running the systems any good? I'd ask you: how many of you could pass a background and attitude check? You think the process etc at your workplace is bad, imagine how bad it is in the Executive branch.
That said, it might be fun...but it's probably a nightmare. "I can't print out this $15 billion dollar appropriation because the f*cking printer doesn't work!" "People in PA are starving because the email server ate all of our emails!"
Every minute is a crisis, with everyone breathing down your neck 24/7. Does that sound like something you'd want to do for 24/7/365/4 years?
Re:You get what you pay for (Score:5, Insightful)
If a system is end of life and dies can you say it was too bad or crappy without knowing how long it was kept running?
Everything has a shelf life and must be either upgraded or replaced eventually. Even the Large Hadron Collider is nothing more than a replacement of the Large Electron–Positron Collider before it which reached the end of its useful life. I had a similar discussion with an engineer at the industrial plant I work it. We have a vibration monitoring system which died and needed replacing, and he also called the system "crap." For some perspective the system was obsolete in 1995. The two subsequent models are now also obsolete yet this thing has been humming away just fine for 17 after the vendor stopped supporting it.
Yet someone called it crap.
Re:Floppy... (Score:5, Insightful)
Office politics is really no different to real politics, the vast number of people who work for large organisations be they private, public or charitable are for the most part reletively efficient at whatever it is they do, but one or two clowns in the wrong position can turn the whole thing into a circus. In an evolutionary sense large organisations exist because they can do what no man can do alone. However our tribal instincts are still evolving such that we can live with and within groups of more than ~150 that are required to produce what a single mind can imagine, large groups (civilization,cities) simply did not exist until we invented agriculture and yet our current civilization(s) cannot function without them.
For example the multi-national I work for has about 175K people, a death in that "tribe" would happen quite frequently (say one a week), but it's only the handful of people I personally work with that I care (or even know) about. I think the fact that telecommunications have gone from simple morse code to their current star trek capabilities is part of that evolution, we are tool-makers, it's in our nature to invent tools to overcome the problems caused by inventing tools. So in a way that will probably upset bioligists it can be said that our tools and our instincts are co-evolving to accomplish greater feats, but our tools are evolving at a geometrical rate whereas our base instincts evolve at a glacial rate. So I'm betting our tools will evolve to the point where the size of the organisation is (almost) irrelevant to the effectiveness of its internal organisation long before our pumy minds can name, let alone care about, 175k individuals.
Re:I would deploy a Domino cluster (Score:4, Insightful)
I would deploy IBM Domino like in the days of Clinton, which Bush switched to Microsoft Exchange. Reliability went downhill with that decision.
Domino runs cheap and fast and reliable. And has always active clustering so you don't have to deal with downtime. IBM simply has a much longer track record of delivering reliable computing than Microsoft.
People are not replacing Domino with Exchange because it is more reliable. They do it because *everybody* hates Lotus Notes.
This being said, any user that has a complaint against Lotus Notes should be required to work with Groupwise for a week.
Re:Not a bad number (Score:5, Insightful)
You miss the fact that he INHERITED that system. That's how politics works. The lame duck guy in charge of the White House (which would be the former President DIRECTLY) let the thing rot...
Part can be attributed to the old staff being done with the position and the new guy will just buy new stuff anyway... Almost fair?
Part of all the downtime was a FEATURE that the pervious administration used to their full advantage... They were über controll freaks... Controlling information of their own "trusted" employees was part of D.C.'s daily routine. The rot was deliberate to stop communications from being added to the archives.
Re:Not a bad number (Score:5, Insightful)
Group scheduling and email are different applications. Combining them in one backend is shortsighted.
True, but Exchange isn't an "email" application--it is a group productivity application that includes email, group calendaring and scheduling, tasks, and collaboration.
I understand there are MS haters who will bash Exchange relentlessly, with any label on it, but let's try to be even a tiny bit accurate. Exchange isn't an "email" service and hasn't been exclusively that for nearly 15 years: Time to come up with some new criticisms, the old ones don't apply.
Re:Not a bad number (Score:5, Insightful)
More to the point, businesses haven't wanted an email service for nearly 15 years.
They want the group productivity application. But they don't call it that because the most visible part - the part they really see - is the email.
Re:Not a bad number (Score:4, Insightful)
in my experience, Zimbra is a bloated pig with its share of availability problems as well. I hate the whole "let's take a bunch of open source pieces but just throw them together as an inflexible blob of crap in /opt" approach. The installer leaves a lot to be desired as well, with key components around setting proper permissions resulting in an install that will never work until you manually fix it.