Forgot your password?
typodupeerror
Government United States IT Your Rights Online

White House CIO Describes His 'Worst Day' Ever 333

Posted by Soulskill
from the need-a-bipartisan-effort-to-swap-out-a-machine dept.
dcblogs writes "In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago. There were no redundant email servers."
This discussion has been archived. No new comments can be posted.

White House CIO Describes His 'Worst Day' Ever

Comments Filter:
  • TFA indicates clearly not only that there's no urgent need for IT geeks to unionize, but also what havoc they could wreak if they ever did.

    • by ryanov (193048)

      My union covers our IT personnel (I am chair of the union's IT committee).

      • Awesome. Hope to Möbius you guys never think of accepting contracts from rival companies on the side.

        ...Well, I guess you have now. (innocent look)

  • No surprise (Score:3, Insightful)

    by Anonymous Coward on Tuesday March 13, 2012 @11:16PM (#39348583)

    They inherited a system that "lost" months/years worth of emails during the Bush administration. Of course it all sucked, it was designed to.

    • by Vladius (2577555)
      Working as intended...
    • by rhook (943951)

      Big surprise. The contract (like all government contracts) went to the lowest bidder and they surely cut corners.

      • by ryanov (193048)

        You say this as if it is stupid, when surely you know it is the faux outrage from taxpayers that causes this shit.

    • Re:No surprise (Score:5, Insightful)

      by TubeSteak (669689) on Wednesday March 14, 2012 @12:14AM (#39349005) Journal

      Of course it all sucked, it was designed to.

      It wasn't originally designed to suck, but when you refuse to spend money on infrastructure improvements,
      you end up spending your time putting out fires instead of making improvements.

      This applies equally to computer hardware/networks as it does to our highway/bridge, electrical, and water infrastructures.
      FFS, there are critical metal pipes in DC's water distribution network that date back 150 years to Lincoln.

      • It wasn't originally designed to suck, but when you refuse to spend money on infrastructure improvements, you end up spending your time putting out fires instead of making improvements.

        Ding ding ding!!!!

        I think in Slashdotspeak, I should say "^This", and HARD.

      • by Darinbob (1142669)

        It's not that they refused, but it's a typical government problem. You think corporations have budget problems wait until you see the government. Everything is cut to bone. If it still works then there's no reason to upgrade, and if it doesn't work well maybe you can share with someone. Even if it might cost more to maintain, that's a person cost and not a capital expenditure.

    • Re:No surprise (Score:5, Insightful)

      by Sir_Sri (199544) on Wednesday March 14, 2012 @12:17AM (#39349023)

      Well and as I have learned the hard way lately, if it's going to cost 500k per year to run IT for a couple of hundred employee outfit when it's government money, someone will complain. When I did private sector stuff the biggest issue was downtime, a million dollars, no problem if that means good uptime. I used to go into insurance companies and banks at 4pm, the regular staff left at 5 -5:30, if it wasn't ready to go the next day by 8 or 9am you were in seriously trouble. In government it's all about how much money they have to explain to some jackass who wants to make political hay out of it.

      The way I count it from http://www.washingtonpost.com/wp-srv/opinions/graphics/2006stafflistsalary.html the white house has about 400 employees. Figure 350k a year in desktop computers alone, for IT staff, another couple of hundred K in 'mobile' and accessory devices, ancillary office equipment you could easily be looking at 1.5 million or for just the non classified IT stuff. That isn't, in the grand scheme of things, a lot of money, but you have to know that whomever isn't in charge is going to want to curtail that spending, because it's 'wasteful'.

      (how you count IT spending can vary wildly. When you're up into that many people you have a lot of dedicated IT staff in various sub groups who may or may not count towards the total and so on). On top of the mess that would be trying to deal with 400 spoiled brats who want everything their way (I'm sorry, executives who want to maximize their productivity), you have to try and plug into everything else in government and have the secured computers/networks as well. That isn't cheap.

  • Appropriate (Score:4, Funny)

    by sixtyeight (844265) on Tuesday March 13, 2012 @11:17PM (#39348601)

    Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives ...

    Considering the sort of people who are using these machines, it seems almost appropriate somehow.

    Maybe keeping them technologically underpowered is actually a good thing. Those crafty, crafty White House IT gurus.

    • by wvmarle (1070040)

      Besides, in an environment like the White House, I think it's more than just a good idea to keep a few PC's with working floppy drives at hand. Preferably down to 8" drives. Just in case you need to read some long-archived file, that has never been put on a more modern medium.

      • by scdeimos (632778)
        You laugh, but I recently had to find a working 5 1/4" disk drive and an old linux distro (modern ones don't support them any more) to recover files from a floppy disk for an astronomy society.
        • by ryanov (193048)

          What was removed that took out support? I'd think floppy controllers are still supported -- was there something special about those drives vs. current drives?

  • by gimmebeer (1648629) on Tuesday March 13, 2012 @11:21PM (#39348629)
    The problem is the procurement process. It takes a hell of a long time to get IT resources ordered, and often by the time they are actually put into service half of their warranty life-time has expired. It has nothing to do with a lack of knowledge on the OMB IT front, it's got everything to do with the red tape they have to cut through to make anything happen.
    • Well it sounds like a CIO to manage and streamline the procurement process is what is needed and well overdue. Obama did the right thing as past presidents were old and out to lunch in terms of technology.

      If they need to check for spyware crap and security then set a budget with interns providing the wiping of the hard drives 7 times with an IT department to provide the encryption and come up with procedures to retire and fix PCS and so on.

      I would think a job as important as the executive branch would be im

  • by forgottenusername (1495209) on Tuesday March 13, 2012 @11:33PM (#39348707)

    Too bad we probably paid billions for such crappy infra.

    • by thegarbz (1787294) on Wednesday March 14, 2012 @02:45AM (#39349761)

      If a system is end of life and dies can you say it was too bad or crappy without knowing how long it was kept running?

      Everything has a shelf life and must be either upgraded or replaced eventually. Even the Large Hadron Collider is nothing more than a replacement of the Large Electron–Positron Collider before it which reached the end of its useful life. I had a similar discussion with an engineer at the industrial plant I work it. We have a vibration monitoring system which died and needed replacing, and he also called the system "crap." For some perspective the system was obsolete in 1995. The two subsequent models are now also obsolete yet this thing has been humming away just fine for 17 after the vendor stopped supporting it.

      Yet someone called it crap.

  • by Anonymous Coward

    That's madness! Everyone knows that the floppy drive dictates the speed, quality, and age of the computer!

    • Yeah, it seems to me most Windows computers up until very recently came with floppy drives by default.

      Having a floppy drive is not an indication that a computer is out of date - unless that's the only drive type the computer has.

      • Yeah, it seems to me most Windows computers up until very recently came with floppy drives by default.

        Whats a floppy drive?

      • by Sancho (17056) *

        I want to say most vendors stopped shipping with floppy drives by default around 2003. Presumably there was a hardware refresh between 2003 and 2009. It seems likely, then, that the inclusion of the floppy means that the IT person specifically included floppies.

        The alternative--that there wasn't a refresh between 2003 and 2009--is, in fact, a bit sad. Six years is a pretty long time for computers, and their value increased considerably throughout last decade.

        An interested person could probably file a FOIA

      • They've been obsolete for years.

        From Wikipedia: [wikipedia.org]

        In February 2003, Dell announced floppy drives would no longer be pre-installed on Dell Dimension home computers, although still available as a selectable option and purchasable as an aftermarket OEM add-on. On 29 January 2007, PC World stated that only 2% of the computers they sold contained built-in floppy disk drives; once present stocks were exhausted, no more standard floppies would be sold. In 2009, Hewlett-Packard stopped supplying standard floppy drives on business desktops.

        • Re:Floppy Drives! (Score:5, Interesting)

          by kenh (9056) on Wednesday March 14, 2012 @01:33AM (#39349433) Homepage Journal

          OK, perspective is called for - Obama took the White House in 2009, up until 2009 HP had floppy drives STANDARD on business desktops - so as Obama took the White House, HP was still shipping floppy drives as STANDARD.

          Yes, sitting in 2012 we can all agree that floppy drives have been obsolete for years, but in 2009 HP was still shipping them as standard.

          The note about Dell Dimensions is nice, but those are "home" computers, not "professional".

          And that 6 year-old software? I can guarantee you it was Office 2003 - sure, as Bush was preparing to leave office his staff certainly could have gone around and upgraded everyone to the latest/greatest version of office (Office 2007), but it is now 2012, and the latest version of Office on PCs is 2010 - does that have 100% market penetration, or are there a few stragglers on 2007 or even 2003?

          Maybe, like most office users at the time, the Bush White House wasn't a big fan of the ribbon interface introduced in Office 2007 [wikipedia.org]

          • Yes, sitting in 2012 we can all agree that floppy drives have been obsolete for years, but in 2009 HP was still shipping them as standard.

            Not quite. In 2009, HP stopped offering standard floppy drives as an option. The Wikipedia article didn't say when they stopped including them as a standard feature.

          • The note about Dell Dimensions is nice, but those are "home" computers, not "professional".

            WTF is the difference between a "home" computer and a "professional" one? If anything, I'd expect home computers to be more powerful since professionals don't generally need to play the latest games...

      • by tlhIngan (30335)

        Yeah, it seems to me most Windows computers up until very recently came with floppy drives by default.

        Having a floppy drive is not an indication that a computer is out of date - unless that's the only drive type the computer has.

        Desktops maybe. Laptops though... I don't think I've seen a floppy drive standard on a laptop since the early 2000's.

  • 'first presidential Blackberry.'

    Perhaps also known as the last presidential Blackberry. [vancouversun.com] Of course, parts of the government still favor Blackberry [570news.com], but then apparently parts still like floppy drives too. With the recent /. posts on DOD Androids (not the kind that lead to Skynet comments) and the like, one wonders how much longer even this will last.

  • by Anonymous Coward

    The equipment of some government agencies I have worked for is kind of prehistoric. I you are using Windows 95 as a file server, you are in serious trouble.

  • Too bad they didn't teach George W Bush about IT stuff. With how often he went to Texas to clear brush, he'd have that exchange DB backed up in 5 places, compressed properly, and on SSD RAIDs :-P
  • by CAIMLAS (41445) on Tuesday March 13, 2012 @11:59PM (#39348899) Homepage

    I recently took over for a staff which had been interned in their positions for the better part of a decade. Out with the old in-house staff, in with the new outsourced IT 'team'.

    I can easily see how this happens, outside procurement and ineptitude problems on the part of the previous WH IT staff. When you've got what amounts to 'institutional knowledge death', with the institution carrying on, you've got to over-staff for some time or things fall apart completely while you play catch up. With a situation where you don't understand it all, are under staffed or under skilled, you're faced with only a couple options when you come in behind the curve, with aging equipment and software: you either start replacing everything you can, as you are able, as quick as you can, or you start suffering outages. It's even worse if things are mismanaged and things are failing all around you.

    As for the claims of the article? Meh. I'm actually not that impressed by his claims to the poitn where I think 'this is bad':

      In 2008, "floppy drives" weren't all that uncommon. I remember servicing Core machines which had floppy drives, still. We're not talking biege boxes with ISA slots here, necessarily - with a 4 year replacement schedule for desktops, floppy drives don't speak of ineptitude.

    The 80-hour-week thing means nothing. It might mean he was understaffed, or that he's a workaholic. To me, it sounds like the meaningless words of a political appointee.

    "Over 82% of the White House technology had reached end of life" means nothing. If they were on a 3-year replacement schedule for desktops and they had 10/100 switching, I can easily see where you'd come to that number.

    He had one "data center", with no redundancy. A bit of a contradiction, yeah? This is made somewhat less impressive by the fact that this administration, in particular, was a bunch of Nancys when they came in with "oh woes, look at this mess", quite obviously overstating things for dramatic media effect.

    "Our email servers went down for 21 hours" isn't a statement of disaster, it's a statement of ineptitude. If they got the mail servers back up, with the data intact, the problem wasn't with the environment but the people involved (or the lack of staffing). His BB starting to have mail incoming suggests a reinstall wasn't required, so safe to say BES was OK, so who knows what the real 'problem' was which caused a day of outage...

    Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%. Realize that a) this is a political appointee talking, b) it's a seemingly non-technical manager (he's up in his datacenter, lookin' for redundancy!), and c) this is the government we're talking about, after all. Anyone who's had any dealings with them on a technical level realizes that 'setbacks' and 'shortcomings' or 'difficult problems' or the like are (probably!) due to ineptitude. Yes, sadly, even amongst the elite (though not necessarily of their own doing - thank you bureaucratic bullshit).

    Granted, this may not have been the case when BO came to the WH and took over. They may have had previous IT staffers who stayed through the transition, but I'm guessing they did not (due to political mistrust issues). It could've been a genuine clusterfuck. Sometimes it's nothing and people cry about the sky falling as they pull down the curtain; sometimes, it really is bad. (If you understand weather patterns, you may recognize a summer storm to not be the disaster that chicken little claims...)

    • by TubeSteak (669689) on Wednesday March 14, 2012 @01:35AM (#39349451) Journal

      Sorry, I've got a very thin skin when it comes to management making any sort of technical claim. They're usually about 50% lie, and of the remaining 50% truth, only about 1/5th of that is factual with the rest being augmented by misunderstanding, disillusions of grandeur, and over-simplification to pull up the full 100%.

      This was the first relevant article kicked up by google:
      http://www.computerworld.com/s/article/9142268/_Lost_Bush_e_mail_settlement_requires_that_White_House_reveal_IT_practices_ [computerworld.com]

      The e-mail problem began in 2002 and 2003 after the White House moved from Lotus Notes to Microsoft Exchange. As it moved to the new platform, the President's IT staff also discontinued use of legacy, circa 1994, electronic management and archiving system, called Automated Records Management Systems (ARMS.) Development began on a new archiving system that ran into its own issues and wasn't implemented.

      Without an automated archiving system, the White House relied on manual processes to archive e-mails, and that's when the problems evidently began. Files were mislabeled and commingled on back-up tapes containing all types of information.

      The public didn't find out about this for years until federal prosecutor Patrick Fitzgerald disclosed it in 2006 while investigating the outing of Valerie Plame.
      The Bush Jr. IT infrastructure was broken from the day they installed it and remained broken for the full 8 years he was in office.

    • by mveloso (325617) on Wednesday March 14, 2012 @02:03AM (#39349589)

      From what I've read, there's only one firm that does White House transitions. I think it's Bechtel, but it's been so long that I've read anything about transitions that I have around a 15% confidence level in that piece of data.

      Google "white house transition" and you'll see that it's a total mess. If you want to read about it, there's info here:

      http://whitehousetransitionproject.org/ [whitehouse...roject.org]

      From what little I've read, you basically get a mostly empty building (the White House). It's up to the team to build/rebuild the infrastructure...but as any operations person knows, IT infrastructure is usually way behind everything else. The general executive branch IT has been a low priority for decades. What's more important, email or setting up the phone so the president can call someone (or someone can call the president)?

      At that point, the team is probably so far behind that they're screwed continuously for the one or two terms.

      Are the guys running the systems any good? I'd ask you: how many of you could pass a background and attitude check? You think the process etc at your workplace is bad, imagine how bad it is in the Executive branch.

      That said, it might be fun...but it's probably a nightmare. "I can't print out this $15 billion dollar appropriation because the f*cking printer doesn't work!" "People in PA are starving because the email server ate all of our emails!"

      Every minute is a crisis, with everyone breathing down your neck 24/7. Does that sound like something you'd want to do for 24/7/365/4 years?

  • The intern convinced Colangelo that there was a great need for automation.

    WTF did he think computers were for before that little piece of enlightenment hit him?

    I really hope that was just some random idiot fluff from author of TFA and not an actual sentiment from someone with "Information" in their job title.

  • "Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago."

    Rahm Emanuel has few redeeming virtues and deserved that fitting computer he got. He too had reached his end that the White House, no too soon enough, however. Floppy, Rahm? Memory? Nah...

  • At least all the keyboards still had the "W" key [nytimes.com]!

  • Apparently their systems are from the 1990s which was, in retrospect, the worst decade to buy any kind of IT equipment. Imagine they had some Unix system with VT100 terminals. This would have given them an easy upgrade path. They could have made simple and secure ways to remotely login, plus they could have simply replaced the system with a more modern Linux system, etc...

  • ""In the first 40 days of President Barack Obama's administration, the White House email system was down 23% of time, according to White House CIO Brook Colangelo, the person who also delivered the 'first presidential Blackberry.' The White House IT systems inherited by the new administration were in bad shape. Over 82% of the White House's technology had reached its end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief o

  • They should follow the example of Sarah Palin and use Yahoo Mail.

    I can picture Kim Jong-un calling the helpdesk: "please add leaderofthefreeworld61@yahoo.com on my trusted recipients list, we almost initiated a nuclear holocaust because I missed an important message in my junk mail folder".

  • ... the day you find you're out of IPv4 addresses, and all your systems are not IPv6 ready, in spite of it being a government initiative to have all systems IPv6 ready by this year.
  • by Anonymous Coward on Wednesday March 14, 2012 @08:03AM (#39351311)

    This article is partially correct but leaves out the actual technical issues involved.

    Someone *from* that Datacenter here at that time. Here's what really happened.

    The old administration did not care about the existing IT infrastructure because they were on their way out. They wanted no changes made- just that things be left up. Yes the email system was old and past EOL, but the outages were really the perfect storm of everything that could hit the fan actually hitting the fan at the same time.

    The facility was doing work on the power system- the UPS to be specific. Somewhere along the line they messed up, and cut the power. *All* of the power. Datacenter goes dark. They brought the power back up, but then tripped it again before bringing it up for good. This detail is what caused the weekend of hell.

    The SAN that the clustered email servers (yes, clustered, they *were* redundant) had the stores on was an EMC Symmetrix. It has a built-in battery backup system so that if the SAN looses power it has enough stored to flush the cache to disk. The power going off started this process. The power going back on triggered the response to stop flushing the cache and start checking and rebuilding. Then the power went off again. This is the part where the specific details get hazy but in effect the SAN did not like this. I don't believe it had enough power to totally flush the cache and/or it did not have the logic built in to handle an outage while in recovery mode. The result was a downed SAN that *would not come back up*. Now all of the data was down and nothing could be done but wait for the vendor to show up and try to fix it.

    At the same time we were dealing with *every* server being off and having to come back up. There were hundreds. Luckily most did. Some did not. Some were important, such as in the case of *both* the servers in a clustered system that would not boot- which just so happened to be the system that some of the say "more important" VIPs were on. These were old systems running Exchange 2000 on Windows 2000. Long past due, but kept up by the staff since the EOP would not approve a new email infrastructure.

    Eventually the systems would be restored and everything would be back on-line. In the meantime though Brook thought it would be a good idea to spend untold amounts of money to bring in MS Engineers to look things. They cost a lot of money and made a bunch of reports but they didn't fix a damn thing. The staff that was already there found the issues with the servers and fixed them.

    There were later headaches, such as when mentioned that the Sonnet was cut (thanks Verizon!) and further SAN maintenance but that was the weekend from hell.

    Things to note:

    • There has been 24x7 NOC (Network Operations Center) for the EOP data center long before the current administration.
    • There was a DR (disaster recovery) data center. It wasn't *great* but it was there. Due to the SAN outage and estimated time to fail over it was determined by those in charge that the best call was to repair instead of failing over.
    • The "some previous experience" listed for Brook was *all* of his previous experience.
    • The GOALIE position is a joke. They took perfectly good technical government people off of doing technical work and put them in a useless role basically overseeing time sheets. Unfortunately things were now slower because changes had to go through the GOALIE's who generally didn't have quite the overall end-to-end system expertise to make final decisions.
    • Brook really wanted to push a "mobile desktop initiative" that was a joke. He wanted the remote experience to be "just like working in the office" with requirements of the laptops being encrypted. Let's disregard that this can never happen if not simply for bandwidth constraints. But still, they tried. Vista SP1 would have been perfect for this (because of bitlocker), or hell even waiting until Windows 7. But no, Brook just said "no" to Vista because it was Vista and forced the engineers to

It's a naive, domestic operating system without any breeding, but I think you'll be amused by its presumption.

Working...