White House Email Follies

Presto Vivace forwards a link detailing a recent House Committee on Oversight and Government Reform hearing on the White House missing emails mess. David Gewirtz's report, carried in OutlookPower and DominoPower (in 6 parts, keep clicking), makes for scary reading. "If, in fact, the bulk of the White House email records are now stored in bundles of rotting PST files, all at or above their maximum safe load-level, that ain't good in a very big way... I object to using the inaccurate and inflated claim of excessive cost as a reason to avoid compliance with the Presidential Records Act."

What?

[palegray.net]

all at or above their maximum safe load-level

What exactly is the safe load level for a PST file? If you're talking about stuff that's not reliably archived, the answer is "there isn't one." I recall reading a story a while back about a debacle wherein several thousand emails were "inadvertently" deleted... what the hell is so hard about implementing a sane backup policy? It's email, not terabytes of images or anything.

Lost

[arigram]

They just need some excuse for "losing" dangerous email messages...

PST files for archiving

[call-me-kenneth]

Really, that's an accurate write-up - click past a couple of pages to get to the technical details. It'd be hilarious if it weren't so tragic.

After all, it's not like there aren't answers to the question "how shall I archive my user's email for legal and regulatory purposes?" [google.com] (Disclaimer- I work for a player in that market, but we're not on the first page of results for that search. So I don't feel too bad. Oh, wait - )

Re:What?

[cridanb]

the safe load level of a pst is around 2gb , after that it is subject to corruption
as to the amount of data , email systems are the largest systems on earth these days encompassing tens of terabytes of data in their live stores and tens of petabytes on tape

and yes they should have an archiving system not just doing tape back up tape

p.S if they used an enterprise email system like lotus domino this would not be a problem after all thats what the CIA uses

digitally signed and time stamped archive

[wkk2]

Given all the convenient archival problems, every executive branch email should be archived as a PDF and digitally signed and time stamped by a secure server with the private key in protected hardware. The archive needs to be outside of the executive branch.

The real question

[WindBourne]

is why are the dems allowing the White house off? They should be paying to have all the PST's restored. By now somebody has told them that the white house lied about the costs of the PST files. The need to go after them for perjery as well as getting the emails.

What really bothers me is that not this white house makes nixon and reagan look like boy scouts, but that the dems PROMISED to go after them, and really has done nothing.

Re:The real question

[Stanislav_J]

What really bothers me is that not this white house makes nixon and reagan look like boy scouts, but that the dems PROMISED to go after them, and really has done nothing.

Politicians making promises and then failing to keep them? I'm shocked....SHOCKED...

wow

[grasshoppa]

Here I am, some lowly line level system tech for a smallish town, and I'd be handed my marching orders were I even a quarter as incompetent as the white house staff seems to be. Which leads me to suspect:

1) Either they are that incompetent, and it's just a symptom of big government not knowing it's ass from it's face
OR
2) These people are purposefully appearing this inept.

Either option isn't pleasant, and both lead to a serious problem with our government where there will likely be no repercussions from this.

But then, we all knew that already, didn't we?

Re:What?

[Anonymous Coward]

the safe load level of a pst is around 2gb , after that it is subject to corruption

Depends. Starting with outlook 2003, there is a new format of .pst file which doesn't have size limits. Prior to that, 2 gig was the limit.

and yes they should have an archiving system not just doing tape back up tape

Well, that depends if you want your email to be subpoenaed/leaked or not! G.W. Bush is on the record saying he deliberately doesn't use email for that reason.

p.S if they used an enterprise email system like lotus domino this would not be a problem after all thats what the CIA uses

There are lots of enterprise email systems that can archive email, including exchange. I think this case is more of a deliberate choice not to archive, or plausibly deny archiving.

"Excessive Cost"

[SeaFox]

I object to using the inaccurate and inflated claim of excessive cost as a reason to avoid compliance with the Presidential Records Act

Considering how much we're spending in what are arguably other countries' wars, I'd find a claim of "excessive cost" for anything laughable.

Re:wow

[Anonymous Coward]

US Government IT is that bad.
Actually, it's worse.

If you already know, it's too late for your sanity.

If you don't, trust me, you don't want to truly know.

Re:wow

[laird]

The problem, I believe, is that the Presidential Records Act has no enforcement provisions or penalties for non-compliance. Thus, if the White House prefers to ignore it, there's no risk in doing so. So if the value of non-compliance is higher than the value of compliance, which is the case right now, the PRA loses.

This is not simply a case of incompetent IT staff setting up a system badly. The White House had an email system that by all accounts worked very well, archiving everything properly, and it was shut down and the staff let go, and the new system was set up by someone over-ruling their own IT staff in order to make sure that it couldn't work properly. That means that someone made the decision to spend a lot of time and money to eliminate a system that worked properly, to replace it with a system that didn't, over-ruling the recommendations of their own IT staff, which can only have been done intentionally.

What would be ideal would be for the PRA to be given real teeth so that the cost of violating it becomes clearly higher than the cost of not hiding whatever it is you want hidden. Given the extremely high value of keeping embarrassing or illegal behavior secret, the penalty needs to be extremely high as well, as it is for destroying evidence. That is to say, courts should presume that the records that were destroyed were incriminating. Judges take destroying evidence of a crime quite seriously.

Haha - anybody surprised?

[rainer_d]

Are you new here?
Honestly, this is a classic, almost Hollywood-style presidential-aid-villain-style tactic.
First, you dry up all the funding to something so that you can later claim there was not enough money to do it right.
And in the process of doing it "not right" some important stuff gets lost so the people in charge can't be charged later (which they can't anyway, because presidents make a habit of indemnifying their successors and most of the senior staff around them, because if they wouldn't, their successor wouldn't indemnify them...).

Still wondering why people actually get out of their bed and vote?

Re:Strangling kittens

[Entropius]

Elsewhere in the world destruction of evidence is taken as guilt. Is that not the case in the USA?

In the USA, it matters a whole lot who you're talking about whether or not XYZ counts as guilt.

Re:Solution

[Entropius]

The Bush presidency has helped OBL's cause beyond his wildest dreams.

The SPCA has a much easier time recruiting when there's a serial puppy-kicker on the loose.

Re:Delete the White House

[dpilot]

> operates at all near the level of minimum performance required

We should be so lucky to see such a high standard.

> Anyone still think all this incompetence that always protects Bush and his team is some kind of accident?

I would rather. The alternative explanation is EVIL and probably treasonous.

Re:What?

[Anonymous Coward]

I know I'm stepping into the middle of a religious war over email servers/clients, but....
It seems to me that any solution that globs a bunch of individual emails into a single database file is the wrong way to go. Why not take the simpler approach, and leave each email in the standard mbox plain text (binaries are MIME-encoded) format? Group them in directory structures that make archiving simple (like date). If you're worried about accessing terabytes of email information, the email metadata can easily be stored in a SQL (MySQL, SQLite, etc) database, which even if you have a 2GB file size limit, could reference terabytes and terabytes worth of actual emails. There are several solutions out there that take this exact approach, and all of them are quick, robust, and easily managed. If there's a disk failure, only those mbox files on that disk (or more likely on that bad block) are affected, and restoration is quick and simple. If your index database goes bye-bye, it's fairly easy to reconstruct from the existing mbox file structure.

Re:wow

[postbigbang]

While there's no seeming penalty, civil or criminal, there is a bigger penalty: ongoing confidence in government. The hubris and arrogance has become intolerable. This is just one symptom of a government gone berserk. Vetoing a bill on torture was another. Sliming the US House of Representatives because it won't pass a bill allowing the telcos to violate the very tenets of liberty in the constitution is another. The list is long. The list is sad. These are evil days, my friends.

Re:Lotus Notes?

[thirtimecharm]

The actual moral to the story is don't switch FROM Lotus Notes. At least not to Exchange. If they had multiple replicated servers, it wouldn't have been as easy for them to ,ahem, accidently lose all those emails.

Re:Delete the White House

[jthill]

No, but a great many people have gone tribal: they like it that the President is willfully violating oath, honor, duty and law. It means the man at the top of the hierarchy they worship, and therefore the hierarchy itself, is above all, and they're part of that hierarchy. The only rules they have to follow are what Big Men say.

The trouble started when they migrated from Notes

[markdowling]

" Mr. ISSA. Okay. So here we have a situation where the Clinton Administration is on a platform that has to be phased out. Simply, they lost the war of who is going to supply emails. A period of time goes on in which Yes, we are dealing, to Dr. Weinstein's concern, with getting good archives, but we are also dealing with the fact that I can't play my Betamax tapes any more, either, and I can't seem to find anybody who has a Betamax player any more."

Maybe Mr Issa should look here [ibm.com]. And Republicans are the ones who lose wars these days.

Meanwhile, the General Services Administration just saved a million bucks of taxpayers money [fcw.com] with Notes.

The reason the White House won't give up the email

[Master of Transhuman]

is because they DON'T KNOW what's in them.

With the morons they have on staff up there - and that includes Bush - they can't be sure all sorts of incriminating stuff isn't in them. In fact, they probably assume there is.

So they stonewall.

Read TFA. They're making estimates of the cost of recovery of the PST files as wildly off the mark. They're claiming it would cost $50K just to recover ONE PST file! And half a million bucks to recover 5,000 PST files!

That's deliberately false testimony - i.e., perjury.

Face it, folks. This country is being run by criminals now - just like in Warren Ellis' comic, "Reload". Look up Sibel Edmonds [justacitizen.org] on Google and see just how bad it is.

There is no folly here

[snaildarter]

Slashdotters may think that it looks bad for the President's email systems to be horribly unreliable. Compared with what was probably in those emails, this is nothing. In fact, this whole missing email thing is brilliant, and from the Pres's perspective, a job well done. The missing content gives the Pres a get-out-of-jail-free card. Not that he really needs one, since our Constitution makes kings out of our Presidents here in America, with the ability to do whatever they damn well please.

Re:What?

[aeschenkarnos]

after that it is subject to corruption

This is the Bush Administration. It started subject to corruption.

Re:What?

[wish bot]

I couldn't agree more. The Exchange database gives me the heebie-jeebies. Postfix/mail.app on OS X is so much saner for a low level server & client system.

Re:What?

[Nefarious Wheel]

The Exchange database gives me the heebie-jeebies

It used to give me no end of grief, and I'd specialised in the product for several years since the first test release. It's not really designed as a low level server, it was always strictly targeted at the large enterprise. The record structure was very close to full X.500 (I'd gone through the Exchange DB in raw mode and have studied the full X.500 standard) and I was tempted to publish the term "irrelational" for the DB structure, but I had to act the apologist because I made my living off the product. X.500 always looked like a comp sci major wrote it before he learned about key structures. The DB was kind of crap, being iirc one of the iterations of JET but the replication feature was pretty strong as long as you didn't try to put servers on slow/unreliable lines in the same site (if one site lagged, they all did, I/O completion problem).

However they did eventually clean it up and you no longer had to worry quite as much about the IS falling apart like a badly stacked sandwich. Eventually the Exchange directory evolved into Active Directory because the MSFT strategy veered toward LDAP. In fact, the newest release has Exchange using the AD paths instead of internal connectors. It's evolving, and it still scales pretty well. I don't know if they have, but I'd think moving the engine to SQL Server would be a good idea if they haven't already. But they've got so much grief atm working on compliance issues (SOX, Basel 2 etc.) that I doubt they have a lot of spare time on their hands.

Oh, and I've done a few Lotus / Domino installations and a bit of L3 support too, and I honestly believe that Exchange at its worst is easier to manage than Notes at its best at the enterprise level. Personal experience, no formal measurements, but mostly I didn't like Notes' fire-and-forget admin interface (I like fast feedback) or the fragile security structure.

PST's used the old JET engine too (I think it was the same one used for MS Access) and I always found they became unreliable after about 50MB, and you did *not* want the PST on a network share (it wasn't supported anyway).

Re:Depends on format.

[ta bu shi da yu]

Really? And, what, precisely, is it meant to do?

It's primary function would be to store, send and receive emails. Does it do that well? Sure, it sends and receives emails fine, but it sure as heck can't store them correctly. Like the parent says, what sort of idiot decided that a 2GB limitation would be a good idea for a PST file? And what sort of moron let's it save past this point, corrupting the file?!?

Re:Delete the White House

[JavaRob]

Realize that the Peter Principle [wikipedia.org] applies to everyone. Especially the incompetents at all levels in government - even POTUS hisself.

This is not a normal example of the Peter Principle. Bush would have already been promoted to his level of incompetence if they'd hired him as assistant to the secretary of the White House caterer. He'd already been "promoted" beyond his level of competence when he was governor of Texas.

Re:wow

[n dot l]

While there's no seeming penalty, civil or criminal, there is a bigger penalty: ongoing confidence in government.

You know, I used to wonder why Parlaimentary systems have votes of confidence. It always seemed like such a waste, calling an election and disolving the entire government, just because the ruling party and the opposition couldn't aggree on some line of the budget.

The last six years of US politics have explained it to me far better than any text or teacher ever could.

Re:Depends on format.

[sjames]

How good can a product be if it periodically trashes all of the data entrusted to it? Even better, when it does trash the data, it's in a proprietary format that makes it hard to try to parse through and salvage anything. The proprietary format also means that (like all MS products) if you'd like to use the data in a way MS didn't anticipate, you're SOL.

Based on comparing my experiance with non-MS mail programs (for DOS, Windows, and Unix over the years) to the experiances of MS users, I'd have to say that Outlook (in it's several variations) is a TERRIBLE program.

Re:What?

[Catmoves]

"...using the inaccurate and inflated claim of excessive cost as a reason to avoid compliance with the Presidential Records Act." This has to be hilarious coming from a branch of government that is spending more than a billion bucks a day on a war in the middle east that will gain us nothing but more enemies. Sigh.