Bush Administration's E-Mail Deluge May Overload Archive System 169
Lucas123 writes "The Clinton administration generated 32 million e-mails. Bush's administration has generated 50 times as much data — 140TB, 20TB of which is email — which soon will have to be archived through a new government-built records management system. The new system may not be up to the task because the technology behind it may not be able to handle the sheer volume of data along with the fact that the Bush administration has been slow in providing the National Archives and Records Administration (NARA) with needed information about the records, according to a Computerworld story. Questions have also been raised about millions of missing e-mails from between March 2003 and October 2006. 'It wasn't until this summer that an intensive effort began to share information,' said Ken Thibodeau, director of NARA's Electronic Records Archives."
It is Clinton's porn stash! (Score:5, Funny)
The other 120 TB was probably just Clinton's porn stash that the Bush administration found while purging off records.
Re:It is Clinton's porn stash! (Score:5, Funny)
Top posts (Score:2, Redundant)
Actually, 139TB is redundant data from endless logs of previous emails being top-posted over. Come on... you didn't expect bush's administration to actually be able to quote properly and tell youtube spam from important government work, did you?
Re: (Score:2)
Or 200 copies of "FederaL Buget 2002.xls", forwarded back an forth.
Number of emails generated. (Score:5, Insightful)
"The Clinton administration generated 32 million e-mails. Bush's administration has generated 50 times as much data -- 140TB, 20TB of which is email -- which soon will have to be archived through a new government-built records management system.
Well, to be fair, email wasn't quite as popular during Clinton's administration as it is now. Then again, the 400GB of e-mails that the Clinton administration must have generated (if it is 50 times less than 20TB) must have been rather hard to store when he left office.
Re: (Score:3, Interesting)
Re: (Score:3, Interesting)
It isn't storage and it isn't finding it, the problem is preserving it long enough to look through and index it. I'm sure that Google and companies that do similar work have the technology to do it. I'm also quite sure that for the right price the Federal government could obtain software to do most of the heavy lifting.
The problem is that the Bush administration deliberately migrated only partially to a new system leaving it in a state of constant risk for bit rot and corruption. It's hard to say how much o
Re: (Score:2)
Re: (Score:3, Funny)
Well, to be fair, email wasn't quite as popular during Clinton's administration as it is now.
Good point. I mean hell, Al Gore had just invented the Internet the year prior. Cut the guy some slack.
First, not enough emails... (Score:4, Insightful)
...Now too many many emails.
Whining is Washington's most favorite thing to do.
why insightful? (Score:2)
Funny, maybe.
First, they broke the freaking law. Now, they've put the people who need to handle their data in a difficult situation by not being cooperative.
They're not mutually exclusive, IMO
Text only, no html (Score:5, Insightful)
No more fancy signatures and html crap will cause a 60-80% drop in volume if not more.
Mandate the Usenet way with replies after the original, (it will) teach people to cut irrelevant repeats.
Stop the addition of stupid and ineffective disclaimers.
Teach the use of (ftp) servers for sharing large documents, no more Microsoft sized attachments, send a link.
Re:Text only, no html (Score:4, Insightful)
Re:Text only, no html (Score:5, Insightful)
Longer email threads seem to end up forwarded and brought to the attention of many people you never expected at the outset.
Judicious editing of the emails to include only the relevant sections for the replies, giving the context of the emerging thread of conversation means that someone being brought up to speed with that segment of the conversation doesn't need to trawl through masses of irrelevant junk to get at the meat of the issue.
I tend to do it as an efficiency gain, rather than taking storage space into account. All comes back to that quote you hear people come out with after sitting through a bad movie "Well, that's an hour of my life I'll never get back". It may only be a few minutes at a time, but they mount up over time. Plus, crafting things to cut to the heart of the matter puts things into sharp perspective, and means people are far less likely to digress, saving even more wasted time.
Re: Juducious editing FTW! (Score:3)
Yea, and there's an aesthetic feel to it too. If I'm in a 20 reply discussion, I like to edit out anything more than 2 exchanges old, and I change the subject title every two mails.
Nothing annoys me more than 20 mails titled "re: call"
Re: (Score:2)
Judicious editing of the emails
And you want to be the guy who has to explain what you cut out to Congress when this email ends up being part of a scandal?
Seriously, Bush sent an email telling everyone that he knew that he wouldn't be emailing while president, in order to protect them from any chance of special prosecutors and the like. Obama is going to have to do the same. The presidential record-keeping required post-Watergate has made damned sure that nobody says anything not suitable for public consumption in any way except face to
Re: (Score:2)
Editing of emails when replying is what was being discussed. Not editing the originals.
Re: (Score:3, Informative)
Deleting the bit that's already answered, not relevant or whatever can hardly be called 'editing', it has more to do with comprehension.
One of the worst things for the latter is a typical corporate Outlook mail exchange (I know that word...) with at the bottom text that hasn't been read for the last ten replies.
Re: (Score:2)
Deleting the bit that's already answered, not relevant or whatever can hardly be called 'editing', it has more to do with comprehension.
Indeed. Think <quote> tags.
Re: (Score:3, Insightful)
140 1TB Hard disks (plus another for RAID) probably costs less than a couple of government office chairs so what's the problem?
[Most likely the fact that it's in secret, proprietary formats and spread across hundreds of PCs instead of being archived by the mail gateway]
Re: (Score:2)
Most likely the fact that it's in secret, proprietary formats and spread across hundreds of PCs instead of being archived by the mail gateway
Of course, the government is sinister. Even something as benign as e-mail must involve inflated costs, secret programs and sheer ineptness. Get over it, there is this myth of a super competent government out to screw over it's citizens. There is no doubt the government screws us over as it tries to be an expression of our will. But to suggest it maintains nefarious purposes and isn't just a cluster fuck of red tape and retarded bureaucrats is just tiresome.
Re:Text only, no html (Score:5, Informative)
Re: (Score:3, Informative)
Some database driven mail servers like Citadel, Exchange, Zimbra and probably Domino support only storing the message and attachments once no matter how many people it was sent to.
It goes a long way in preventing the attachment * user mess.
Re: (Score:2)
Excahnge's underlaying database and storage engines support pointers to single copy attachments and even the message bodyies themsevles but if you don't have some third party product that does that then Exchange stores multiple copies.
Re: (Score:3, Informative)
Your statement doesn't make sense. Exchange supports and automatically takes advantage of single instance storage right out of the box. What do you need 3rd party software for that disables it?
I run Exchange on a NetApp SAN so everything gets deduped and archived to tier 2 storage if it hasn't been accessed within 90 days. Tier 2 is a lot SATA disks that are backed up to tape. It's not even an expensive solution when you start talking about the cost of enterprise storage.
Re: (Score:2)
Re: (Score:2)
Or just removing the previous attachment when forwarding and replying.
These are all solutions that make far too much sense when it's easier to just cause problems.
Re: (Score:3, Insightful)
In our organisation (government but non-US) we just give people a document management system and we educate people about why they should use it. If that doesn't work we point out it's policy a
Re: (Score:2)
Re:Text only, no html (Score:5, Insightful)
No more fancy signatures and html crap will cause a 60-80% drop in volume if not more.
I know you hate it when your mom or the boss' secretary at work sends out a cutesy formatted email but some people can actually use HTML email effectively in lieu of sending a document or a link
Mandate the Usenet way with replies after the original, (it will) teach people to cut irrelevant repeats.
Irrelevant repeats for you may be important context for someone else.
Stop the addition of stupid and ineffective disclaimers.
Often times, those disclaimers are required by law. Most people don't add them for fun or to make themselves feel important.
Teach the use of (ftp) servers for sharing large documents, no more Microsoft sized attachments, send a link.
FTP? Are you serious? Sending documents by carrier pigeon is more secure and reliable than FTP
Re: (Score:2)
It might have been better had the parent suggested SFTP which would be slightly encrypted as compared to email which can be heavily encrypted on the back-end for sensitive content.
Parent clearly isn't responsible for email for a corporate entity as much of what he says conflicts with their interests as you effectively pointed out.
Re: (Score:2)
Often times, those disclaimers are required by law. Most people don't add them for fun or to make themselves feel important.
Really? I've never seen anything firm stating they are required. Like a lot of things, it seems more like legal fashion than legal necessity.
Re: (Score:3, Insightful)
Stop the addition of stupid and ineffective disclaimers.
Often times, those disclaimers are required by law. Most people don't add them for fun or to make themselves feel important.
I don't know about the situation in the USA, but in most parts of the world, this is exactly the reason.
"Because everybody else does it" is another.
In multiple european countries, those disclaimers are entirely worthless, and even in some cases came back
to bite those using them in court by proving that the sender was aware that some piece of information might
end up in the wrong place.
Disclaimers don't replace common sense or encryption.
Re: (Score:2)
A better example of people who don't know how to use email is the subjectless and textless email with a 400kb MS Word attachment called "document1.doc" which contains
Re:Text only, no html (Score:5, Insightful)
Sheesh. I call this phenomenon "technological puritanism". All tech must be ugly! 80 columns should be enough for anyone! Fixed-width fonts were good enough for my granddaddy, they were good enough for me, and they should be good enough for everyone! Words are worth a thousand pictures! Get off my damn lawn!
Nothing personal, but if people like you were in charge of the world, we'd all be living in gray, cast concrete cubes. Think of the efficiency! No more wasted paint. You can just make a bigger house by stacking the blocks and adding a ladder.
Most of us *like* color, pictures, paragraphs, and most of all, convenience. Use FTP when I can just add an attachment that goes directly to the source? Give me a frickin' break. No one gives you respect points when you prove how miserably you can live.
Let's put this in perspective... that 120 terabytes costs 12,000 dollars in hard drives. Retail at Fry's. The entire output of the Bush Administration costs less than what they probably spend on coffee in a month.
P.S. And, yes, this is from someone who used a teletype in high school, and was ecstatic when we got a 300 baud modem (whoa! It's almost 3 times faster than the ol' 110!) and a Televideo terminal. Those days were not better.
Re: (Score:2)
that 120 terabytes costs 12,000 dollars in hard drives
Where the hell do you live that you can buy a terabye of storage at a retail store? (unless maybe you're talking about backup tape?) I can't even get that online. Overall, your cost assessment isn't significantly off though, since I can't push it higher than half a million (including all the equipment to access the data, and redundant storage), even assuming the worst possible purchasing decisions (its the government after all).
Re: (Score:2)
Where the hell do you live that you can buy a terabye of storage at a retail store?
What? Did you just step out of a time machine from 1990? Where can't you buy a terabyte of storage?
Here's [frys.com] a terabyte for $119, with an external enclosure.
Re: (Score:2)
Where the hell do you live that you can buy a terabye of storage at a retail store?
Where do you live that you can't? Even Walmart [walmart.com] sells them!
Re: (Score:2)
Quite apart from the fact that terabyte drives are everywhere, you could also do it the same way you're storing the whole 120TB - ON MULTIPLE DRIVES!
Re: (Score:2)
>that 120 terabytes costs 12,000 dollars in hard drives.
Or a few minutes of the iraq war.
Not to mention the proper compression can really shrink that down to size. I bet theyre just looking at the exchange data store of various servers and adding them up.
Re: (Score:2)
While I agree with much of what you say, that $12.000 comment was particularly silly.
A load of raw data on top of unreliable physical media does not an archive make. I happen to work as a support engineer for e-mail and medical imaging archives, and before that I've worked with data storage, backup and recovery for 12 years of my life, and I've observed that the hard disk is irrelevant. Associated with archiving there are a couple of problems that need to be dealt with:
- Data Life Cycle.
If you want to store
Re: (Score:2)
That's *exactly* why a 120TB archive is *not* and *never will be* a $12.000 question.
My point wasn't that the actual cost of archival was $12K, my point was that the actual quantity of data was irrelevant. Whether it's 1TB or 120TB, you still have all the issues you mention. The OP's point that we should care how big our emails are in the interest of conserving bytes is ridiculous.
Re: (Score:2)
Teach the use of (ftp) servers for sharing large documents,
Considering how terrible ftp is for security, do we really want to teach government officials to use that? Not that email is good either, but ftp almost asks to be compromised.
And I could just see our government IT officials trying something brilliant with ftp like changing the port number with each successive POTUS to match their number in the order of US presidents.
Re: (Score:2)
Start by mandating text only mail.
No more fancy signatures and html crap will cause a 60-80% drop in volume if not more. Mandate the Usenet way with replies after the original, (it will) teach people to cut irrelevant repeats. Stop the addition of stupid and ineffective disclaimers.
Teach the use of (ftp) servers for sharing large documents, no more Microsoft sized attachments, send a link.
Holy timewarp batman, I thought I had just picked up my E-mail etiquette guide from 1993 when I read this.
These all would work very effectively if not for one minor technicality. Common Sense is also known as Kryptonite in Government. They try and stay away from that stuff.
As far as your disclaimer comment, perhaps if we got rid of stupid and ineffective lawyers who bring pointless lawsuits to a courtroom where stupid and ineffective judges actually allow said lawyer to continue....Well, you get my point.
Re: (Score:2)
Why would image based signatures take up a lot of space? You would only need to store it physical once and then the rest of the emails use links to the content when they are opened. If I recall they went to Exchange so they have single-instance storage available to them right out of the box.
Archiving 20TB of content is not a significant problem at all and could be done in less than 30 days from scratch. All you need is your SAN of choice, mine would be NetApp where 20TB is pretty much the starting capacity
Re: (Score:2)
Re: (Score:2)
Various departments work together on a proposal and eventually I am involved for the final say.
It's not uncommon I ask for some information on how they have come to the proposal and I'm confronted with these weird -read-from-bottom-to-top- conversations, they're not conductive to a smooth process.
Re: (Score:2)
Besides, the mail server can strip all html before storing, all it needs is support from a corporate policy.
What's up? (Score:3, Interesting)
It hasn't helped that the Bush administration has been slow in providing NARA with needed information about the types and volume of data that will need to be archived. It wasn't until this summer that an intensive effort began to share information, Thibodeau says.
I can understand the reasoning that for national security, some information needs to be kept secret. The thing is, the more I hear of this administration's obfuscation of their communications and dealings, I can't help but wonder what in the World they are hiding.
Shadowy Government (Score:4, Interesting)
Re: (Score:2, Interesting)
I'd say it's par for the course and if you think just one side is doing shady stuff, it might be because you're a bit partisan.
Re: (Score:2, Interesting)
Re: (Score:3, Informative)
Well, Clinton never tried to insist that his VP wasn't part of the executive branch,
It's called "Unitary Executive." That is, there's only one guy in the executive branch that gets to make the decisions. It's entirely up to the President how much of a role he gives the Vice President. Under George Washington, John Adams lamented that the only thing he could do was preside over the Senate and then, he had no say on anything unless there was a tie. It drove him nuts.
If you read the Constitution, Article II groups the Vice President in with the executive branch, but the ONLY place it provi
Re: (Score:2)
Re: (Score:2, Informative)
It's real simple, the Constitution isn't that hard to figure out.
Here is every mention of the Vice President in the Constitution:
Article I, Section 3:
...
The Vice President of the United States shall be President of the Senate, but shall have no Vote, unless they be equally divided.
The Senate shall chuse their other Officers, and also a President pro tempore, in the Absence of the Vice President, or when he shall exercise the Office of President of the United States.
Article II, Section 1
The executive Power shall be vested in a President of the United States of America. He shall hold his Office during the Term of four Years, and, together with the Vice President, chosen for the same Term, be elected, as follows:
In every Case, after the Choice of the President, the Person having the greatest Number of Votes of the Electors shall be the Vice President. But if there should remain two or more who have equal Votes, the Senate shall chuse from them by Ballot the Vice President.
In Case of the Removal of the President from Office, or of his Death, Resignation, or Inability to discharge the Powers and Duties of the said Office, the Same shall devolve on the Vice President, and the Congress may by Law provide for the Case of Removal, Death, Resignation or Inability, both of the President and Vice President, declaring what Officer shall then act as President, and such Officer shall act accordingly, until the Disability be removed, or a President shall be elected.
Article II, Section 4:
The President, Vice President and all civil Officers of the United States, shall be removed from Office on Impeachment for, and Conviction of, Treason, Bribery, or other high Crimes and Misdemeanors.
The office is also mentioned in the Twelfth Amendment (which changes how the VP is selected), the Fourteenth (regarding who can vote for VP and that they can't have been part of
Re:Shadowy Government (Score:5, Insightful)
No; you are partisan when you think an accusation against one side can be answered by an accusation against the other side. They are both bad (they are US politicians; corruption is so endemic that it's legal and called lobbying), but Clinton's presidency ended about eight years ago and isn't something worth discussing now.
The questions are; how to make sure Bush follows the law for what he still does? How to make sure Obama doesn't start off like Bush?
Re: (Score:2)
Not the same thing. (Score:5, Insightful)
Perhaps you're too young to remember, but Clinton's administration had a problem with missing emails during investigations too (Lewinsky, why hundreds of FBI records on their political enemies ended up in the White House, illegal campaign donations from China, etc).
Yes, but there is a magnitude of difference in importance between lost emails about blow jobs and a little dirty money, and emails about the loss of privacy and civil liberties of US citizens, torture of POWs, and the various other nastiness that GWB et al are suspected of. Much different.
Re:Shadowy Government (Score:5, Interesting)
I have understood this outgoing administration to be more than secretive. they're positively paranoid and the only administration in memory that was similar was Nixon. All internal memos have been classified first. Declassification only happens when there is a strong and abiding reason why the memo should be declassified. Contrast that with Clinton, where all internal memos are not classified, unless there was a strong and abiding reason why the memo(s) should be classified.
When Bush announced that his administration would immediately prepare for a transition (before the 4th of November, which was election day in the US), I assumed that the first course of action was that this Bush administration would do what the last Bush administration did: [Rip] the hard drives out of their computers and tried to erase "sensitive" computer files in the White House and West Wing. [consortiumnews.com]
To say that the Clinton Administration started with a "clean slate" was an understatement. Later, Clinton lawyers ignored the dangers of historical archive deletion when faced with Republican destruction of historical records. Presumably, they wanted a "pass" from future Republican administrations.
Republican administrations tend to be very secretive. Democratic administrations tend to not. I shall expect the Obama administration shall have to purchase all new computers -- or at least hard drives -- in order to simply start up in their first week. This is a horrid waste of taxpayers' money all in the name of whitewashing one's past deeds (for good or ill).
Due to record-keeping, we now know that Nixon did know about the Watergate break-in. And we do know that he was very interested in its coverup. Nobody can be prosecuted at this time for that (those who were found guilty have all ready served their time). I would be very interested to know if Reagan's CIA planted the stacks of AK-47s used as evidence by his administration that the attack on Grenada was justified. And we still do not know everything about the Iran-Contra affair. These historical records are worth keeping because, well after the Statute of Limitations, America gets another look at how an administration dealt with the world.
It is a shame that any Administration is that interested in "rewriting history" in order to unfairly burnish a legacy, which in the case of "W" is hardly salvageable.
Just join the dots (Score:2)
It's simple. Here's how to play.
For, all of the top people in an administration, do:
Find out who they worked for. Then find out who owns or runs that organisation. Draw lines between the names to represent associations. Then simply count the number of associations each of the names gets.
For example. The shiny new Timothy Geithner worked for:
Kissinger Associates -> Which is a member of Council of the Americas -> Which was set up by David Rockefeller.
or ...
he's a member of the:
Council on Foreign Relatio
Re: (Score:2)
Personally I think it will be petty little being hidden corruption like in the Oliver North case. IMHO the overblown secrecy and mass email deletions weren't really to hide the sale of weapons to Iran which a lot of people (right up to the President and maybe even some Democrats) knew about, it was about hiding the embezzlement that paid for Oliver North's convertable and airconditioning Oliver North's house. We don't
Massive Typo (Score:2)
Personally I think it will be petty little corruption
Good strategy (Score:2)
For hiding all your nefarious emails in the noise.
Half those old geezers will be dead before anyone get's around to reading them.
Welcometotheclub (Score:2)
Great, now they've got to deal with the same sort of things we do. Archiving every bit of email that comes into the system, and making sure it's available online for searching and retrieval.
I'm interested in how they're going to be doing it. I've been looking at Global Relay [globalrelay.com] for my own mail archiving. I wonder what they'll end up going with. I asked this a while ago on my blog [blogspot.com], too.
Re: (Score:2)
Do you know how much Global Relay costs? I'm looking at setting up a DR site and archiving content there, I would be interesting if these guys are cheaper than me doing it myself at IO Data.
Re: (Score:2)
I don't think I'm allowed to tell you the price they quoted me, but they're really friendly if you give them a call and ask for a quote. I don't think their prices are excessive at all. The steepest part is that there is an initial cost of purchasing a WORM (write once, read many) drive for archival purposes. The nice part of that is that the drive is yours. You take it if/when you cancel your service along with your discs.
If you'd like, I can get you in touch with my contact there.
ISPs (Score:2)
Although this is about white house e-mails, this sort of stuff shows how ridiculous it is trying ask ISPs to record all traffic. At least here tax payer money is being used, but an ISP simply does not have that sort of budget. I feel all to often the layman confuses IT with magic and the people in the field as magicians. We are lucky enough if manage to become a level one mage :)
Re: (Score:2)
A rather big difference.
Comment removed (Score:4, Interesting)
Re: (Score:2)
How much of that is spam?
That is an interesting question. Though believe it or not, a large amount of spam does come from address-harvesting. Have you tried doing a google search for any email address that you receive spam at? You'll likely find that address on a website somewhere.
Not much spam goes around by way of spammers randomly trying email addresses, and last I heard the address *@*.* does not actually send an email to every valid address.
I can imagine they are not allowed to delete spam
The current administration has already shown lack of concern over what they are
If they hadn't gone to exchange.... (Score:5, Informative)
The Bush administration moved the White House from a Notes/Domino based system to a Microsoft Exchange based system.
Before moving, they'd had no downtime -- even when congress was taken out for 2 days by the code red word (they were on Exchange).
In moving, they mysteriously 'lost' all their backups for a period of time that was suspicious as hell, and now they can't scale to handle the capacity issues they face.
In a Notes/Domino world, this kind of archiving problem wouldn't be all that hard to deal with. You'd just need enough storage for it, and create archives per week/month/year (or an archive per individual's mailbox, or whatever) to put on as much hardware as was required. I single checkbox would be all that was needed to have it encrypted as well.
Oh well. I guess if conveniently "loosing" mail when you don't want it found is one of your design goals, than you probably want to migrate to something less reliable.
Desired Outcome on a Platter (Score:2, Informative)
Seriously, if "conveniently [losing] mail" was the goal of the transition, they could have moved from Exchange to Domino and gotten the same effect.
Forget not, throwing storage (read: money) at any system tends to fix the problem given a competent staff. You don't make a very compelling argument.
Throwing storage won't solve Exchange's issues (Score:3, Interesting)
There's an inherent architectural difference between storing mail in a database built on Microsoft's JET technology, and one which stores its data in something that is (although distinctly odd) very much like an xml data store. The Domino architecture makes segmenting the archive into manageable parts by date, by person, or by any combination thereof much simpler.
Essentially, the Domino architecture results in exactly what you describe -- throw more storage space at it and you can keep storing more data.
Re: (Score:3, Informative)
Yet they 'lost' the important/incriminating ones.. (Score:2)
n/t
Dear staff (Score:5, Insightful)
Re: (Score:2)
"PS: Tacos rule
- George W. Bush, The Decider "
So that's the excuse now for "losing" data (Score:2)
Reply all (Score:2)
See what happens when you keep sending that same Excel spreadsheet back and forth to the whole distribution list?
The problem is Law, and convention, not volume (Score:3, Insightful)
Today we are blessed with ignorant self serving legislators who do not, and are far too happy to follow hard-case makes bad law hurd thought, eg children, porn, paedophilia, drugs and terrorism. The courts have long held that you can read post-cards, but that if your letter-in-an-envelope is opened then a felony is committed or the information is normally in-admissible.
For this to work people have to start encrypting and signing their e-mails and the Congress and the SCOTUS must enforce identical rules for electronic and hand-written communication.
Specifically you can not go out and discover the entire contents of someone's library and papers in a law suite, and expect to go on a search-engine enabled fishing expedition.
re: re: re: re: re: re: re: no subject (Score:2)
> check out the blue dress! forward to everybody you know.
we need to save 140 Tb of THAT ?!?
Should've used ZFS (Score:3, Funny)
Re: (Score:2)
Let the Internet Archive and Google do it (Score:2)
Just ship copies of the raw files to the Internet Archive. 140TB isn't that much; they put a petabyte in a rack.
Once the file formats have been translated, just point Google at the starting URL and wait a day while it indexes everything.
Moore's Law: 16x in 8 years (Score:2, Informative)
Well yeah... (Score:3, Funny)
Of course that happens when you embed the 1600x1200 raw image of dick cheney giving everyone the finger with each email
Not to worry, Bush has a solution (Score:2)
Re:What the hell does the summary say? (Score:5, Insightful)
Besides, only 140TB (or 20 TB)? That's child's play for any competent DB admin, never mind only about $2k worth of hardware to hold it.
Assuming that none of it's been put into the archival system yet, that means they're dumping 140TB on it in one go.
You index 140TB on $2k worth of hardware and come back to me when you're done. Hopefully I won't have died by then.
Re: (Score:3, Insightful)
Maybe not $2k worth of hardware but $200k will do. Which is still peanuts in government terms. They probably spend that amount on paperclips and toilet paper in the pentagon alone.
Honestly, storing and indexing 140TB of e-mail is a trivial task when you can apply a six digit budget to it.
If their "archival system" blinks at the sight of 140TB of mostly text then it doesn't even deserve the name.
Re: (Score:2)
After some quick back-of-the-envelope math I'd even say heck, pass me $300k and I'll build the damn thing for you in under 6 months.
With encryption, authentication and all the she-bang that your little governmental heart desires.
Governments. Sheesh...
Re: (Score:2)
"all the she-bang that your little governmental heart desires."
Yea, it's the Personal Services that get expensive.
Re: (Score:2)
After some quick back-of-the-envelope math I'd even say heck, pass me $300k and I'll build the damn thing for you in under 6 months.
I think they're looking for something more long-term than just six months. 8-)
Anyway, 300K for processing an unknown number of documents in an unknown number of formats is a bit optimistic.
Re: (Score:2)
Sure, it's just a ballpark figure.
This is a government project. So in reality it won't even get started for under $5 mio, will not show visible results earlier than 2 years after kickoff and become obsolete long before completion...
Re: (Score:2)
Yes, but whenever a subject involving storage comes up some numpty announces "I can store all that for $2k!"; I've even seen raging arguments based on people pointing out that no, it'll cost quite a bit more than that.
Granted, in governmental terms that's still a blip on the balance sheet.
Re: (Score:2)
I was thinking though, this is definitely a job for Sun's new ZFS-based SAN. Obviously you'll need more space to index it, also.
LTO-4 tapes to hold 140TB uncompressed is probably about $14K also (about $100/TB)
You're looking at $3-400K for storage, plus some hardware for indexing. You'd probably want something you can add storage to as you index it, which ZFS is perfect for. You could get whatever storage you need for say 3 months worth of importing, then in 3 months buy more storage at future (lower) pr
Re: (Score:2)
Using infiniband or FibreChannel at 4 or 8GBps, it would take about 140,000 seconds or about 39 hours at full speed to transfer that data.
Assuming you can get data off the system(s) it's currently on at that kind of speed - and even then, indexing's going to be an absolute killer because you wind up doing a lot of seeking.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Then there were (thankfully less frequent) times when you'd get something like "Shortcut to Shortcut to New Document[1] [2].doc" with the user insisting they sent you an important screenshot in that.