White House Website Limits Iraq-Related Crawling 837
oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."
Queue somebody... (Score:5, Insightful)
Of course, people would be less likely to trust random-Joe from the Internet than, say, The Wayback Machine, but I expect this is what will happen...
Just Ordinary Web Activity (Score:3, Insightful)
If this was some crazy government conspiracy and they were trying to hide the information, why would they put it on their website? Could be any number of reasons they have done this perhaps they were getting loads of hits from google about iraq related things but if anyone really wants the information surely they can just visit it.
Re:More American Cencorship (Score:3, Insightful)
They do, it's called voting, not to mention public opinion polls, which were near 70% for the invasion when the US invaded.
Comment removed (Score:4, Insightful)
Re:Oh please (Score:5, Insightful)
I have to admit, when I first read the story I thought someone was being paranoid. But you really should RTF robots.txt file before you accuse the poster of being paranoid. The disallowed files are extraordinarily specific. I really can't come up with a plausible explanation beyond simoniker's.
Diff between fact and fict: Fict must be believed (Score:3, Insightful)
Winston's greatest pleasure in life was in his work. Most of it was a tedious routine, but included in it there were also jobs so difficult and intricate that you could lose yourself in them as in the depths of a mathematical problem -- delicate pieces of forgery in which you had nothing to guide you except your knowledge of the principles of Ingsoc and your estimate of what the Party wanted you to say. Winston was good at this kind of thing. On occasion he had even been entrusted with the rectification of the Times leading articles, which were written entirely in Newspeak. He unrolled the message that he had set aside earlier. It ran:
times 3.12.83 reporting bb dayorder doubleplusungood refs unpersons rewrite fullwise upsub antefiling
In Oldspeak (or standard English) this might be rendered:
The reporting of Big Brother's Order for the Day in the Times of December 3rd 1983 is extremely unsatisfactory and makes references to non-existent persons. Rewrite it in full and submit your draft to higher authority before filing.
It does seem questionable... (Score:3, Insightful)
Chris
Re:Devil's Advocation Follows. (Score:2, Insightful)
I agree that this is yes another step in the misinformation campaign surrounding the current administration. The policies that we've heard flip through hoops like trained seals. There's just no logic to all the reversals of focus, the "misquotes" and the public snafus we've seen happen. This is just another one of them.
Re:Devil's Advocation Follows. (Score:2, Insightful)
Comment removed (Score:3, Insightful)
Re:Why the fuck does the government use robots.txt (Score:4, Insightful)
Nosirree, no legitimate webmaster would ever use robots.txt to gently guide visiting bots to the appropriate parts of the site and to keep them from trying to do silly things. The only possible use is to trample your rights while installing the new corporate-owned government.
Geez, people. Honestly.
Re:More American Cencorship (Score:5, Insightful)
And 70% of the people in this country STILL think that Saddam played some part in 9/11. What was your point again?
Re:More American Cencorship (Score:2, Insightful)
Re:Devil's Advocation Follows. (Score:2, Insightful)
Implementing a system where out-of-date materials are in robots.txt, thus decreasing the possibility people will accidentally stumble onto them, requires an intern, a perl script, "find
Re:Drawing farfetched conclusions (Score:2, Insightful)
Why should a government-authored site (which, under the Constitution, by definition is public domain text) be exlcuded from non-government electronic publishing sites?
By the way, show me where in that Robots.txt file there's a command that would block http://www.whitehouse.gov/holiday/2002/art/01.html from Google? If you're right, there should be a line
disallow /holiday/2002/art/ . I don't see one. So, yeah, it's explicitly Iraq-related stuff that they're trying to block. Either 1. they're afraid that sensitive information might end up on the site by accident and want to make sure that it isn't archived if it is - in which case, they've got a lot more serious problems than political connivance - or 2. the theory is correct, and they're trying to set up a memory hole. Given Karl Rove's history, which do YOU think it is?
I honestly think this is stuff that goes on beneath GWB's notice. I'm with Molly Ivins on him: he's not evil, mean, or stupid, just wrong.
Re:Just Ordinary Web Activity (Score:4, Insightful)
Actually, the motivation around this could be to prevent caching of the documents, so that it's not so easy to compare differently dated versions of the same document. See this piece at Caltech [caltech.edu] for an example of how things change with time.
Re:And your ... (Score:5, Insightful)
It looks like someone blocked off parts of the site to web-crawlers; I don't know for sure why all those blah/bloo/iraq entries are in there but they sure as hell don't lead to anything.
Censorship: 0
Screwups: 100
country is not at war (Score:3, Insightful)
re: and your ... (Score:4, Insightful)
let's not get reactionary here, folks. it wouldn't make sense to do what's being alleged:
1. every major journalist worth his/her salt would be all over it within hours. so it wouldn't succeed in obscuring information.
2. it would create an incredible backlash as soon as detected. what purpose would this serve?
ed
We are at war with Eastasia. Eurasia is our ally. (Score:4, Insightful)
User-agent: *
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
And now, an offering for the lameness filter...
Oceania was at war with Eastasia: Oceania has always been at war with Eastasia. A large part of the political literature of five years was now completely obsolete. Reports and records of all kinds, newspapers, books, pamphlets, films, sound tracks, photographs- all had to be rectified at lightning speed. Although no directive was ever issued, it was known that the chiefs of the Department intended that within one week no reference to the war with Eurasia, or the alliance with Eastasia, should remain in existence anywhere. The work was overwhelming, all the more so because the processes that it involved could not be called by their true names. Everyone in the Records Department worked eighteen hours in the twenty-four, with two three-hour snatches of sleep. Mattresses were brought up from the cellars and pitched all over the corridors; meals consisted of sandwiches and Victory Coffee wheeled round on trolleys by attendants from the canteen. Each time that Winston broke off for one of his spells of sleep he tried to leave his desk clear of work, and each time that he crawled back sticky-eyed and aching, it was to find that another shower of paper cylinders had covered the desk like a snowdrift, half burying the speakwrite and overflowing onto the floor, so that the first job was always to stack them into a neat-enough pile to give him room to work. What was worst of all was that the work was by no means purely mechanical. Often it was enough merely to substitute one name for another, but any detailed report of events demanded care and imagination. Even the geographical knowledge that one needed in transferring the war from one part of the world to another was considerable.
This was written in 1948. Things have really progressed!
Re:Truly Frightening. (Score:3, Insightful)
Re:Drawing farfetched conclusions (Score:5, Insightful)
It looks like someone did a
find . -type d|perl -e 'while(<>){print "${_}/iraq\n"; print "${_}/text\n";}' > robots.txt
I have no idea what the purpose would be, but it seems like a funny thing to do if you were trying to hide something.
By the way, who is going around looking at people's robots.txt files?
Re:A CLASSIC QUOTE... (Score:5, Insightful)
Paranoia aside, I object to these restrictions as a matter of principle. They're making it more difficult to access publically available information. It's not classified, and it never was. I, as a citizen of the U.S.A., have a right to know what my leaders have said and done.
Let's assume the whitehouse.gov search engine is completely honest, and faithfully returns a complete listing of all materials on the site having to do with Iraq. If that's so, then there should be no reason to disable other search engines, since their results would just confirm the internal results.
But the restrictions are in place, meaning that someone thought there was a good reason to do so. Restricting access makes it more difficult for people to research information pertaining to Iraq on the whitehouse.gov web site. Who are the people most likely to be doing that? Answer: journalists, activists, and concerned citizens. Obviously these restrictions aren't enough by themselves to dissuade a determined researcher; but it might slow them down. And it might actually stop a diffident researcher completely.
I'm not even going to go into scenarios where the whitehouse.gov search engine is not trustworthy, because serving up "doctored" speeches or information is highly unlikely. There are too many other archives to compare against, and it would be a major scandal if the administration was found to be altering records on its website. They'd have to be really, really dumb to do that.
The whole thing still leaves a bad taste in my mouth, though.
Re:Other, arguably more reasonable explanations (Score:1, Insightful)
Arghh... How can you people be so dumb? Why don't you actually look at the website and figure some of this stuff out? The URLs ending in "/text" are text versions of pages. They prefer search engines to dump people to the graphical versions. Here, I'll spell it out:
Graphical version (not found in robots.txt): http://www.whitehouse.gov/firstlady/recipes/ [whitehouse.gov]
Text version (found in robots.txt): http://www.whitehouse.gov/firstlady/recipes/text [whitehouse.gov]
Nonexistent version (found in robots.txt): http://www.whitehouse.gov/firstlady/recipes/iraq [whitehouse.gov]
The "iraq" entries were probably added by mistake. Most likely a junior webmaster didn't understand the script that is (apparently) used to generate robots.txt for whitehouse.gov.
The only not-completely-ridiculous conspiracy theory that I can think of which is that someone wanted to discourage archiving of some pages, and decided to hide the fact by making it look like a script had screwed up. But I personally don't find that explanation plausible. Why not use meta tags? Most spiders simply do not respect robots.txt in this form. Pages like whitehouse.gov/iraq are still in google's cache anyway.
So personally I'm 100% convinced that this is a simple screwup. And even if it's not a screwup, most of the accusations made by the paranoids around here make about as much sense as a Wookie deciding to live on Endor.
Re: and your ... (Score:4, Insightful)
Don't be naive. How long do you think that any mainstream journalist who made a story of this would have a job for? The answer - not long. The US media in particular, although the UK is getting as bad, is little more than a relay system for government propaganda and real, detailed, complete examination of government behaviour, with equal air time to truly dissenting opinions (how many times has Chomsky been on CNN in the past 4 months?) is out of the question. What the government does is Good and Right and Should Not Be Questioned.
Media by the elite, serving the elite.
Re:A CLASSIC QUOTE... (Score:5, Insightful)
The other rule for transparency is that all material information be made available, kept, or destroyed in accordance to public regulation and individual policy. Individual policy must be consistent and decisions must be defensible based on policy.
The fact that people do not understand these two aspectsof transparency are what allow situations like Enron to develop. The later is what caused the destruction of Arthur Anderson. They have done nothing wrong, but they did not follow their own policy on document destruction, which made then look like at best idiots and at worst criminals.
We may compare this to other ventures to suggest policy. The NYT does not want google to cache articles because the NYT sells those articles after a certain time. Many other companies do not want deep linking because it reduces ad revenue. A fascist government may want to insure all users enter their site from a top page to make sure all users must go through the daily propaganda. A library tries hard to not track patrons so that no is afraid of using the library. The rational of the White House is beyond me.
The White House is not hiding documents. However, they are reducing the transparency of the government by limiting the avenues by which the public may access documents. Since the White House has stated many times that it believes in transparency, and in fact requires transparency when dealing with other governments, one can stipulate that transparency is the appropriate standard. So, until someone comes up with a policy that was developed and vetted through the normal processes used in the U.S., one has every reason to suspect nefarious motives.
And, if I may modify a statement that conservatives like to make, if you do not like transparency, go move to Iraq.
Re: and your ... (Score:1, Insightful)
Why don't you people get a grip on reality. There are a lot of problems with the media, but a basic inability to question government is not one of them.
Re:country is not at war (Score:5, Insightful)
Sample Quote from obscured document with link. (Score:1, Insightful)
"Simply stated, there is no doubt that Saddam Hussein now has weapons of mass destruction. There is no doubt that he is amassing them to use against our friends, against our allies, and against us. And there is no doubt that his aggressive regional ambitions will lead him into future confrontations with his neighbors -- confrontations that will involve both the weapons he has today, and the ones he will continue to develop with his oil wealth."
I can't possibly imagine why the Bush administration would want to keep these kinds of quotes out of search engines...
Re:Devil's Advocation Follows. (Score:1, Insightful)
I guess one of those would be not finding it at all. That's what this robot.txt file will do as google drops pages on its list.
Re:Funny (Score:2, Insightful)
Re:Missing Iraq and 9.11 files (Score:2, Insightful)
Two conspiracy theory leaps (Score:3, Insightful)
1) First, a lot of these docs involve Iraq. So, wihtout real factual information, it's assumed they're trying to do something fishy regarding Iraq info
2) Using that assumption, the next assumption is that they're purposely trying to keep people from trying to find contradictory statements.
This could all be true, or it couldn't be. Either way, by making two assumptions without any real facts is just pathetic yellow journalism.
Re:Drawing farfetched conclusions (Score:3, Insightful)
Sorry, I'm with Al Franken on him. (though Ivins is great!)
"I think he's mean. I think we're all too ready to blame Karl Rove, or Dick Cheney, or Ari Fleischer, or Gale Norton, or Donald Rumsfeld, or John Ashcroft when this administration does something despicable. When South Carolinians get push polls saying John McCain fathered an illegitimate black child, you know Karl Rove had something to do with it. But it's really Bush. When our energy policy is set by cronies from the oil, coal, and automobile industries, you can shake your fist at Dick Cheney. But it's Bush. When Ari Fleischer feeds rumors that the Clinton people vandalized the White House, doing $200,000 worth of damage, but month later a GAO report say that ain't true, you can say that Ari Fleisher is a chimp. And he is. But it's Bush."
...
"And I'm through with him."
Re:who cares? (Score:2, Insightful)
Day by day and almost minute by minute the past was brought up to date. In this way every prediction made by the Party could be shown by documentary evidence to have been correct, nor was any item of news, or any expression of opinion, which conflicted with the needs of the moment, ever allowed to remain on record. All history was a palimpsest, scraped clean and reinscribed exactly as often as was necessary. In no case would it have been possible, once the deed was done, to prove that any falsification had taken place.
Re:Other, arguably more reasonable explanations (Score:5, Insightful)
http://www.bway.net/~keith/whrobots/disdirs.html And, yes these files *are* relevant.
I have no problem with this (Score:3, Insightful)
Re:Not conspiracy, but I don't know what it *is* e (Score:3, Insightful)
I really do wonder what brings people to zealously defend actions like this. Sure, it could be a mix up, but a really ill conceived one. It's obvious that you don't have all the answers, just like others here.
My guess is that the poster feels that Slashdot posters are simply leaping to unjustified paranoid conclusions, and the depth of this faith (or so he pictures it) outrages him (or her).
The intensity of the poster's reaction is simply a reflection of his or her perception of Slashdot readers' zeal.
There are many possible explanations which do not involve conspiracy to hide information. For example, this could just be the work of some low-level IT guy who wanted to filter out one URL which happened to contain 'iraq' because the search-engine robots were burdensome to the webserver. I, for one, prefer to remain suspicious.
Re:Not conspiracy, but I don't know what it *is* e (Score:2, Insightful)
Re:EXACTLY (Score:5, Insightful)
Let's present an alternate scenario - since you have no evidence for yours, I don't have to present any evidence for mine.
It's May - Pres. makes his speech on the Carrier, the assumption by those-in-charge are that Chalabi's government will have control of the country within a couple of weeks and the US troops will be heading on home. The web folks (who want to make B & C look good) declare "combat's done! the troops are coming home! re-elect Bush!"
A few months later, that rosy scenario hasn't quite panned out. The aircraft carrier speech is becoming a liability for Bush - people started counting the number of dead troops in Iraq since he gave the speech, and it keeps going up. The web folks (who want to make B & C look good) say to themselves "this is a potential embarrassment to the president - let's see how we can make it less embarrassing."
And there you have it.
Re:country is not at war (Score:3, Insightful)
I suspect that many of the people captured met neither condition.
Re:country is not at war (Score:4, Insightful)
In which case they should be charged with something, either spying (unlikely if they were in their own country) or something else. They should then have the opportunity to defend themselves in open court with the ability to avail themselves of all the rights guaranteed by the Universal Declaration of Human Rights [un.org] which the US has signed up to. If US soldiers in Britain arrested me, I would not be wearing a recognisable uniform because I am not part of the military or any recognisable fighting force of government. That does not give them the right to forcibly remove me from my home country and lock me up without ever even charging me with anything! The actions of Bush and his cohorts in the Whitehouse are absolutely disgusting.
What a bunch of flamebait (Score:2, Insightful)
I'm so sorry I expended my mod points earlier in the day. What a bunch of flamebait bullshit this line of crap is. "Dictatorship?" Get fucking real. Let me ask this in non-partisan terms:
If the fiasco that was the 2000 presidential election went in Gore's favor, would you care to label his administration a dictatorship?
Has martial law been declared?
Are SS agents en route to your residence right now to conduct a little Q&A over this post?
Snap the fuck out of it. While I completely disagree with this appraisal of the Bush administration, I can (barely) live with you posting it. Just don't such nonsense to go unanswered and undebunked by me.
Re: and your ... (Score:1, Insightful)
That is, absolutely, the primary problem with the American media. Please pull your head out of your ass and inform yourself.
Re:And your ... (Score:5, Insightful)
Native people fighting against an occupying force are known as freedom fighters, not terrorists.
ry again sparky.
Re:not that far fetched (Score:2, Insightful)
Re:Furthermore... (Score:2, Insightful)
so us common folks should just stay out of the political game altogether? we shouldn't have opinions about politicians, and hell, let's all stop voting while we're at it. career politicians do a fine job governing this country, and if we question their wisdom, it's only out of a sort of working class ignorance, right?
Re:Not conspiracy, but I don't know what it *is* e (Score:2, Insightful)
Re:country is not at war (Score:3, Insightful)
Damn the terrorists to hell! I pray to God that He will strike all those who think like the terrorists down, and thrust them into the deepest recesses of hell. How can He be a God of Justice and Love if He allows this kind of crap to go on unpunished? They are not honorable, and they should feel DAMN lucky we didn't go and slaughter every man, woman, and child even remotely related to their cause.
We are in Iraq right no doing something that hasn't been done in a long time -- brought what is naturally coming to a terrible, terrible cockroach and his friends their just desserts. Just like when you stand up to a bully and crack him in the nose and explain to him that, "No, you are not going to take his lunch money, but you are going to eat a knuckle sandwich if you stop behaving so" you are totally justified, so are we when we step into a country like Iraq and unleash the wrath of justice on an immoral leadership.
You'll remember the Japs (as I intentionally refer to them) decided it would be a good idea to negotiate a peace treaty and then sneak attack. They too understand that we could've ethically justified a cleansing of their leadership as well. In other words, if, when we walked into Japan, we demanded that all the generals, officers, and officials of the nation present themselves to be beheaded for their crimes against our country, they would've understood and most likely complied.
Turn around and stop pointing your finger at our country -- why don't you go take a look at what is happening every day in North Korea, China, Iran, Rwanda, and other countries? You'll see that we are treating the people in Guantanamo more like men than they've been treated their entire lives.
Yes, I DO question your patriotism, because it seems like you sure as hell don't love what our country stands for. You'd rather side with the terrorists than side with a millitary commander who is doing the right thing.
Re:Furthermore... (Score:2, Insightful)
At the best, it's causing hatred in the Muslim world. At worst, its provoking terrorism (combined with our unconditional support of Israel).
Re:What a bunch of flamebait (Score:3, Insightful)
I'm so sorry I expended my mod points earlier in the day. What a bunch of flamebait bullshit this line of crap is. "Dictatorship?" Get fucking real. Let me ask this in non-partisan terms:
Yeah, so what? I don't know about you, but part of my governmental conditioning program, er, public education, included a long history lesson attached to the flimsy statement of "Those who forget the past are doomed to repeat it." Noticing that many of the things going on in this country during Bush's term as president are reminiscent of things that happened in Germany in the '20s and '30s isn't "bullshit". It's trying not to repeat the past.
You can disagree with it all you want, of course, and there are plenty who want to portray this country as a dictatorship when it's not--yet. It may not become one, either. One thing we have that the Germans of the '20s and '30s did not have is the history of Germany in the '20s and '30s. We can apply the hindsight and use the lessons in the present to prevent this country from becoming this "Fourth Reich".
But we can't do it if we spend our time in denial of history and present events. It may well turn out that there's no correlation, and that all that's really happening here is an incompetent president during a time of crisis (after Homeland Security failed to become the Gestapo upon inception, I'm inclined to think our president's just incompetent, and I recall from his governorship that it's a documented fact). But we have to be prepared for the worst, and hope for the best. Reality, as always, will be somewhere in the middle.
Re:country is not at war (Score:2, Insightful)
Agreed. What country does the US administration accuse of this act? The US government blames Al Queda, a terrorist organization, not a state.
We are in Iraq right no (sic) doing something that hasn't been done in a long time -- brought what is naturally coming to a terrible, terrible cockroach and his friends their just desserts.
And do you think that the 10,000 (maybe, no one bothered to count) murderered Iraqis are getting "their just desserts". How about those who were maimed? How about their families? How about the orphans who no longer have a family?
Just like when you stand up to a bully
How was Saddam Hussein bullying you? Even the Whitehouse does not claim that he had anything to do with 9/11:
when we step into a country like Iraq and unleash the wrath of justice
Would that be the justice of murdering Iraqi civilians, selling off their state assests and appointing a puppet government for them?
You'll remember the Japs (as I intentionally refer to them)
Racism does not add value to your arguments
we could've ethically justified a cleansing of their leadership
In the case of Iraq, of course, the US have spectacularly failed to capture the former leader - Saddam Hussein is still at large. Meanwhile, many Iraqis are without the peace, security, water and electricity that they had before the invasion.
Turn around and stop pointing your finger at our country
Certainly not! I would condemn this illegal invasion if it were anyone else who had led it. Why should the fact that it is your government that is terrorizing the world make it immune from criticism?
You'll see that we are treating the people in Guantanamo more like men than they've been treated their entire lives.
What a ridiculous statement. They are being denied their basic human rights.
Yes, I DO question your patriotism
Your notion of patriotism disturbs me, but please explain why I should have any sense of patriotism towards the US, a country of which I am neither a citizen nor a resident.