White House Website Limits Iraq-Related Crawling 837
oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."
Other, arguably more reasonable explanations (Score:4, Interesting)
Maybe, but I would think they might also be looking for "shady" spiders that ignored robots.txt. I wouldn't be surprised if there aren't a few honeypot pages in there too.
Devil's Advocation Follows. (Score:3, Interesting)
Completely ignoring for the moment the fact that these views and actions are really somewhat embarrasing for the Bush administration, this really makes sense from a practical viewpoint. Few things are as annoying as searching for something news-ish and finding primarily material from two years ago. And after all, if they ONLY were interested in people forgetting the old materials, they could have just removed those materials from the site totally. (Though perhaps they were aware removing the materials completely would cause mirrors, which would be fully searchable, to spring up.)
Mmmm.. Robots.. (Score:1, Interesting)
Those robots consumed Many millions in system capacity.
Of course this is completely different as our freedom is at stake.
Seriously though... (Score:4, Interesting)
While anything is possible in politics, is it possible that the web admin is trying to limit the amount of traffic on the site? Is it possible that his analysis of the weblogs show a lot of traffic from robots looking for Iraqi-related info?
Questioning any statements put out by the WH ... (Score:4, Interesting)
If you persist in contemplating a world where whatever statements that the WH puts out, no matter how they might seem to contradict previous statements, are not totally true and correct, then a relocation expert from Guantanamo will be by in a few minutes. Just step away from the computer.
Re:Oh please (Score:5, Interesting)
related links (Score:5, Interesting)
Here's a minor example of something those two sites didn't catch: Remember Iraq's so-called "mobile biological weapons factories" [fas.org]? A month after the story broke that they were for weather balloons [slashdot.org], the CIA moved their report's URL [informatio...house.info].
An intriguing fact about this whitehouse.gov/*/iraq thing is that they do in fact cover some of the important statements [bway.net] which are apparently not duplicated in the press release, conference, and briefing directories. Perhaps there was a "unique urgency" to cover up some poor choices of words?
Re:Other, arguably more reasonable explanations (Score:5, Interesting)
Re:Queue somebody... (Score:3, Interesting)
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
$ua->agent("Mozilla/4.0 \(compatible; MSIE 6.0; Windows NT 5.0;
my $req = new HTTP::Request 'GET' => 'http://www.whitehouse.gov/pathtostuff';
$req->h
'Accept' => 'text/html',
'Referer' => 'http://www.yahoo.com' # pour faire flipper le webmestre
);
my $res = $ua->request($req);
if ($res->is_success) {
# traitement resultat $res
}
else {
print "Erreur : ".$res->status_line."\n";
}
It looks like all dirs have "iraq" added (Score:2, Interesting)
File looks auto-generated (Score:3, Interesting)
Having said that, I'm not even sure that this robots.txt file would work the way it's supposed to. Seems like these iraq references should all have a trailing slash or a
Someone clearly doesn't want Google caching Whitehouse content on Iraq. The question is why? And how come they're so lame about it?
climate change? (Score:2, Interesting)
Disallow:
Now why would they want to stop these being crawled?
Paul.
I've seen worse than what Dan mentioned (Score:1, Interesting)
Re:Funny (Score:5, Interesting)
Re:Not conspiracy, but I don't know what it *is* e (Score:1, Interesting)
I guess the googlebot doesn't visit the page, but knows of its existence from other pages??? Either that, or the googlebot is a bad boy that ignores robots.txt.
Re: and your ... (Score:5, Interesting)
Where have you been living the past five years? Journalists don't criticize Bush.
They still have not published the fact that he deserted from the national guard during Vietnam and they practically ignored his DUI conviction.
The GOP has the media cowed with their constant 'liberal media' babble. There number of journalists who are prepared to hold Bush to account is tiny - Krugman, Conanston, Irvins, Alterman. After that its Al Franken, Jon Stewart and David Letterman.
it would create an incredible backlash as soon as detected. what purpose would this serve?
The chances that the mainstream media will pick this one up are very small. Just think how they would have reacted if it was Clinton!
Re:Funny (Score:1, Interesting)
Actually, we don't. The fix is in, friend. Much like the Roman Empire, where most people did not realize the republic was dead until over a century after the fact, our republic died many years ago and most people don't know it. Something akin to professional wrestling (i.e. a good guy and a bad guy... both working for the same promoter) was put in its place.
Re:Already queued? (Score:3, Interesting)
See the GOP trying to spin this... FOIA time (Score:3, Interesting)
Bullshit.
The Iraq entries could only have got there if someone was told to go and stop stories appearing in the Google cache.
The person who got the job appears to have done it in a pretty clumsy way, that is pretty much par for the course for this type of work. Nixon did not expect Gordon Liddy and his pals to get caught in a third rate burgalry either.
It looks to me like someone was told to block out the Iraq files and simply did a directory listing on the web server and then appended /iraq to everything.
If you want to find out for sure file some FOIAs.
Re: and your ... (Score:4, Interesting)
The crux of the matter is that he refused to have his pilots medical just after the Pentagon added a check for illegal drug use.
You can try to spin this whichever way that Karl Rove tells you but the facts are against you. The fact is that your great leader is a coward who ducked the draft and then deserted to avoid a drug test.
Is robots.txt enforceable? (Score:3, Interesting)
Would the White House sue for violation of the robots.txt file? Under what laws could they sue? Is robots.txt an implicit grant of permission to view copyrighted content? Would GWB press the Congress for a new bill, to mandate legal enforcement of the robots.txt?
That's probably not going to happen anytime soon, but it raises an interesting question. Is robots.txt legally enforceable? And if it was, would that be a good thing or a bad thing?
Your thoughts?
Re:country is not at war (Score:3, Interesting)
These people were probably by and large draftees, which unfortunately in Afghanistan, meant they weren't going to _get_ a uniform. They certainly have a right to public trial, but by and large they were probably arrested legitimately. I see this more as an indictment of the unfairness of the Geneva Conventions with regard to poor nations, or forces that aren't backed by recognized governments.
It would be a lot easier to classify this as "disgusting" if we knew just what was happening down there. Right now, we don't really know much of anything, which is disturbing on several levels. But isn't disgusting in the way I'd classify the very well documented types of supression that were commonplace under the government these combatants were fighting for.
Re:And you're ... wrong (Score:2, Interesting)
news/releases/2003/05/text/ or news/releases/2003/05/
See for yourself:
http://www.whitehouse.gov/news/releases/2003/05/t
Compare the headlines.
Re:Funny (Score:2, Interesting)
Regarding your comment: it's a childish retaliation against another poster's .sig that appears (the link is broken so I'm going from the link text) to down France about it's quite obvious ties to Iraq. You know.. the whole "Pot calling the Kettle black thing"?
So no, it's not news that the fundamentalist USA supported secular Iraq in a war against fundamentalist Iran.
Gee... and we wonder why so many Muslims in that part of the world think we're just a bunch of marauding, Koran-hating Christian crusaders. Mm mm... no mixed messages coming from THIS side of the ocean... noooooooo.
Here's a tip for anybody thinking about replying to start an argument over Iraq:
I don't care. Bush fucked the whole thing up from the beginning by "going it alone" and now it's too late, so we'll just have to slog through it.
And vote that asshole out of office when the elections come around. "Bring em on". Yea fucker... bring em right on in to the White House and see how big you are then. Tough talk from a military deserter... goddamn idiot. "Bring em on"... yea, as long as it's not YOU and YOUR kids that are meeting them on the field.... right?
Re:Funny (Score:2, Interesting)
I don't think USA can ever be a dictatorship. However, it can very well turn into a fascist government. Look at someone like Hitler or Mussolini. People always claim that Hitler was a dictator but that is missing the point. If there were elections held in Germany (open fair elections monitored by the UN), Hitler would still have won by a massive majority.
The US govt CAN start practicing fascism. I'm not saying it is doing that now but it isn't inconceivable. Dictatorship, on the other hand, is highly unlikely...
Sivaram Velauthapillai
Re:WMD's found! (Score:3, Interesting)
On a more serious note, as much as I hear people joke about "We kept the receipts" that actually is how the UN Weapons Inspectors were able to find the weapons that they did.
(btw, what percentage of the country think that it was Saddam Hussein that kicked out the inspectors in 1997?)
Anyway, according to Scott Ritter, by the time that Clinton kicked the inspectors out of Iraq they had accounted for 95% of the WMD, and the main reason they were not able to complete the job was more because of the Clinton administration than the Iraqis. (Not to say that there were not a bunch of problems from the Iraqis.)
Scott Ritter has been very outspoken about these issues and as a Marine Corps Captain durring Desert Storm and a Chief UN Weapons Inspector he is a very qualified authority. He risked his life searching for weapons and I think more people need to listen to him.