U.S. Gov To Spider Internet

HopeSeekr of xMule writes "Perhaps as one of the first high profile uses of Alexa's WebSearch Platform, the U.S. government plans to search, link and reference every news site, blog and email on the Internet, using sophisticated AI codenamed ADVISE to do the correlations. Unlike traditional dataveilance like Echelon, ADVISE aims to find terrorists before they strike and even deduce their motivations in wanting to commit their crimes. Part of the breakthrough is a way for humans to view data as 3D holographic images with tech recently used at the Superbowl."

again..

This won't help dealing with the terrorists at all.
What if they communicate via

- plain old websites/ftps
- internet storage servers, irc, etc?
- instant messangers
- VoIP
- decentralised networks?

Lets not forget that they can

- obsfucate.. simplest method would be typing stuff into a CAPCHA-like image. OCR has no chance...
- use slang
- encrypt!

It will end up as an intrusion to the privacy of ordinary people unaware of this and/or private communications among companies.

Re:again..

I sort of remember a Bill Hicks quote about the War on Drugs that I'm thinking applies to the War on Terrah- "Its not a war on drugs, its a war on personal freedom. Keep that in mind at all times".
What this amounts to is tracking thought-crimes, how can you know someone is going to commit a terrorist act until they do it? People say lots of things, people think lots of things. Whither freedom.

Re:again..

Hate crimes are in essence thought crimes too. Think about it.. hate is a thought. Your reasons for feeling a certain way are thoughts. So in essence you can be subject to more severe penalties purely based on your personal opinions while performing a crime. Not only that you can be convicted of a hate crime alone where your sole criminal act was an expression of hatred for something.

This differs from premeditated murder in that, yes, ultimately, premeditation is a thought.. but the key difference is that you were planning a crime. Planning to kill someone is not an opinion. Hating a group of people is an opinion.

It is basically legislating what sort of motivations for doing a crime is worse than some other motivation. So if you rob a bank because you hate banks should you be subject to stiffer penalties? If you kill a gay man because you hate gays how is that anything other than a murder? Hate crime, for me, goes way to far down the path of thought crime. Double plus bad.

Jeremy

Re:again..

Yeah, you misunderstand the idea of a ``thoughcrime"---which is the crime of THOUGHT. That is, the crime is merely thinking, absent the commission of any other crime. Hate isn't punished, though if it is a motivation for a crime it is an additional facto

Re:again..

An investment banker killing a gay a colleague over a promotion isn't going to keep gay people from becoming bankers

No, but it might make his other colleagues leary of getting promotions.

Different circumstances, same result, if you ask me. The "magical" d

Re:again..

If you kill a gay man because you hate gays how is that anything other than a murder?

If the purpose of your murder is to incite fear and terror in all gay people, then yes. It is far beyond a death threat - a criminal offense in itself - you've already gone through with it, the only question is who's next. Multiply that with the number of people you've threatened and we can easily put you away for good.

Hate thought isn't illegal, any more than other thoughts. What is pure hate crimes would have been called slander, libel, threats and more if they were done against an individual. You can't treat gays as inferior to straight people without treating a single gay man as inferior to a single straught man. Where does that leave your "All men are created equal..."? That it's okay as long as you insult many enough at once?

Re:again..

Hate crimes are in essence thought crimes too.

Huh? No, they're not. The people who killed Matthew Shepard, for example, did not go to jail because they were homophobes; they went to jail because they killed someone. The fact that they did so out of ho

Re:again..

Well, in that case, premeditated murder is a thought crime. The only difference between manslaughter and murder is *intent* -- a thought. The only difference between murder and pre-meditated murder the the *plan* to commit the murder -- a collection of tho

Re:again..

"And why is a murder committed out of hate worse than a "regular" murder? The victim is equally dead either way."

Good question, perhaps this will put it into perspective.

First, let's make this battery instead of murder. Murder is so heinous it is indeed hard to consider a major difference between one derived from "hate" vs. other reasons (being cheated in some way, result of a robbery, etc.). They are both "high crimes" no matter how you measure it, serious punishment will likely occur regardless of the hate component. Battery might be a better crime to illustrate the point, and probably one of the more common uses of hate aggravation (along with vandalism).

A non-hate battery crime usually is an event with some guilt on the part of the victim. Not saying they deserve it, but they probably did something to instigate it (insulted your friend, smashed into your car, etc.). The escalation was likely avoidable by apologizing, running away, or just keeping a cool head about you.

Take a hate-crime battery, and the victim was probably completely innocent, just being black, gay, muslim, etc. and at the wrong place at the wrong time was enough.

The perpetrator in the non-hate battery is likely regretful later, and is probably not an inherently evil person. They may need drug/alcohol treatment and/or anger management classes, etc. They are likely to learn a lesson, and will likely avoid repeating the offense in the future. There likely was no premeditation to it either.

The hate-based batterer is generally not regretful, perhaps even proud and satisfied. They will highly likely repeat it, and there is very little a victim can do to avoid it. This is a MUCH more dangerous person, and the punishment (and/or rehabilitation) needs to be much stronger (IMNSHO).

Another situation is a gay neighbor of mine that got burglarized. He came home to find his home burglarized, and "die fags" spray painted on his wall. I have been burglarized, and it was painful and scary, but I did not take it personally and I was not traumatized. I took it as a random, unfortunate, event not directed at me personally. Some druggy looking to finance his next fix. I could definitely see a difference in these situations, my neighbor was totally traumatized, as would I be. I definitely see the crime perpetrated against my neighbor as a far more serious crime than the simple burglary I endured, even though, at their core, they were otherwise similar.

Also be cognizant of other factors that can aggravate criminal sentences, such as recidivism, no remorse, etc. These are along the same lines as hate crime aggravations. They are all an attempt by society to allow for differentiation between one-time mistakes by the offender, and the much more dangerous criminals that will likely repeat and perhaps escalate their crimes. It is just codifying "hate" as an aggravation at the same level as some of the other factors.

I am not sure how well I am making my point, but I guess the bottom line is if you look at the victim impact, the impact of a hate crime on the victim (including their family and community) is far greater than than a non-hate crime. There is little a victim of a hate crime can do to prevent it. As well as the perpetrator of a hate crime is much more likely to repeat it.

I agree with earlier points, it can be difficult to determine when a crime is hate-based, or not. But in many cases, it is not all that hard. I do agree that assigning "hate crime" status to a crime should not be taken lightly or capricously, and if there is any reasonable doubt, should not be applied. I think (I hope) it is applied very carefully, and sparingly, in real life.

Re:Dude! You are too sane to be posting here.

Unfortunatly the reality is 'hates crimes' laws are a product of the modern 'civil rights' movement and just as much of a sham of doublespeak and deceit hatched by Democrats.

So rather than having federal civil rights legislation, you would have us go b

Re:again..

Not only do I agree that this is a 'war on personal freedom,' i also feel that this project has disaster written all over it. This 'AI' will have to be pretty intelligent to tag and organize all of this content in a meaningful way, and on top of that, those analysing the data will need to be pretty friggin' brilliant to use it correctly.

as you say, "People say lots of things, people think lots of things." I personally feel that there is no one who can honestly or accurately see all comments and verbalized streams of thought for what they are worth - usually just contemplation or teen angst.

while it is certainly *possible* that terrorists might use (or have used) globally accessible modes of communication to plan a major attack, monitoring the news wires and blogs is probably not the most effective way to prevent the attack.

we must continue to demand privacy at all times, however i feel that the push by the top levels of government to gain access to our souls could be our downfall as a society as we distance ourselves from each other in fear of relinquishing too much information.

The most terrifying quote in the article

"This sort of technology does protect against a real threat," says Jeffrey Ullman, professor emeritus of computer science at Stanford University. "If a computer suspects me of being a terrorist, but just says maybe an analyst should look at it ... well, that's no big deal. This is the type of thing we need to be willing to do, to give up a certain amount of privacy."

This is not something "we" need to be willing to do! My civil liberties are NOT YOURS TO GIVE AWAY! I'm terrified that a CS prof at Stanford thinks that it's no big deal that the US wants to spy on its own citizens and deprive us of our rights under the 4th and 5th amendments. (Yes, the 5th ammendment too, since US Citizens have been held on US soil without being charged with a crime, and thus deprived of due process of law.)

How can any educated person think this loss of privacy is "no big deal"? I'm at a loss for words.

Re:again..

This won't help dealing with the terrorists at all.

No, but it'll sure help keep the lid on political dissent, won't it?

Portions of this have already begun: the data mining only extends prior government watching of the web [washingtonpost.com] for "terrorists" like the ACLU [irregulartimes.com]. But [11alive.com] not [aclu.org] for [sfgate.com] political [icdc.com] speech [thirdworldtraveler.com], of [nytimes.com] course [aclu.org]. Never that. [findlaw.com]

So shut your mouth and shut down your blog and stop commenting here if you don't want to end up on a list of people to be "neutralized" -- like Mario Savio, hounded for ten years despite never breaking a law [sfgate.com].

Savio's "crime" was, ironically, leading the Berkeley Free Speech Movement. We'd do well to remember today 0Savio's words then:

There is a time when the operation of the machine becomes so odious, makes you so sick at heart, that you can't take part, you can't even tacitly take part, and you've got to put your bodies upon the gears, and upon the wheels, upon the levers, upon all the apparatus. And you've got to make it stop.

Wrong name

Its called Skynet. But it is looking for terorists...like Sarah Conner.

Google can help

After all they have all our data in their cache and they are inside US jurisdiction as well. I don't see why US Gov has to develop something fresh and duplicate all the effort Google has put into their search engine.

On the other hand, is NSA working with G

BUSH BOMB WHITE HOUSE

Let's see how well it works.
Sorry slashdot.

Re:BUSH BOMB WHITE HOUSE

I hate to be the one to tell you this, but you're not really anonymous when you post as an AC here...

Re:BUSH BOMB WHITE HOUSE

Lets do a better test of this new system.

I'm gonna shoot some Americans!

Negative. No knock on the door.

I'm gonna blow up the empire state building!

Negative again.

I'm gonna poisen the American food supplies!

Negative yet again. Damn I might as well stop and start downloading some mp3s...

KNOCK KNOCK KNOCK!

minority report

Yesterday, Bush saw Minority report (the movie), and and was totalllllly impressed with it. You see the result here.

The Erosion of America

Proponents of this initiative boast that other data mining systems, such as Starlight, have already proven their worth in the fight against terrorism. However, given the fact that the current administration knew full well [inthesetimes.com] that Osama bin Laden intended to use hijacked airliners as missiles in a terrorist strike, but chose not to act, and that the CIA managed to uncover this information without a wholesale violation of the privacy of American citizens, I really can't see the justification here.

Why exactly does the Bush administration need such vast amounts of information to conduct their 'war on terror'? And why were they unable to use the perfectly good intel they did possess to thwart the worst terrorist attack ever on American soil?

One thing's for sure...it doesn't really matter whether the people OK this initiative or not, as Dubya & Company have amply demonstrated a complete contempt for the law of the land.

The quote that annoys me...

Starlight has already helped foil some terror plots, says Jim Thomas, one of its developers and director of the government's new National Visualization Analytics Center in Richland, Wash. He can't elaborate because the cases are classified, he adds. But "there's no question that the technology we've invented here at the lab has been used to protect our freedoms - and that's pretty cool."

Excuse me?

If what he says is true, then it's possible that the technology has been used to protect our lives. Our freedoms are a different matter. Which of the two you consider to be the more important is a pretty strong indicator of whether you're a free country or a police state.

Re:The quote that annoys me...

Very true. It seems that there are a lot of folks out there, in both political parties, who are confusing safety and freedom. The irony seems apparent to me. Freedom, by its very nature, compromises much of our security.

Finding a balance between the two

Re:The quote that annoys me...

Give me liberty or give me death!- Patrick Henry

If you're so afraid of being killed by terrorists that you will consistently choose safety before
your own personal freedoms...

then those who wish to destroy our nation have already won.

Pretext Incidents used by the Elite to start wars

Oh, this 911 wtc pretext incident is only the latest of a long history of the America elite using/allowing/manufacturing "pretext incidents" in order to start wars and grab power. See this page on HOW TO START A WAR. [mindfully.org]

However, I think this War On Terror has

Re:Pretext Incidents used by the Elite to start wa

"So try the elite in court for treason. We now have the legal precedent. Perhaps."

First of all, how do you try a class of people in a court? 1 at a time?

Second, even if we assume that its possible, how do you plan to win?

Your only chance is revolution. Goo

What "Erosion of America"?

However, given the fact that the current administration knew full well that Osama bin Laden intended to use hijacked airliners as missiles

The article you are linking to is from 2003. The commision, creation of which they are talking about there, has actu

Spider info

IP ranges and user agent please.

Also, does it obey robots.txt?

www.terrorists.evil

User-agent: US-govt
Disallow: /

Big Brother ADVISEs you!

I wonder how long it will be before this system is used for political and/or selfish purposes?

George Orwell would be writing non-fiction if he were alive today.

Re:Big Brother ADVISEs you!

I wonder how long it will be before this system is used for political and/or selfish purposes?

Oh, I'd say in about 5... 4... 3... 2... 1... [signal terminated]

Robots

"Perhaps as one of the first high profile uses of Alexa's WebSearch Platform, the U.S. government plans to search, link and reference every news site, blog and email on the Internet, using sophisticated AI codenamed ADVISE to do the correlations."

I don't suppose this is going to honor the rules in my robots.txt.

Re:Robots

Then it'll get firewalled just like the engines that spammers use.

I have a script that automatically notifies me if a certain page is accessed - this page is in robots.txt, and very hard (damn near impossible infact) to click on accidentally.

Robots that ac

Re:Robots

The nice thing is that if it doesn't, you can trap it forever in a recursive link search.

Re:Unlawful Search & Seizure

But by placing your content on the public Internet, it's in "plain sight". There's no warrant required to look at it.

ADVISE

Unlike traditional dataveilance like Echelon, ADVISE aims to find terrorists before they strike and even deduce their motivations in wanting to commit their crimes.

"Hmm... ADVISE seems to think the terrorists are fed up with the 'nazi-like spy regime,' and are planning to use undead monsters to attack its servers.

Also, the terrorists want more boobies."

This was a good use of a few billion dollars to Haliburton.

It won't work.

Terrorists already know how to work around this stuff for critical communications. Go low tech. Don't use phones, don't use email, don't use the web. The method that Al Queda uses to get the videos to the media demonstrate that they already have a very good low tech infrastructure to do this.

This just looks like the security people are getting desprate and trying to cast a wider net. The secret wiretaps used on citizens was a wide net that seems to have had poor results [nytimes.com].

Exactly how

are they going to monitor e-mail?

Blogs and news sites are things we publish to the world and are easy to spider. Emails are private communications. In order to monitor them you have to either intercept them in transit or search records on private servers. Even if the email is available via webmail, you have to gain unauthorized access in a way that is generally considered trespass.

Re:Exactly how

Fairly easy to intercept email- just place a server on the network that reports false routing times and the data will flow (path of least restistance).
And if you don't think that's possible, there was a case in MA where a rare book dealer did just that, he

Can it detect sarcasm?

Can it detect sarcasm?
I highly doubt it.

Can it translate accurately from Arabic to English?
Somewhat more likely, but I still doubt it.

Do they really think the terrorists get their books from Amazon.com?
They are going to spider 2 million porn sites, and keep records of my purchases from Amazon, and never catch a single terrorist.

Sounds like a big waste of time and (my) money.
i.e. business as usual.

My guess is upper management at DHS doesn't understand computers too well, so they hire a bunch of nerds who can program, and give them absurd instructions, like "We need to get a handle on this whole internet thing!"
Or something ridiculous like that.

Something I've always wondered...

Why is it that it is always the US government that seems to have been up to all this stuff since WW2 and increasingly even after the Cold War? I thought you were supposed to be the people from the land of the free and whatnot, really suspicious of government intrusion into people's lives, et cetera. Considering that a lot of you are always willing to disparage the Europeans for their love-affair with government, I certainly wish a lot of you would just take the log out of your own eye first... it's your government, despite all the rhetoric, that is horribly control-mongering at home and eager to support whatever right-wing dictator abroad, while ours concentrate more on making sure that kids with cancer don't die in the name of economic efficiency should they be unfortunate enough to be born to parents of financially limited means.

Go ahead, mod me troll/flamebait... at least I won't post this AC.

When "24" is the source of knowledge..

Someone should have told them that 24 is not a reality show.

Jack Bauer : Chloe, I'm sending you a picture. Can you datamine for him?
Chloe O'Brian: Sure. send it to my screen.
Computer: Blip...blip...blip.
Chloe O'Brian: Jack - it's the well known terrorist named...

Interesting

I can't wait to see history books in about 100 years or so. Bin Laden's going to be up there with Sun Tzu and General Meade for the title of "greatest strategist ever."

Singlehandedly causing the West to self-destruct is no small potatoes.

smoke and mirrors

As any web server can choose the page to display to any given client, how exactly does the system work out what is real and what is not?

For that matter, exactly how do they expect to access password or IP protected sites?

War on Terror - my a...

Could we can it already? Or is there still some moron out there who believes that bullcrap?

Sorry for the language, people, but I feel insulted. Just how DUMB do they think I am?

Terrorists don't use the net. At least not if they're halfway smart, and hell, they are! They ain't some dumb, mindless bomber drones (ok, some are, but look at the US soldiers... same way 'round, just with rifles). The key heads are very bright individuals, they know what they're doing. They know logistics, they know psychology, they know how to build a network right around your feet without you noticing.

Do they use the 'net? Let's assume they do, ok? Let's for just a moment assume they do.

First of all, they WILL NOT use the net for anything but the minimally necessary form of communication. They won't blog, they won't chat, they won't spend time in a bboard, all they do is MAYBE sending some data from A to B. And it won't be much data.

This data will be encrypted by best state-of-the-art encryption.
A good deal of this data will be plain false, and it will be false in a way that they can discern whether the feds were sniffing. Simply for testing their communication channel for being tapped and their key for being broken.

If you consider, all this incredible effort just 'cause some oil countries dared to think 'bout taking Euros instead of Dollars for their crud... it's amazing what some old hydrocarbones can move and shake in this world.

how do you spell encryption?

If I had something I wanted to move over the internet, without anybody being able to read it, I would use a one-time pad or some other nearly-as-secure encryption. It's so easy to do.

This program will only catch the foolhardy, and will could be used for nefarious purposes against (mostly) law-abiding American citizens.

So it is a bad idea.

Remember, as Americans, we have the right, and duty, to inform our congress-critters and other representatives when we think the government is heading the wrong way. Send a fax to your Senators and Representatives today. Fax their local office and their Washington office.

Rulesets

Okay, it won't obey robots.txt. But what will the spider present itself as so we can lock it out? Or, even better, what are the sure signs that it's really Google or Yahoo or MS snarfing up my sites? Because I don't really care if other spiders get don't ahold of anything - close to 100% of legitimate searchers come through the big three engines. Should be possible to configure and script it so that anything but the spiders we approve of don't come up with much. If there are more than so many requests per minute, for more than so many pages - or it it goes to honeypot pages that aren't what the real public is interested in - lock the suckers out or feed them garbage. They'll find an Internet filled with hagiographies of the Bush family.

I'm not defending Bush but

One thing the intel community does is collect information from traditional news sources both foreign and domestic. There is a lot of useful information in the press. It sounds like they've merely extended that to web-based information sources. I'm not sure it's as much a thought control measure as simply making a catalog of existing public information, which a web site is. To me this seems like a normal function of intelligence gathering.

I think the inclusion of email is what gives this the swarmy, big brother overtones. We've also have ample evidence that the Bush administration can't be trusted. The combination of Bush political flaks with no regard for privacy or the law and large amounts of personal data is what makes it scary to me.

It would be much cheaper to build a Google API app

But of course, from the "brains" who are behind bolstering the costly, debt-exploding military-industrial complex (for fighting unjustified elective wars, no less), we are now seeing the formation of yet another unneeded program to scrape the web, with American tax dollars^W^W^Wproceeds of treasury bond sales to China (interest paid for by our children/grandchildren).

On top of this, we have a regime with widely demonstrated incompetence and/or willful negligence deciding to build a program like this. They couldn't even deal with the plain-language warnings they received regarding al-Qaeda's plans to hit tall buildings with jet planes. What I'm driving at is they can collect all the data in the world, and they have no ability to understand it or act on it, at least as long as His Lordship, King George is in power.

Publicize the IP ranges

What we need is a Web site to aggregate any known information about this project, especially the IP ranges from which it is operating. They can't spider anything if they don't get past the firewall.

The power of monolithic government can only be opposed by the organized efforts of informed citizens. The Internet makes it easier for us to be spied upon, but it also makes it easier for us to know who is doing the spying--and stop them.

No, the cat does not "got my tongue."

> The U.S. government plans to search, link and reference
> every news site, blog and email on the Internet, using
> sophisticated AI codenamed ADVISE to do the correlations.
> Unlike traditional dataveilance like Echelon, ADVISE aims
> to find terrorists before they strike and even deduce their
> motivations in wanting to commit their crimes.

Seventeen minutes later, Spynet became self-aware, and induced a nuclear exchange, destroying ANYONE NOT WEARING LIKE SIX MILLION SUNBLOCK! Have you ever had anything growing inside you? Do you know what it's like to create something? Wait, Statue of Liberty? That was our world! You maniacs! You blew it up! Damn you! Damn you all to Hellllllllllll!!!!!!!!!!111!!111!11oneone!!one

Re:Is this starting to create a bigger problem?

I seem to provide this quotation quite frequently these days. It was said by Lord Hoffman, sitting as a British Law Lord, in their ruling on the UK detention-without-trial fiasco a few months back:

"The real threat to the life of the nation, in the sense of people living in accordance with its traditional laws and political values, comes not from terrorism but from laws like these."

And, unlike the rest of us, the Law Lords sitting in that case presumably did have access to any classified information they required. It's very convenient that the government can always tell us how its draconian policies are protecting us from imminent doom (but they can't tell us how for security reasons). That argument is rather less powerful when its critics include people on the inside who would be well aware of the full facts.