When Metadata Analytics Goes Awry 88
jfruh writes "When blogger Dan Tynan started seeing lots of Latvians in his LinkedIn People You May Know list, it was pretty funny, considering he'd never been to Latvia or ever met anyone from there. But now that shadowy spy agencies are using algorithms similar to LinkedIn's to see if we're terrorists, mistakes like this are a lot scarier. From the article: 'More than ever -- and online in particular -- who you know can be more important than who you are. In fact, who somebody thinks you know may be more important than who you are, especially if that somebody is a faceless government bureaucracy with limitless power to izjaukt savu dzvi (mess up your life).'"
Analysing Degree of separations creates conformism (Score:5, Interesting)
Originally when the concept of "degree of separation was invented" the idea was that everybody was connected to everybody through 6 degree of separation.
At the same time people think that are "the good guy" who does not keep "bad company".
With social media the length of the separation chain has considerably shrunk.
Add to this that most people who "do something interesting" (like making really nice flower arrangement for instance) will tend to travel and meet a "much smaller" crowd of people who "move around".
In this "smaller world" you can make "very short chains" to quite shady people. Actually it is trivial to create a chain from any US politician to "big list of officially evil guy" that at most 4 level deep. (For instance Ex HP Head Carly Fiorina went to KSA and met large HP clients including the heads of SBL managed by the brother of that really bad guy who did get some support from the ex President(s) Bush when he was against the Soviets...
And now comes the "suspicion creep" if you know Fiorina and one or tow of the Bushes, then you know 2 suspicious characters that are 3 or less level away from Really Suspicious guy.
So "one" could be ok, but 2 humm very bad...
So unless you take great pain to avoid anybody that might "be out of the ordinary", you imediatelly are 100% sure to become somehow "in contact" with somebody "suspicious".
Or seen another way, being not completely boring gets you something like 200 contacts, among which you can expect at least 3 "super connectors" who do not really overlap, particularly if you are travelling, so taking in account diminishing returns it is hard to avoid having less the 3M "level 3 contacts"
or 1/1000 of all adults in the world
the probability that less than 2 are "bad guys" is quite low.
so be boring or be afraid, very afraid...
Well.... Duh! (Score:4, Interesting)
This is what I, and a host of others, having been screaming about for years. People are blindly using analytics and "big data" to make important decisions decisions about health care, insurance, credit ratings, terrorist affiliations, etc. I have encountered so much bad data in my career the thought that it is take as "gospel" makes me sick. Bad data are out there and cleaning up a polluted data stream, when possible, is expensive and takes a long time.
Then you add in the use of NoSQL databases engines such as MongoDB which are not ACID compliant. You are virtually guaranteeing data will be corrupted. But then again, maybe I "just don't get it". But personally I think contributing to bad data is unethical.
Re:Unfortunately true... (Score:5, Interesting)
Then I have to prove a negative, that I do not know this person. All their evidence points to the opposite. "He was in New York at the same time!" (BUT I LIVE THERE) "Doesn't matter". "Your fathe'rs, cousin's, uncle's former roomate went to Iran as an exchange student", etc, etc.
That's an excellent point - its the classic example of the Prosecutor's fallacy [wikipedia.org]. By definition anyone found through a meta-data search will have strong evidence against them. If someone was caught independently plotting terrorist activities then it would be valid to say that it would be very unlikely for them to have a lot of connections with known terrorists. Trawl through databases and find someone who has a lot of known connections and it doesn't say a lot. Its like if you have evidence that someone tampered with a lottery and won you could say the chances of winning are one in 14 million (or whatever), but if you look for people who have won then it is not valid to say that they must have cheated as the odds against winning are so low - because someone will through chance!
Re:Analysing Degree of separations creates conform (Score:4, Interesting)
My work requires me to keep up to date with the computer industry. This means I must be connected with the hacker sites and ipso facto it also demands I am 1 degree separated from Mr. Snowden and many others who the US Government takes a dim view of. Get real people, mere contact isn't criminality, it is in the case of the investigator necessity. This is why the whole concept of Probable Cause is such a necessity!