Whistleblower: NSA Is So Overwhelmed With Data, It's No Longer Effective (zdnet.com) 209
An anonymous reader cites ZDNet's Zack Whittaker report: William Binney, a former NSA official who spent more than three decades at the agency, said the US government's mass surveillance programs have become so engorged with data that they are no longer effective, losing vital intelligence in the fray. That, he said, can -- and has -- led to terrorist attacks succeeding. Binney said that an analyst today can run one simple query across the NSA's various databases, only to become immediately overloaded with information. With about four billion people -- around two-thirds of the world's population -- under the NSA and partner agencies' watchful eyes, according to his estimates, there is too much data being collected. Perhaps that's one of the reasons why NSA wants to dump the phone records it gathered over the past 14 years.
T.S.Eliot called it! (Score:5, Insightful)
"Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?"
Old news (Score:1)
This is named infoxication and is known for decades.
Search Tools (Score:2, Insightful)
Sounds to me like their search and filtering capabilities are the problem, not the amount of data available.
Re: (Score:2, Insightful)
False positives, false negatives.
If you have a correlation that gives an impossibly good 1% false positive rate and 1% false negative rate, you can expect that 1% of the subjects you are looking for will be overlooked and 1% of those who you are not concerned with will match. So, let's apply that to the current nuisance.
1% false negative: for every 100 people with hostile intent, 1 will slip through the net and either bomb something or be stopped by civilians.
1% false positive: for every 100 people without
Re: (Score:2)
You have a number of assumptions, the worst of which is symmetric false reports (1% and 1%). The more likely scenario, which is also tunable (gets better with more data) is that it is asymmetric in nature, and thus the conclusion is not only inaccurate but terribly so.
It would have been MUCH better for you to not assume anything, and give general impressions of how the numbers might work. And 1% is way to high. You're more likely to see numbers in the "per 100,000" range (.000001 vs .01000), which is 1000 t
Re: (Score:2)
Of course there is a question of what do you count as a "false positive". Every time you make someone toss a box cutter in the trash where you wouldn't have before. Every time you arrest somebody where you would have let it slide before, every time one of those people wasn't a terrorist, that is a false positive. Or at the very least a side effect.
Re: (Score:2)
Re: (Score:2)
The problem isn't merely the volume of data. After all, LHC produces terabytes of data with each run. The problem is one of volume and variety. Imagine tracking every phone call made in the US, out of the US and into the US. We're talking about everything from calls to working spouses to pick up bread on the way home to ordering of products to sex chat calls and thousands of other topics. Filtering and searching those calls and the metadata surrounding them would be a monumental job of incredible complexity
Re: (Score:2)
Re: (Score:3)
having a pool of known terrorists
If we know who they are, why do we need all that data too?
Re: (Score:2)
having a pool of known terrorists
If we know who they are, why do we need all that data too?
To find many more of them, any private or corporate or state supporters, and all associated people including family.
Re: (Score:2)
Well, I probably contributed in some small way when talking with my brother on the phone a few nights ago. We were talking about the Apple case, NSA surveillance, etc, and I mentioned how just by saying "allahu akbar!" we'd probably set off a flag and get our conversation flagged for automatic transcription and further analysis.
Re: (Score:2)
Oh, great. Now you've flagged this story in the NSA results as well as everyone commenting on it!
Re:Search Tools (Score:5, Insightful)
Even with good search tools, signal to noise ratio is still important.
Excess data with no correlation to the problems NSA is trying to solve (without getting into a debate over what those are) is simply noise.
Re: (Score:2)
Even with good search tools, signal to noise ratio is still important.
The signal to noise ratio doesn't change when you merely use less data. The whole point of good search tools is to extract the signal from the data, and filter out the noise. If you believe that "less data, but better data" is the answer, then you should also believe that whatever algorithm you use to decide which data are "better" during the collection phase, can also be used to filter the existing data during the analysis. So collecting less data would not help.
The NSA may be wasting resources by colle
Re:Search Tools (Score:4, Insightful)
The signal to noise ratio doesn't change when you merely use less data.
False. Your statement is not true by default; it requires all the data to be known to be of equal quality.
Any time that some data is more strongly correlated than others, your noise is going to go down when you throw out the lower quality data.
Don't wave your hands, think it through and make a logical, reasoned argument.
Re: (Score:3)
your noise is going to go down when you throw out the lower quality data.
Except that if you have an algorithm for recognizing "lower quality data" then you can exclude that data from your search results. So it is not going to affect your results.
Re: (Score:2)
If only they had more money, they could solve the problem.
Re: (Score:3)
The problem is errant and false data. No amount or search or filtering can clear up a poisoned data base. Bad data creates false association and relationships, which means bad results are generated by all searches. You can not clean up bad data without it also taking out good data, you can not get good data without also getting flooded out by bad data. If fact the best defence against those deep total data acquisition system is to simply generate false data and allow those systems to flood themselves with
What if it was streamable (Score:1)
What if there was a way to mark the data in a stream, not storing it permanently but being able to refer to the markers during a specific period of relevance?
Oh but it is effective... (Score:1, Informative)
... they want a google database of peoples data/chats/records and behavior they can use against them at any time for political purposes.
Our brains are much worse at reality and thinking than thought.
Science on reasoning:
https://www.youtube.com/watch?v=PYmi0DLzBdQ [youtube.com]
The (mass surveillance) by the NSA and abuse by law enforcement is just more part and parcel of state suppression of dissent against corporate interests. They're worried that the more people are going to wake up and corporate centers like the US and
oh boy, time to make money (Score:2)
Let's give them the "Big Data!" and "Analytics!" spiel that all the marketing wanks are cramming down our throats. Sounds bites and spending huge bucks on them is the solution!
Re: (Score:2)
you make an assertion without a shred of proof and without knowledge of either the structures of the data nor the tools they currently use on it. are you a Big Data marketing wank? you reason like one.
Wait a minute... (Score:5, Insightful)
Re: (Score:2)
Who knows? Not us, it's classified. It may have helped much in whatever NSA is targeted to or have been totally useless. I don't think that question is that interesting though, the important question is what are the costs of the program. And I'm not talking money - trust, morality and freedoms are worth much more.
Re:Wait a minute... (Score:5, Interesting)
Re: (Score:2)
At least not against terrorism. These people would have turned up in the legal system if caught. AFAIK no actual terrorist act has ever been prevented by the NSA and there would at least be a few were we know about if they were actually effective in that area. On the other hand, their shoddy targeting information for state-sponsored murder has probably created a few terrorists.
Re: (Score:2)
But hey, their budgets and staff count went up so it's a win, right?
If they do not regard themselves as part of the society they help destroy, sure.
Total lie (Score:5, Interesting)
Flood Them Until They Drown. (Score:3)
Re: (Score:3)
Step 1: Develop open-source AI that can carry on rudimentary conversations - occasionally peppering in some words like "bomb" or "ISIS" to trigger NSA searches.
Step 2: Have people register multiple VoIP accounts to run the AIs on.
Step 3. AIs call each other, have conversations, and hang up to call the next AI. (This step repeats ad infinitum.)
Step 4. Sit back and watch as the NSA's servers burst from too much data.
DUH! (Score:5, Interesting)
The NSA and FBI etc are trivial to thwart.. I did it to my ex NSA professor at college.
I bet him a solid 4.0 in his class that I could get an encrypted message past him and he would not be able to detect it. He agreed.
I sent him 10 files 1 had a message that I encrypted. the other 9 had the contents of /dev/random encrypted into them that matched the same bit length message all encryption blocks were 100% identical in size.
I won and was told I cheated.... I asked him if Spies follow rules and get in trouble if they cheat....
Re: (Score:2)
Interesting. Applying this example in a completely different area, I've said that we might not recognize signals from alien life because they would be using alien encryption/compression/protocols that might be indistinguishable from random "data." If your professor couldn't tell which of the 10 human encoded files had real data and what it was, what's the chance of us telling that some signal is actually an alien's video file in an alien codec using an alien compression/encryption algorithm?
Re: (Score:2)
Absolutely, spread spectrum communication is very hard to detect, and we are currently only looking for intentional beacons on a specific frequency based on a scientific guess.
Also looking for signals not beamed into space at insane power levels is pretty much impossible for us. even if a standard 100,000 watt omnidirectional TV transmitter was only 1 light year away we would never be able to detect it's signal because of the inverse square law will put the signal below the background noise well before it
Re: (Score:2)
I had thought it would be obvious to use deltas from true randomness to check whether something had info in it, but I don't think that's what they do.
Re: (Score:2, Funny)
I bet him a solid 4.0 in his class... He agreed.
I'll take 5 mod points in "Things that didn't happen." Alex.
Sun Tzu had it right 3000 years ago (Score:2)
He wanted plenty of spies! They were far cheaper than tons of warriors and all their food supplies and equipment.
Re: (Score:2)
Shhhh! Everyone stop typing so much! (Score:5, Funny)
You insensitive clods! The NSA is having trouble keeping up with all your jibber-jabber!
Re: (Score:2)
If they figured out that being "liberal" or supporting Free Software is just political speech that they should ignore, that would help them pare it down a little. :)
Tracking the Linux Journal readers alone probably costs them a lot of storage and search noise.
Re: (Score:2)
I've been trying to be considerate to the NSA by just posting jibber and leaving out the jabber. Won't someone think of the NSA?
Did you hear about that in your country ? (Score:2)
Re: (Score:2)
They can't exactly delete it before they get in trouble for having it unless they get the others who have it to delete it also. ;) Just my one conspiracy angle on this.
William Binney? That was 15 year ago (Score:2)
Unless you are living in the cave, you should have noticed never ending AI advertisment from IBM: Hi, my name is Watson!
Reality, is that it does not take Binney to say that having too much information is counterproductive. Thus be assured, that military versions of AI, are continuously are poring and monitoring through the dossier files, currently maintained as relationship databases.
You can be assured that there is an automated never-ending surveilance and the code, the AI, the algorithms will get better o
Re: (Score:1)
Not even AIs are going to be able to overcome the nature of false positives. Even if you get them to the same discriminatory level a human filter can have, and presuming they're operating at millions of times the speed of any human filter, you still have to deal with the fact that the intrinsic nature of the data itself makes false positives inevitable.
Re: (Score:2)
Newsflash, you don't have to live in a cave to not watch TV advertising. Wow, talk about "living in a cave." You think everybody does things the same as you!
5% of American households don't even have a TV; up from 1% 10 years ago. Of those who have a TV, many only watch public television, or only watch during 1 sport season.
Many more use time-shifting devices that automatically skip ads.
I've certainly never seen the ad. And I don't live in a cave; only rich people can afford properties with caves. Good luck
Big Brother is Acting (Score:2)
10 years as a civilian? (Score:5, Interesting)
How does he know that the NSA hasn't hired more informaticists in the past 10 years? If I read TFA correctly, he's been out for over a decade. I kind of doubt he's privy to top secret (or higher) information like that, although civilians are granted security clearances too sometimes.
I'm not saying he's wrong, I'm just not clear on HOW he knows what he's saying is accurate. Just so you know, I'm not fan of Patriot Act or the NSA's "hoovering" of data, meta or otherwise.
Re: (Score:2)
Re: (Score:2)
He was in for 30 years prior. I'm sure he made some friends and still has contact with a few of them on the inside. They want to get the message out, they tell him, he tells us.
Re: (Score:2)
It's not a secret the NSA has been hiring more informaticists. He's saying you're not going to fix this with more informaticists, and while he's on the outside the little data he's got since (including from unofficial contacts of which there are a lot) confirms his opinion and his warnings. The article is a year old but his opinion hasn't changed the last 15 years. Data gathering should be targeted, not trying to do a wide sweep.Targeted spying is much more effective, it's more legal and it's more moral. Th
Re: (Score:2)
If the programs worked, we'd be thwarting terror plots more than getting hit by them.. yet that is not what is happening.
How do you know? The FBI arrests plotters and ISIL recruits all the time. And simply because they have not publically disclosed "big plots" that they have broken up doesn't mean that such things haven't happened.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Just didn't want anyone to think I was some government counteragent!
Pretty much everyone in the trade knew that (Score:3)
Re: (Score:2)
9/11 was not an example of not being able to spot valuable information hidden among mountains of chaff. The CIA was actively investigating some of the conspirators prior to 9/11. The CIA managers in charge of that investigation weren't interested in sharing any of the glory from busting the conspirators and so when they entered the USA didn't inform the FBI. The CIA wasn't aware what the plan was but hoped that they'd be leaving the USA again soon so they could continue investigating them and eventually bus
Re: (Score:2)
Re: (Score:2)
Data Hoarders (Score:5, Funny)
DOJ: So NSA... we've got some recorded phone calls here from August 3rd, 2003 between a Darlene [redacted] and her grandson [redacted]
NSA: Yes.. she's born in 1948, lives in Arlington, TX and her SSN is [redacted]. I remember when we first collected those calls.
DOJ: Well then, we listened to this a few times, and it sounds like some fairly innocuous conversation. Nothing criminal whatsoever.
NSA: Right
DHS: So... do you think we can delete these calls then? I mean, there's no..
NSA: NOOOOOO!! There could still be connections to terrorism in those calls... somehow! You never know what we might find on meta-data analysis
DEA: Look... we've identified all the phone references with mentions of drugs, and made copies of those for investigations. We never use the rest of those recordings, and I'm the only one here that really uses those at all. Maybe we could just.. y'know.. delete...
NSA: Don't touch that data! It's mine! I own it!
Wow... (Score:2)
Fish school (Score:2)
Burner phones !!! (Score:3)
Re: (Score:2)
All it takes is doing a few other logged activities at the same time you're carrying the powered-on burner phone and it's linked to you. That could be using any non-cash form of payment, driving home every day, or going to work/school.
Re: (Score:2)
I think you're not quite up on the concept of a "burner phone".
Re: (Score:2)
"or you have to communicate the new information"
You load the new numbers into the address book on the new sim card that you already have to put in the the new phones that you already have to hand out on a regular basis. The end users only need to know which contact name to use for which purpose.
To Paraphrase Frank Zappa (Score:2)
Data is not information.
Information is not knowledge.
Knowledge is not wisdom.
There's a Difference (Score:2)
between data and information.
Free The Data! Let the World See Itself For Once! (Score:3)
This data should be released to the world for all to see along with search tools to suit.
Sociologists and citizens alike could plumb the depths of human behavior for years and finally, for once, get a clear view of political, economical and social alliances in all their (formerly) clandestine glory. Some changes might even result.
It's for later... (Score:2)
It doesn't matter now. They'll store it until they have the capability and need to mine it properly. Data never really goes away, it will all come back to bite us later.... and if they do dump something, it's because its worthless and they probably have something juicier to replace it with. Then again, I'd be surprised if there isn't a backup somewhere. These things have a way of popping back up, long after you had forgotten about it.
It also wouldn't surprise me if this is disinformation designed to put eve
Big hay stack (Score:2)
Re:Don't conflate those things (Score:5, Insightful)
The real problem? An ages-old human tendency: Power corrupts, and absolute power corrupts absolutely. Knowledge is power. Also, power seeks more power. These are no-brainers; no tinfoil hat required; everyone knows this. The NSA (and pretty much every other 'intellgence' organization) always wants more, more, more information, even if they can't use it -- but still they want more. They're like a little kid who discovered sugary candy; it's up to the parents to tell them no. Here in the U.S., citizens must play the role of Parents -- but we haven't been doing our job. The NSA/CIA/FBI/{insert government agency here} has been holding their breath until they turn blue, pitching fits, and throwing their dinner on the floor (read as: doing everything and anything they can to keep us in a constant state of terror) so we'll just give them the candy they want (read as: ability to surveil anyone and everyone) to keep them quiet. What they need at this point is a good spanking on their spoiled little bottoms (read as: U.S. citizens speak the hell up to their representatives and tell them in no uncertain terms that mass surveillance has to stop!) and send them to their room for a good long spell without dessert (no more data for you!). It needed to be done years ago but we've been neglectful, overly-permissive parents. Time to fix that!
Re: (Score:2)
Here in the U.S., citizens must play the role of Parents
Too many children turns everything into the Lord of the Flies. Which Is where I fear we are heading.
Re: (Score:3)
Some of the most persistently corrupt governments are local governments. For example, in China the biggest corruption problems are with regional and local officials rather than with the central party. Larger governments get more scrutiny so that corruption is known publicly instead of being secret, local governments often don't even get local citizens to show up to vote and so end up being more easily controlled by special interests. Not saying that I'm all in favor of giant centralized governments but j
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Today we think of corruption as something like having our decisions swayed by outside money, bribes, special interests, nepotism, etc. But the meaning of the word corruption is broader than the modern usage. Corruption is also the word used for decay and putrefaction. So absolute power metaphorically causes your ideals to decay, your morals to decay, and putrefaction of the soul.
Re: (Score:2)
But if you have true absolute power, you are GOD, YOU decide what is morally right and wrong, there is none to gainsay you, hence you cannot be corrupted.
Re: (Score:3)
Re: (Score:3)
Re: (Score:2)
Sure it's ethics, which means that every individual will have to decide for themself. We have laws and a system to apply them, so an individual can't decide that something should be legal or illegal.
Re: (Score:3)
I don't think governments scale well. The larger the structure the less the possibility of oversight. Why is it that the Canadian government can get things done that the US can't? I suspect dunbar's number is the answer. https://en.wikipedia.org/wiki/... [wikipedia.org]
Or just that it's a Parliamentary system, where the party with the majority chooses the leader and so the party and leader in power are actually in power.
Re: (Score:3)
For some value of "working". Many of the objections to health care, for instance, aren't that we're going to immediately implode from free health care. It's that 50 years down the line, we'll have completely fucked it up. And there is plenty of evidence to show that this could happen.
Add to that the observation that any entitlement that gets into place will become permanent because it will quickly create dependency among voters. What person near retirement age is going to vote to overhaul Social Securit
Re: (Score:2)
Don't you see, it's the inability for folks like you to accept the occasional failure - and those who have vested interest in keeping things as they are, but they could be overcome without you - that keeps us from ever hoping to change anything, and not the entitlement types you speak of.
No, the problem is that people like you only see one solution to a problem. It's the State or nothing. Nothing else works but the State, or the evil insurance companies may destroy us.
I don't mind change, when I feel like someone has actually thought it through. I just see people who think that voting for higher taxes is equivalent to charity, and that government programs are the only possible or desirable solution for badly managed health care. "Change" and "progress" doesn't mean "the government does
Re:Don't conflate those things (Score:5, Insightful)
The reason the U.S. government can't get anything done these days is this Republican idea that Government isn't good for anything. This leads to putting people in charge of government agencies that don't believe that government can do anything. Think back to Heckofajob Brownie during hurricane Katrina. The guy had no emergency management experience and ended up running the worlds largest emergency management organization. This is pervasive through many agencies though and leads to a self fulfilling prophecy.
We were able to effectively end childhood hunger in the 70s, we were able to create the national highway system, put a man on the moon, lots and lots of large projects that were ultimately very successful but now a large chunk of the country thinks we don't even need the IRS anymore. We have a Presidential candidate campaigning on that very idea. Cruz is probably equally as crazy as Trump but they represent a good solid chunk of the population.
I hate how the conversation has been turned into Government should do everything versus nothing. That's why I like Bernie, he thinks Government could do more but recognizes that some things are better in private hands, he probably goes too far but when you think of it more as a direction instead of an overnight mandate things look a whole lot more sane. My health or my parent's health should not come down ensuring someone makes a profit for a hospital stay. Injecting money into health is counter productive, much the same way insurance is. Insurance companies should take our money and use it to invest, but they have people who's job it is to deny you your claim rather than figuring out how to fairly deal with a situation. So we pay care insurance for years and don't use our benefits, the moment we have a traffic accident we have to start paying more, or we have to hire a lawyer to make sure the insurance company actually provides the coverage they promised.
Re: (Score:2)
I think politics insists on having a tug of war contest. So if you try to win your tug of war by standing in the center and holding the middle of the rope you'll never accomplish anything, so the game show contestants running for office stake out extremists positions by default. Many of them then lose sight of the fact that the vast majority of the country is in between those extremes. So Bernie is a good counterbalance to Cruz.
The point of insurance, in the minds of insurance companies, is to create inc
Re: (Score:2)
"My health or my parent's health should not come down ensuring someone makes a profit for a hospital stay"
You could say a similar statement about a roof over your head, or food on your table.
But as my father is a food producer, should he have to provide service to you for free? He's self employed so everything he makes as profit goes to putting food on the kitchen table (we were not rich growing up).
Re: (Score:2)
Sure, you could say the same about food and shelter. The big differences are that they are usually predictable expenses. I know pretty much what my mortgage payment will be five years from now, so I can budget it. Medical costs are unpredictable. I can go years without incurring serious expenses, and then suddenly have to have treatment costing tens of thousands of dollars. (Been there, done that.)
I assume your father would be happy to provide service in return for payment, and the UBI would give mo
Re: (Score:2)
We were able to effectively end childhood hunger in the 70s,
... by redefining it to food-insecure [feedingamerica.org]. Not necessarily the best example.
Re: (Score:2)
The Republicans are the ones propagandizing that government doesn't work, and some of them are willing to do pretty much anything to prove it. Democrats are more trusting of government.
I see that AC has not looked over Bernie's campaign site. In the Issues section, there's details on exactly how all of Bernie's changes would be paid for. It's reasonable to argue about this (and how good the changes would be, for that matter), but we need to give Bernie credit for making such an argument possible.
Re: (Score:2)
But we have so many examples of smaller government bodies having ineffectual oversight. In American everyone's paying attention to the presidential election but so few are worrying about who's getting elected to their local school board and city council. In California we've had several municipalities that went bankrupt or nearly so based upon just a few individuals controlling the money and investing it badly, or a few individuals making broad decisions about long term retirement benefits for city workers,
Re: (Score:2)
Both size and homogeneity are critical factors in successful programs. Look at Canada or Denmark... or New Hampshire. Small populations, not very much actual diversity.
They also don't have the same traditions as the US as a whole. I daresay, it has helped them in some ways, but I think some of this is also what allows us to be a leading country rather than a country content to fall into line. We have the ambition to do things, which makes us loud and brash, but also got us to the Moon.
Mostly, I think it
Re: (Score:2)
Welcome to Slashdot. Two drink minimum.
Re: (Score:2)
> Put simply, what the hell sort of problem do you have with some consistency in global law?
I'll bet you anything you're not ok with the second amendment, and would want to eliminate it.
People are different.
Cultures are different.
Re: (Score:3)
Yes, because nearly-open borders have worked out so well recently in Belgium and France and other places in the EU.
Only an irresponsible fool allows any random stranger off the street to enter their home with all their loved ones' lives and all their possessions at risk.
smi (so many idiots)
Strat
Re: (Score:2)
This is total B.S. With that facility in Utah and who knows where else... there is an info orgy and they are loving it.
Trouble is, they made an orgy and expect to get married.
Re: (Score:2)
This is total B.S. With that facility in Utah and who knows where else... there is an info orgy and they are loving it.
Trouble is, they made an orgy and expect to get married.
Not a problem, they're in Utah. [instantrimshot.com]
Re: (Score:3)
Like a trip to hell, one-way! Or Detroit!
Which reminds me [youtube.com]
Re: (Score:3)
either that or Chicago
Re: (Score:2)
Every call, funding, fax, digital account, chat room, file, propaganda support, contact and context.
The problem with the NSA and GCHQ thinking on this is bulk collection just gets too much random data after event data thats junk.
The other problem is acting on such collected information is letting the world know of such methods. Use the gathered result