Snowden Used Software Scraper, Say NSA Officials 227
An anonymous reader writes with this excerpt from the New York Times: "Intelligence officials investigating how Edward J. Snowden gained access to a huge trove of the country's most highly classified documents say they have determined that he used inexpensive and widely available software to 'scrape' the National Security Agency's networks, and kept at it even after he was briefly challenged by agency officials. Using 'web crawler' software designed to search, index and back up a website, Mr. Snowden 'scraped data out of our systems' while he went about his day job, according to a senior intelligence official. 'We do not believe this was an individual sitting at a machine and downloading this much material in sequence,' the official said. The process, he added, was 'quite automated.'"
Stunning. (Score:5, Insightful)
Who'd have thought? Experienced IT guy didn't manually download each file!?
"Inexpensive and widely available" - I hope they don't mean some evil subversive communist open-source tool.
Re:Stunning. (Score:5, Funny)
Oh my god .... could it be .... wget ?
Re:Stunning. (Score:4, Funny)
Re:Stunning. (Score:5, Informative)
"Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo."
Even funnier is the assertion that such "web crawling" would be easy to detect. As someone who has done remote automation and data scraping for a living, I can tell you that it doesn't look any different than any other web traffic.
About the only way to detect it is to do traffic analysis, to see if the same IP address is hitting nodes a lot, or hitting many nodes in a short period of time, and especially if they are rapid-fire.
But the latter is easy to get around. I won't say just how here, because even if it's not hard to figure out it's still something of a trade secret.
Re:Stunning. (Score:5, Insightful)
Actually there are options in wget for that.
Re: (Score:3)
"Actually there are options in wget for that."
Well, yes. But while wget could be part of a full-featured crawler, it just doesn't have the functionality to do it all, by itself.
For example: while you can download a whole directory with wget, you have to know that directory exists in the first place. wget does not help you with that part.
Re:Stunning. (Score:4, Funny)
Re: (Score:3)
From the TFA:
So no -- not wget. Unless the NSA is lying about it.
Re:Stunning. (Score:5, Interesting)
The very first program (after "Hello World") I wrote in Java was a website scraper. I used it to download all of Sun's API and tutorial pages for Java and rewrite links to be relative. Younger and dumber. This created two copies of each set of docs: The scraped version and the compressed version, which I only discovered existed after having scraped the downloadable offline version of said docs.
Point being: My scraper was written in a few hours and far less powerful than wget.
Re:Stunning. (Score:5, Insightful)
You're ALL missing the point.
This is not a "news item" because he used some "arcane technology" from the view of the uninitiated.
This is a calculated iteration in the advancement of declaring web automation tools "instruments of terror" and "cyber weapons", for prosecutorial purposes.
You vilate a ToS with these, and WHAM! Terror suspect!
Re: (Score:3)
Have a tinfoil hat.
Yes. I will place it on the grave of Aaron Schwartz
Re: (Score:3)
Re: (Score:2)
Re:Stunning. (Score:5, Funny)
Re: (Score:2)
I read TFA and I still can' figure out... who's "Lou"?
Re: (Score:3)
Yep. 100% sure. I've even researched how so many people can believe this is an actual quote when it isn't (which is a strange phenomenon). I'm also a huge Simpsons geek.
The actual quote is: Wiggum: Well that's some good work, Lou. You'll make sergeant for this.
But almost universally people say and believe it to be "That's some (mighty) fine detective work, Lou"
Re: (Score:2)
Thank you.
Re: (Score:2)
It's a Clancy Wiggum quote. [wikipedia.org]
I don't recall the episode.
Re: (Score:3)
I'd have thought he went in each day with wheelbarrow full of 1.44" floppies and just copied until he got it all... That's some mighty fine detective work, Lou.
Apparently the "wheelbarrow full of 1.44" floppies" was actually what their security was set up to prevent.
Re:Stunning. (Score:4, Funny)
And the joke was on them--he was actually stealing wheelbarrows!
Re: (Score:2)
Is "wheelbarrow full of 1.44" floppies" anything like "binders full of women"?
Re: (Score:2)
They tried to use an automated tool in developing healthcare.gov but were told it was classified. Someone argued, I think, but top management fired him. Automated tools are no way to get more direct reports, you know. And we need unemployment to go down, not up.
Did you know you can double your "lines of code" output with just a few keystrokes? Write for more info!
Re:Stunning. (Score:4, Funny)
Re: (Score:3)
But why would someone with admin level access want to scrape the website rather than just take a backup of the database?
Re:Stunning. (Score:5, Insightful)
Well if you knew a SIEM system had rules which might trigger alters if a database backup is started off hours or if the backup files are accessed for one. As apposed to normalish get query logs with 2XX results, its likely been trained to ignore.
Questioning if who you might eventually leak the data to will have the technical chops and resources put the information together from the database file, as opposed to just reading through a bunch of handy precomputed html pages and office documents for another.
Re:Stunning. (Score:5, Funny)
the malicious and now banned weapons grade tool is called "wget" by underground unix hackers.
Re:Stunning. (Score:5, Insightful)
Why?
Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.
Re: (Score:2)
Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.
What? You're using logic and critical thinking? You must be a terrorist and/or traitor.
Re: (Score:2)
According to the BBC [bbc.co.uk] the new Director of the NSA says:
"There's no place where it's an analyst and a database and you can search for whatever you like and there's no record and no after the fact," Mr DeLong says.
So it should be pretty easy for them to figure out which information Snowdon got and when. Unless nowhere means unless outside of Fort Meade...
Phillip.
Re:Stunning. (Score:5, Interesting)
Now that we have documented proof of a rogue government agency that ignores the law and then lies about it, I'm waiting for some enterprising criminal defense attorney to realize they've got the perfect patsy. Regardless of what crime their client is alleged to commit, just deny involvement in the crime and claim that it was committed by the NSA.
I'm not a lawyer, I just play one on Slashdot. But it seems to me that should be sufficient to raise reasonable doubt.
Re: (Score:3)
There's zero reason to believe the NSA's version of this and every reason to believe Snowden's
Why?
Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.
Once you start using absolutes, we're past the point where every single thing you want to believe is true, and every single thing you don't is a lie.
Consider that.
Re: (Score:3)
Because so far, Snowdon hasn't yet been proven wrong, and the NSA has failed to offer any sort of rationalization that even approximates truth.
Is that better?
Re: (Score:3, Insightful)
I think many here are missing the point. Point the Moon with your finger and the fool will look at the finger.
The entire point about the use of automated tools to scrape data here and there on the NSA network is that Snowden wasn't going at the only data he needs to prove the point he says he wants to make. He was just grabbing a full load of data hoping for some of it to prove something that could make him a credible whistleblower. This is playing against him if he would have to convince a judge he is a "l
Re: (Score:2)
Re: (Score:2)
I like to think of this article as evidence the NSA is manned by ineffectual morons who should be banned from sharp table cuttlery.
I, living in this nation, feel less secure because of the existence of this agency, ironic, isnt it?
Is this really what, Im paying for? Bullshit! I want ALL my tax dollars BACK! Including Medicare and Social security!
I can see that it all an elaborate bait and switch conducted by charlatans like President Clinton and her sidekick Omama.
Snowden COULD be the first President electe
Re: (Score:2)
zomg !
BAN WGET !
(don't ban, fuck beta)
Re: (Score:2)
Next week's report: Snowden used flexible plastic card to break into secure server room. These commie cheats will stop at nothing, I tell you. Better double NSA's funding.
Wow... (Score:5, Insightful)
Knock me over with a feather, spooks. You fucking hired people to build what is probably the largest collection of signals intelligence scraping systems on the planet, targeted at a wide variety of differently structured systems. Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?
(Oh, incidentally, maybe you should spend a bit less time reading everybody's email and work on that 'hilarious leaked diplomatic calls' problem, I'm told that sort of thing used to be your job at some point in the past...)
Re: (Score:2)
Please consider the bureaucracy, of the NSA. They obviously have both technical talent with which to operate and that other thing.
Also, don't forget about weekends. People wants weekends, and their downtime.
Peoples' downtime is like a cancer in any 24/7 bureaucracy as well.
Re:Wow... (Score:5, Insightful)
From TFA:
Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade, Md., which was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded, he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of what the agency's newly appointed No. 2 officer, Rick Ledgett, recently called "the keys to the kingdom" raised few alarms. "Some place had to be last" in getting the security upgrade, said one official familiar with Mr. Snowden's activities. But he added that Mr. Snowden's actions had been "challenged a few times".
So they knew he was doing it, even questioned him, and he still got away with the data. To the people who maintain the NSA has the best and brightest security people perhaps they (NSA security) should use that expertise to improve their own security instead of weakening everyone else's.
And yes, this is precisely why they must not be trusted with the data they are gathering due to mass surveillance.
Re: (Score:2)
Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?
Because all the middle management MBAs don't have flying clue how computers work and feel the need to compensate for their perceived inferiority in the face of the technical employees who deliver real value. Demonizing them is the easiest way to accomplish that.
Re: (Score:3)
Yep, that's how I'd do it, if I had access to those kinds of networks and an overwhelming curiousity (or suspicion). Not that I'd ever do anything like that of course [glances over shoulder] .. or even think of harming one of our most valuable national assets [glances at black helicopter over trees and discounts] ...
Excuse me, there's someone at the ...
Re:Wow... (Score:4, Insightful)
In Soviet Slashdot (Score:5, Funny)
Beta scrapes you!
...and that makes it better? (Score:4, Insightful)
If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....
What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!
Re: (Score:2)
What if a legitimate foreign hacker was ...
As opposed to a bastard foreign hacker?
Re: (Score:2)
If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....
What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!
This is the other big glaring issue... Forgetting that the constitution prohibits the sort of forced collection of people's data for a second, which we really really shouldn't forget ... with the ability and clear intention to eventually put all the business and communications data in the US into one giant repository the NSA is saying they can protect access to it, but the Snowden leaks are a very big glaring example that some policy isn't in control, people are.
And let's not forget that Snowden knew he
Re: ...and that makes it better? (Score:5, Insightful)
"I'm all for whistleblower status, but Snowden should have been DEAD, HORRIBLY"
You sure downt sound like you are for whistleblower. I dont many people forget that the NSA violated many US laws.
Re: (Score:2)
Seems many terrorist organizations seem to feel the same way as you, what makes you different from them?
Re: (Score:2)
Our rulers do make exactly such decisions, things like invading an entire country on a made up pretext, or using drones to blow up weddings. The kinds of people in control of the Executive branch have staged a soft coup, usurping legislative and judicial powers (e.g., signing statements, due process free execution/detention, executive declared wars, extremely broad executive orders). To think that such evil motherfuckers would not blow up a plane is ridiculous.
I was at a presentation in Tacoma last night
Re: ...and that makes it better? (Score:5, Insightful)
You know, whether you agree or disagree with what Snowden did, that in no way justifies killing him without a... oh, what was that quaint thing we used to require? That's right, a trial. Rule of law, and all that. I think that's what the country was based on originally.
Of course, it's embarrassing for the NSA that Snowden waltzed out with so much confidential information, and arguably he should have been ARRESTED within 24 hours of "that flight to Hong Kong", but killed? To even think that sort of thing is disturbing.
Having said that, I am glad he managed to get away, since his revelations are shining an absolutely necessary light on the murky behavior of our government and its actions. An educated populace is necessary to ensuring our freedoms and for too long the government has been hiding its wrong-doings from the ones it purports to serve. Whether Snowden acted as a foreign agent, or for his own advantage, or out of idealism, his actions were necessary and should not be so readily scorned.
(oh right, and fuckbeta and all that jazz. It's gonna be hard doing that boycott tomorrow)
Re: ...and that makes it better? (Score:4, Insightful)
You know, whether you agree or disagree with what Snowden did, that in no way justifies killing him without a... oh, what was that quaint thing we used to require? That's right, a trial. Rule of law, and all that.
If there were to be a trial it is almost certain they would exclude pretty much all avenues of defense [techdirt.com] that support what he actually did and why. Rule of Law is no more in this country. Just ask Aron Schwartz, Bradley Manning and the host of other whistle blowers prosecuted by the self proclaimed most open administration in history. If they want you gone they simple twist the millions of laws that exist and make up new interpretations if that's not enough. But you will be gone.
Re: (Score:2)
Hey you forgot the part where to be able to preserve the way of life "some" murky actions are necessary ,and of course "they" think we can't handle the truth.
You mean the "way of life" where we're free people, not spied on by a fascist government, and executed whenever we piss off the king/emperor? Yeah....that's been preserved REAAAALLY well...
The trick (Score:4, Interesting)
"because he worked at an agency outpost that had not yet been upgraded with modern security measures."
"when he was questioned, Mr. Snowden provided what were later described to investigators as legitimate-sounding explanations for his activities"
Speechless.
Re: The trick (Score:2)
I, too, an speechless.
The NSA scrapes terabytes of data per day in the hope of scanning through it to find juicy tidbits.
The very idea that an NSA analyst should be scanning through data to find juicy tidbits should have raised red flags with the finest Keystone Cops.
Seriously people, this was his job. OF COURSE it was explainable. Hell, the tools he used were probably provided to him.
Re: (Score:2)
His job was to find juicy tidbits in data scraped from external sources. His job was definitely NOT to find juicy tidbits in internal NSA documentation. The fact that he could easily and massively access this documentation without anyone seriously questioning his activities is a huge problem.
The assumption could be made that internal documentation and externally sourced data are stored on the same servers, and accessed using the exact same methods.
There is no possible explanation for this which doesn't in
Middle Initial (Score:2, Interesting)
Anyone else notice that Snowden is increasingly being referred to as "Edward J. Snowden" instead of just "Edward Snowden"?
Re: (Score:3)
Huh. Just like Homer J. Simpson. Coincidence? I'll bet all my mod points on it!
Maybe this is in reference to Jay Ward?
Could this possibly be a conspiracy, brought to you by the same evil mind that gave us Rocky, Bullwinkle, Natasha, and Boris? Stay tuned for the next adventures of...
https://en.wikipedia.org/wiki/... [wikipedia.org]
Re:Middle Initial (Score:4, Funny)
Jerry: David Berkowitz, Ted Bundy, Richard Speck...
Alice: What about them?
Jerry: Serial killers. Serial killers only have two names. You ever notice that? But lone gunmen assassins, they always have three names. John Wilkes Booth, Lee Harvey Oswald, Mark David Chapman...
Alice: John Hinckley. He shot Reagan. He only has two names.
Jerry: Yeah, but he only just shot Reagan. Reagan didn't die. If Reagan had died, I'm pretty sure we probably would all know what John Hinckley's middle name was.
When you are a criminal (Score:2, Insightful)
You use proper tools.
Re:When you are a criminal (Score:5, Insightful)
When you are anyone trying to do anything efficiently (such as the legally questionable automated gathering and storage of records of millions of phone calls and text messages?), you use proper tools.
Re: (Score:2)
Snowden clearly broke the law. NSA, not so clear.
Re: (Score:3)
Re:When you are a criminal (Score:5, Insightful)
Also, I don't care what think your motive is, you don't turn over classified documents to the enemy.
And there it is ... the free press is nurb432's enemy.
Re: (Score:2)
Extra! Extra! Read all about it! Normandy Invasion D-Day set for next week! Generals are concerned!
Re: (Score:2)
Which enemy? The rest of the world?
Re: (Score:2)
Congress. They leak _everything_.
Re: (Score:2)
Which enemy? The rest of the world?
Angela Merkel.
You are confused..... (Score:3)
I'm trying to see how your sig. fits with your comment, and can only conclude you have no clue....
Re: (Score:2)
Yeah, it's clear.
Snowden is a patriot; the NSA is treasonous (Score:5, Insightful)
The idea of military specialists of whatever type being employed against the society they belong to, is treasonous and fucking retarded no matter what legal acrobatics are employed in their defense.
You may have some sort of mystic devotion to the law, but I believe laws are made by (generally corrupt) men for their own interests, and I am familiar enough with the world outside the borders and political influence of the United States to know there is an enormous difference between legality and rightousness. The U.S.A. may not be the kind of country where you are expected to bribe every public official however minor -- we generally reserve that for higher office. It takes a special kind of idiocy to use military forces against their homeland, though.
Government at its core is the body to which we have delegated our inherent right to violence -- a right being defined in this case as something which cannot be taken from you. We delegate this right to others, specialized in its use, with the express understanding that [a] as applied to civilian life, the exercise of violence by police will be applied fairly and equally as men can manage, and [b] that the unrestricted expression of this (as embodied by military force) be only employed against our enemies. War is hell, and we do not bring hell home.
Snowden is a patriot, and the NSA is treasonous -- whether or not the law can be made to serve whichever purpose. Beyond all other argument, potentially felonious violation of the law is so common with the continual proliferation of laws that lawfulness cannot be the only measure of either justice or rightousness. May all those who support the NSA have a fair trial.
Re: (Score:3)
Snowden clearly broke the law. NSA, not so clear.
Why NSA phone-records spying is totally, utterly illegal
http://boingboing.net/2014/01/29/nsa-phone-records-spying-is-to.html/ [boingboing.net]
That evil program, wget (Score:2)
Re: (Score:2)
no no no no. You fail to understand the intelligence at work here. The proper command would be: /dev/urandom > /dev/sda
sudo cat
Dawww, this cat is ultra random! Crazy like a devil / silly damn animal.
Amused (Score:5, Funny)
"This automated and indiscriminate bulk collection of data is unacceptable!"
Re:Amused (Score:4, Informative)
Oddly, government is complaining that people will be able to take the various facts that he assembled and figure out what we're really up to. You know, the kind of thing they say they can't do with our metadata.
Re: (Score:3)
All he got was metadata.
How Many More NSA Employees? (Score:5, Interesting)
Re:How Many More NSA Employees? (Score:5, Insightful)
Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?
I'm sure the NSA assumes they have moles, and none of the data Snowden released is a surprise to the Russians or Chinese.
The NSA was just not prepared for the truth to leak to their real enemy - the general public.
Re: (Score:2, Insightful)
... to their real enemy - the general public.
That is such a load of crap.
Why? I'm quite sure that most governments at the very least had a general idea of what the NSA was up to before Snowden's leak, it's only members of the general public that would get painted as conspiracy theorists and ignored whenever the they tried to draw attention to this sort of thing.
Re: (Score:3)
That is such a load of crap.
How so? The real enemy of any bureaucracy is those who would dismantle, or at least limit it, in terms of power and growth. No foreign power is going to do that to the NSA. If (and I'm afraid it's a big if) anything limits the power and growth of the NSA it will be the outrage of the American people. Therefore the American people are the true enemy of the bureaucracy.
Re: (Score:3, Insightful)
As others have said: "How so?"
It's a logical conclusion based on the available evidence: No safeguards were in place to defend against an analyst stealing data and giving it to someone else, despite this being an obvious threat the NSA could not possibly have been unaware of.
No such measures were taking until someone (i.e. Snowden) leaked this information to the public. Add this to the extremely negative way in which the NSA and the entire administration talks about journalists reporting on this, and the
Re: (Score:3)
The NSA puts too much trust in it's employees obviously.
Re: (Score:2)
All of them. That's why they fired all their IT guys.
Re: (Score:2)
Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?
As has been previously demonstrated on Slashdot the number will be assumed to be as many as are needed to render Snowden's crimes "meaningless," so "therefore he should be pardoned."
Re: (Score:2)
Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?
Is this some variation of "If a tree falls in the woods, and nobody is around to hear it, and it hits a mime, does anyone care?"
Useless NYT article .. (Score:5, Insightful)
This is retrospective ass-covering cyberbullshit. It is precisely at the edge that the security attacks would come from. What they were doing putting such material on Web servers and Wikis beggers credulity. Didn't senior management not realize that as keepers of the nations secrets they would be subject to attacks both internally and externally. Given the state of non-security at the NSA I would suspect that Snowden wasn't the only hostile with access to the “the keys to the kingdom”.
Re: (Score:2)
"Sure, we left the windows open, but the door was impenetrable"
Re: (Score:2)
These people are part of the same defense establishment that thought it was a good idea to make CD writers available on "secured" networks where people like Manning could access them. The people working for defense contractors have fairly well secured and locked down systems because there are serious financial penalties for unintentional disclosures. Within the DoD institutions themselves, however, it's an anything goes wild-west in the name of expediency. In that sphere, it's all a good 'ol boys club where
In his defense (Score:5, Funny)
In his defense mr. Snowden explained that his scraper was only gathering metadata and therefore by their own standards the NSA has nothing to worry about.
When asked for comment NSA director James Clapper replied : "Whoa that's deep bro!"
Re: (Score:3)
And in fact Snowden could not be convicted for anything in relation to a document that he didn't read himself but that he might only have given to journalists to "back up".
There no conclusive proof he looked at anything besides metadata.
A friendly reminder: (Score:5, Insightful)
There's absolutely zero reason to believe anything the NSA says about how Snowden got the documents, or indeed, about anything. They believe they are entitled to lie to congress, so the public isn't even a question.
Regulate it (Score:2)
Maybe I'm missing the point (Score:2)
Shouldn't the shock and horror be that Snowden was able to scrape the juiciest pages from the NSA information dump on basically everyone, without so much as a 403 error?
Re: (Score:2)
To reply to myself; no, the shock and horror should be that there is a database out there with everything in it. :/
Re: (Score:2)
Shouldn't the shock and horror be that Snowden was able to scrape the juiciest pages from the NSA information dump on basically everyone, without so much as a 403 error?
It was the intranet - I guess they trusted everybody with an AD account ;-)
I believe, though, it's no coincidence that Snowden ended up in the HW office. He was probably aware of the lack of security when he requested the transfer.
God only knows how many guys have downloaded that data before him and sold it to the highest bidder.
Re: (Score:2)
It's terrifying, really, that it seems like there's no granular, highly-tuned security system in-place for all this; rather a "You have permission to view", "You do not".
SCP Script? (Score:2)
That spoonfull of sugar (Score:2)
That spoonful of sugar can't mask the bitter taste of your own medicine, can it, fancy espionage agency?
Re: (Score:2)
I look forward to the day when sNOwden hangs for high treason against the government spooks of the United States of America.
FTFY.