Forgot your password?
typodupeerror
Government Software United States

Snowden Used Software Scraper, Say NSA Officials 227

Posted by timothy
from the what-would-christopher-boyce-do? dept.
An anonymous reader writes with this excerpt from the New York Times: "Intelligence officials investigating how Edward J. Snowden gained access to a huge trove of the country's most highly classified documents say they have determined that he used inexpensive and widely available software to 'scrape' the National Security Agency's networks, and kept at it even after he was briefly challenged by agency officials. Using 'web crawler' software designed to search, index and back up a website, Mr. Snowden 'scraped data out of our systems' while he went about his day job, according to a senior intelligence official. 'We do not believe this was an individual sitting at a machine and downloading this much material in sequence,' the official said. The process, he added, was 'quite automated.'"
This discussion has been archived. No new comments can be posted.

Snowden Used Software Scraper, Say NSA Officials

Comments Filter:
  • Stunning. (Score:5, Insightful)

    by quenda (644621) on Sunday February 09, 2014 @09:29AM (#46202069)

    Who'd have thought? Experienced IT guy didn't manually download each file!?

    "Inexpensive and widely available" - I hope they don't mean some evil subversive communist open-source tool.

    • by Anonymous Coward on Sunday February 09, 2014 @09:55AM (#46202183)

      Oh my god .... could it be .... wget ?

      • by s.petry (762400) on Sunday February 09, 2014 @02:06PM (#46203617)
        Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo.
        • Re:Stunning. (Score:5, Informative)

          by Jane Q. Public (1010737) on Sunday February 09, 2014 @03:31PM (#46204337)

          "Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo."

          Even funnier is the assertion that such "web crawling" would be easy to detect. As someone who has done remote automation and data scraping for a living, I can tell you that it doesn't look any different than any other web traffic.

          About the only way to detect it is to do traffic analysis, to see if the same IP address is hitting nodes a lot, or hitting many nodes in a short period of time, and especially if they are rapid-fire.

          But the latter is easy to get around. I won't say just how here, because even if it's not hard to figure out it's still something of a trade secret.

          • Re:Stunning. (Score:5, Insightful)

            by cheesybagel (670288) on Sunday February 09, 2014 @04:51PM (#46204965)

            Actually there are options in wget for that.

            • "Actually there are options in wget for that."

              Well, yes. But while wget could be part of a full-featured crawler, it just doesn't have the functionality to do it all, by itself.

              For example: while you can download a whole directory with wget, you have to know that directory exists in the first place. wget does not help you with that part.

        • by Cyberax (705495) on Sunday February 09, 2014 @04:16PM (#46204699)
          Well, technically all Perl scripts are 'voodoo' by default.
      • by anagama (611277)

        From the TFA:

        [Chelsea Manning] had used a program called âoewgetâ to download the batches of files. That program automates the retrieval of large numbers of files, but it is considered less powerful than the tool Mr. Snowden used.

        So no -- not wget. Unless the NSA is lying about it.

    • by jsh1972 (1095519) on Sunday February 09, 2014 @10:10AM (#46202251)
      I'd have thought he went in each day with wheelbarrow full of 1.44" floppies and just copied until he got it all... That's some mighty fine detective work, Lou.
      • I read TFA and I still can' figure out... who's "Lou"?

        • Here's what the internet has to say ( http://www.reddit.com/r/gaming... [reddit.com] )

          Yep. 100% sure. I've even researched how so many people can believe this is an actual quote when it isn't (which is a strange phenomenon). I'm also a huge Simpsons geek.

          The actual quote is: Wiggum: Well that's some good work, Lou. You'll make sergeant for this.

          But almost universally people say and believe it to be "That's some (mighty) fine detective work, Lou"

        • It's a Clancy Wiggum quote. [wikipedia.org]
          I don't recall the episode.

      • by bigpat (158134)

        I'd have thought he went in each day with wheelbarrow full of 1.44" floppies and just copied until he got it all... That's some mighty fine detective work, Lou.

        Apparently the "wheelbarrow full of 1.44" floppies" was actually what their security was set up to prevent.

      • They tried to use an automated tool in developing healthcare.gov but were told it was classified. Someone argued, I think, but top management fired him. Automated tools are no way to get more direct reports, you know. And we need unemployment to go down, not up.

        Did you know you can double your "lines of code" output with just a few keystrokes? Write for more info!

    • by fatphil (181876) on Sunday February 09, 2014 @10:17AM (#46202277) Homepage
      But you don't understand. Terrorists use wget for drug dealing with paedophiles who use slashdot beta. Wget must be banned!
    • by Bert64 (520050)

      But why would someone with admin level access want to scrape the website rather than just take a backup of the database?

      • Re:Stunning. (Score:5, Insightful)

        by DarkOx (621550) on Sunday February 09, 2014 @11:06AM (#46202521) Journal

        Well if you knew a SIEM system had rules which might trigger alters if a database backup is started off hours or if the backup files are accessed for one. As apposed to normalish get query logs with 2XX results, its likely been trained to ignore.

        Questioning if who you might eventually leak the data to will have the technical chops and resources put the information together from the database file, as opposed to just reading through a bunch of handy precomputed html pages and office documents for another.

    • by gl4ss (559668) on Sunday February 09, 2014 @10:54AM (#46202469) Homepage Journal

      the malicious and now banned weapons grade tool is called "wget" by underground unix hackers.

    • Re:Stunning. (Score:5, Insightful)

      by Arrogant-Bastard (141720) on Sunday February 09, 2014 @10:54AM (#46202473)
      There's zero reason to believe the NSA's version of this and every reason to believe Snowden's

      Why?

      Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.
      • Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.

        What? You're using logic and critical thinking? You must be a terrorist and/or traitor.

      • by horza (87255)

        According to the BBC [bbc.co.uk] the new Director of the NSA says:
        "There's no place where it's an analyst and a database and you can search for whatever you like and there's no record and no after the fact," Mr DeLong says.

        So it should be pretty easy for them to figure out which information Snowdon got and when. Unless nowhere means unless outside of Fort Meade...

        Phillip.

      • Re:Stunning. (Score:5, Interesting)

        by dcollins117 (1267462) on Sunday February 09, 2014 @12:24PM (#46202987)

        Now that we have documented proof of a rogue government agency that ignores the law and then lies about it, I'm waiting for some enterprising criminal defense attorney to realize they've got the perfect patsy. Regardless of what crime their client is alleged to commit, just deny involvement in the crime and claim that it was committed by the NSA.

        I'm not a lawyer, I just play one on Slashdot. But it seems to me that should be sufficient to raise reasonable doubt.

      • There's zero reason to believe the NSA's version of this and every reason to believe Snowden's

        Why?

        Because, so far, every single thing that Snowden has said has turned out to be true when cross-checked. And, so far, every NSA official spokesperson has been caught repeatedly lying.

        Once you start using absolutes, we're past the point where every single thing you want to believe is true, and every single thing you don't is a lie.

        Consider that.

        • Because so far, Snowdon hasn't yet been proven wrong, and the NSA has failed to offer any sort of rationalization that even approximates truth.

          Is that better?

    • Re: (Score:3, Insightful)

      by AchilleTalon (540925)

      I think many here are missing the point. Point the Moon with your finger and the fool will look at the finger.

      The entire point about the use of automated tools to scrape data here and there on the NSA network is that Snowden wasn't going at the only data he needs to prove the point he says he wants to make. He was just grabbing a full load of data hoping for some of it to prove something that could make him a credible whistleblower. This is playing against him if he would have to convince a judge he is a "l

    • by flyneye (84093)

      I like to think of this article as evidence the NSA is manned by ineffectual morons who should be banned from sharp table cuttlery.
      I, living in this nation, feel less secure because of the existence of this agency, ironic, isnt it?
      Is this really what, Im paying for? Bullshit! I want ALL my tax dollars BACK! Including Medicare and Social security!
      I can see that it all an elaborate bait and switch conducted by charlatans like President Clinton and her sidekick Omama.
      Snowden COULD be the first President electe

    • by richlv (778496)

      zomg !

      BAN WGET !

      (don't ban, fuck beta)

    • Next week's report: Snowden used flexible plastic card to break into secure server room. These commie cheats will stop at nothing, I tell you. Better double NSA's funding.

  • Wow... (Score:5, Insightful)

    by fuzzyfuzzyfungus (1223518) on Sunday February 09, 2014 @09:30AM (#46202075) Journal
    You mean to tell me that an NSA tech contractor used wget or something, rather than loading up IE6 and clicking until his fingers fell off?

    Knock me over with a feather, spooks. You fucking hired people to build what is probably the largest collection of signals intelligence scraping systems on the planet, targeted at a wide variety of differently structured systems. Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?

    (Oh, incidentally, maybe you should spend a bit less time reading everybody's email and work on that 'hilarious leaked diplomatic calls' problem, I'm told that sort of thing used to be your job at some point in the past...)
    • by SpzToid (869795)

      Please consider the bureaucracy, of the NSA. They obviously have both technical talent with which to operate and that other thing.

      Also, don't forget about weekends. People wants weekends, and their downtime.

      Peoples' downtime is like a cancer in any 24/7 bureaucracy as well.

      • Re:Wow... (Score:5, Insightful)

        by dcollins117 (1267462) on Sunday February 09, 2014 @12:04PM (#46202879)

        From TFA:

        Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade, Md., which was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded, he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of what the agency's newly appointed No. 2 officer, Rick Ledgett, recently called "the keys to the kingdom" raised few alarms. "Some place had to be last" in getting the security upgrade, said one official familiar with Mr. Snowden's activities. But he added that Mr. Snowden's actions had been "challenged a few times".

        So they knew he was doing it, even questioned him, and he still got away with the data. To the people who maintain the NSA has the best and brightest security people perhaps they (NSA security) should use that expertise to improve their own security instead of weakening everyone else's.

        And yes, this is precisely why they must not be trusted with the data they are gathering due to mass surveillance.

    • Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?

      Because all the middle management MBAs don't have flying clue how computers work and feel the need to compensate for their perceived inferiority in the face of the technical employees who deliver real value. Demonizing them is the easiest way to accomplish that.

    • by Toad-san (64810)

      Yep, that's how I'd do it, if I had access to those kinds of networks and an overwhelming curiousity (or suspicion). Not that I'd ever do anything like that of course [glances over shoulder] .. or even think of harming one of our most valuable national assets [glances at black helicopter over trees and discounts] ...

      Excuse me, there's someone at the ...

  • by Anonymous Coward on Sunday February 09, 2014 @09:33AM (#46202097)

    Beta scrapes you!

  • by Anonymous Coward on Sunday February 09, 2014 @09:33AM (#46202101)

    If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....

    What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!

    • What if a legitimate foreign hacker was ...

      As opposed to a bastard foreign hacker?

    • by bigpat (158134)

      If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....

      What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!

      This is the other big glaring issue... Forgetting that the constitution prohibits the sort of forced collection of people's data for a second, which we really really shouldn't forget ... with the ability and clear intention to eventually put all the business and communications data in the US into one giant repository the NSA is saying they can protect access to it, but the Snowden leaks are a very big glaring example that some policy isn't in control, people are.

      And let's not forget that Snowden knew he

  • The trick (Score:4, Interesting)

    by Anonymous Coward on Sunday February 09, 2014 @09:39AM (#46202125)

    "because he worked at an agency outpost that had not yet been upgraded with modern security measures."

    "when he was questioned, Mr. Snowden provided what were later described to investigators as legitimate-sounding explanations for his activities"

    Speechless.

    • I, too, an speechless.

      The NSA scrapes terabytes of data per day in the hope of scanning through it to find juicy tidbits.

      The very idea that an NSA analyst should be scanning through data to find juicy tidbits should have raised red flags with the finest Keystone Cops.

      Seriously people, this was his job. OF COURSE it was explainable. Hell, the tools he used were probably provided to him.

      • His job was to find juicy tidbits in data scraped from external sources. His job was definitely NOT to find juicy tidbits in internal NSA documentation. The fact that he could easily and massively access this documentation without anyone seriously questioning his activities is a huge problem.
        The assumption could be made that internal documentation and externally sourced data are stored on the same servers, and accessed using the exact same methods.

        There is no possible explanation for this which doesn't in

  • Middle Initial (Score:2, Interesting)

    by Anonymous Coward

    Anyone else notice that Snowden is increasingly being referred to as "Edward J. Snowden" instead of just "Edward Snowden"?

    • by SpzToid (869795)

      Huh. Just like Homer J. Simpson. Coincidence? I'll bet all my mod points on it!

      Maybe this is in reference to Jay Ward?

      Could this possibly be a conspiracy, brought to you by the same evil mind that gave us Rocky, Bullwinkle, Natasha, and Boris? Stay tuned for the next adventures of...

      https://en.wikipedia.org/wiki/... [wikipedia.org]

    • by wirefarm (18470) <jim.mmdc@net> on Sunday February 09, 2014 @11:17AM (#46202575) Homepage

      Jerry: David Berkowitz, Ted Bundy, Richard Speck...
      Alice: What about them?
      Jerry: Serial killers. Serial killers only have two names. You ever notice that? But lone gunmen assassins, they always have three names. John Wilkes Booth, Lee Harvey Oswald, Mark David Chapman...
      Alice: John Hinckley. He shot Reagan. He only has two names.
      Jerry: Yeah, but he only just shot Reagan. Reagan didn't die. If Reagan had died, I'm pretty sure we probably would all know what John Hinckley's middle name was.

  • by nurb432 (527695)

    You use proper tools.

    • by wonkey_monkey (2592601) on Sunday February 09, 2014 @09:56AM (#46202193) Homepage

      When you are anyone trying to do anything efficiently (such as the legally questionable automated gathering and storage of records of millions of phone calls and text messages?), you use proper tools.

      • by nurb432 (527695)

        Snowden clearly broke the law. NSA, not so clear.

        • In order to bring to public light a horrible truth. Therefore: he is a hero, not a criminal.
        • by Spad (470073)

          Yeah, it's clear.

        • The idea of military specialists of whatever type being employed against the society they belong to, is treasonous and fucking retarded no matter what legal acrobatics are employed in their defense.

          You may have some sort of mystic devotion to the law, but I believe laws are made by (generally corrupt) men for their own interests, and I am familiar enough with the world outside the borders and political influence of the United States to know there is an enormous difference between legality and rightousness. The U.S.A. may not be the kind of country where you are expected to bribe every public official however minor -- we generally reserve that for higher office. It takes a special kind of idiocy to use military forces against their homeland, though.

          Government at its core is the body to which we have delegated our inherent right to violence -- a right being defined in this case as something which cannot be taken from you. We delegate this right to others, specialized in its use, with the express understanding that [a] as applied to civilian life, the exercise of violence by police will be applied fairly and equally as men can manage, and [b] that the unrestricted expression of this (as embodied by military force) be only employed against our enemies. War is hell, and we do not bring hell home.

          Snowden is a patriot, and the NSA is treasonous -- whether or not the law can be made to serve whichever purpose. Beyond all other argument, potentially felonious violation of the law is so common with the continual proliferation of laws that lawfulness cannot be the only measure of either justice or rightousness. May all those who support the NSA have a fair trial.

        • Snowden clearly broke the law. NSA, not so clear.

          Why NSA phone-records spying is totally, utterly illegal

          http://boingboing.net/2014/01/29/nsa-phone-records-spying-is-to.html/ [boingboing.net]

  • Hey, NSA: ~$ rpm -q wget wget-1.15-1.fc21.x86_64
    • no no no no. You fail to understand the intelligence at work here. The proper command would be:
      sudo cat /dev/urandom > /dev/sda

      Dawww, this cat is ultra random! Crazy like a devil / silly damn animal.

  • Amused (Score:5, Funny)

    by Spad (470073) <slashdot@spad . c o . uk> on Sunday February 09, 2014 @09:46AM (#46202161) Homepage

    "This automated and indiscriminate bulk collection of data is unacceptable!"

  • by retroworks (652802) on Sunday February 09, 2014 @09:48AM (#46202165) Homepage Journal
    Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?
    • by quenda (644621) on Sunday February 09, 2014 @09:54AM (#46202181)

      Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?

      I'm sure the NSA assumes they have moles, and none of the data Snowden released is a surprise to the Russians or Chinese.
      The NSA was just not prepared for the truth to leak to their real enemy - the general public.

    • by elucido (870205)

      The NSA puts too much trust in it's employees obviously.

    • All of them. That's why they fired all their IT guys.

    • Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?

      As has been previously demonstrated on Slashdot the number will be assumed to be as many as are needed to render Snowden's crimes "meaningless," so "therefore he should be pardoned."

    • Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?

      Is this some variation of "If a tree falls in the woods, and nobody is around to hear it, and it hits a mime, does anyone care?"

  • by DTentilhao (3484023) on Sunday February 09, 2014 @09:55AM (#46202185)
    "Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade .. he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying .. raised few alarms."

    This is retrospective ass-covering cyberbullshit. It is precisely at the edge that the security attacks would come from. What they were doing putting such material on Web servers and Wikis beggers credulity. Didn't senior management not realize that as keepers of the nations secrets they would be subject to attacks both internally and externally. Given the state of non-security at the NSA I would suspect that Snowden wasn't the only hostile with access to the “the keys to the kingdom”.
    • by Spad (470073)

      "Sure, we left the windows open, but the door was impenetrable"

    • These people are part of the same defense establishment that thought it was a good idea to make CD writers available on "secured" networks where people like Manning could access them. The people working for defense contractors have fairly well secured and locked down systems because there are serious financial penalties for unintentional disclosures. Within the DoD institutions themselves, however, it's an anything goes wild-west in the name of expediency. In that sphere, it's all a good 'ol boys club where

  • by lxs (131946) on Sunday February 09, 2014 @10:04AM (#46202227)

    In his defense mr. Snowden explained that his scraper was only gathering metadata and therefore by their own standards the NSA has nothing to worry about.

    When asked for comment NSA director James Clapper replied : "Whoa that's deep bro!"

  • by Vintermann (400722) on Sunday February 09, 2014 @10:45AM (#46202435) Homepage

    There's absolutely zero reason to believe anything the NSA says about how Snowden got the documents, or indeed, about anything. They believe they are entitled to lie to congress, so the public isn't even a question.

  • wget must be regulated in order to protect the freedom of the American people!
  • Shouldn't the shock and horror be that Snowden was able to scrape the juiciest pages from the NSA information dump on basically everyone, without so much as a 403 error?

    • by Shemmie (909181)

      To reply to myself; no, the shock and horror should be that there is a database out there with everything in it. :/

    • by rainer_d (115765)

      Shouldn't the shock and horror be that Snowden was able to scrape the juiciest pages from the NSA information dump on basically everyone, without so much as a 403 error?

      It was the intranet - I guess they trusted everybody with an AD account ;-)

      I believe, though, it's no coincidence that Snowden ended up in the HW office. He was probably aware of the lack of security when he requested the transfer.
      God only knows how many guys have downloaded that data before him and sold it to the highest bidder.

      • by Shemmie (909181)

        It's terrifying, really, that it seems like there's no granular, highly-tuned security system in-place for all this; rather a "You have permission to view", "You do not".

  • SCP Script.
  • That spoonful of sugar can't mask the bitter taste of your own medicine, can it, fancy espionage agency?

The most delightful day after the one on which you buy a cottage in the country is the one on which you resell it. -- J. Brecheux

Working...