Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Spam The Internet Your Rights Online

A New Type Of Realtime Blocklist: The SURBL 219

Posted by timothy from the chicken-egg-spam dept.
Glamdrlng writes "The SURBL, or "Spam URI Realtime Blocklist", represents a nexus of RBL's and content filtering that may bring us one step closer to a spam magic bullet. While traditional RBL's perform a DNS lookup on the connecting mail server, SURBL's take this a step further by parsing the text of the email looking for URI's and doing a lookup on those web servers. They also prevent "joe jobs" by maintaining a whitelist of legitimate web servers whose domain names may show up in spam messages, e.g. EBay, Paypal, Microsoft, etc. The only requirement to implement the SURBL is a plugin on your MTA such as spamassassin that can parse the body of each email. While there is no MTA that directly supports SURBL's without a plugin, the author hints at one being in development."
This discussion has been archived. No new comments can be posted.

A New Type Of Realtime Blocklist: The SURBL More

A New Type Of Realtime Blocklist: The SURBL

Comments Filter:

  • Is this really a GOOD idea? (Score:5, Interesting)

    by beh ( 4759 ) * on Monday April 12, 2004 @05:03PM (#8841396)
    Blocking URLs is an "ACTIVE" measure - and one that opens very bad
    possibilities for abuse. While the While-List would protect against
    this it will protect the BIG players on the market - it can still
    wreak havoc on small/medium enterprises - e.g. a competitor of a
    (pretty much) 'niche' firm could get a spam out advertising the
    COMPETITOR in order to get HIM blocked...

    Or - the other way around - a company gets itself a whitelisting
    (via a "fake" joe-job on itself) and then continues spamming...

    Please stick to PASSIVE measures! They can't be abused...

    • It's a great idea (Score:5, Informative)

      by Rapid Home Offer ( 770408 ) * on Monday April 12, 2004 @05:08PM (#8841450) Homepage Journal
      Combine it with spamassassin, and you can whitelist emails from companies that you want to recieve email from. Heck, with spamassassin you can give it a very small weight, and adjust the results manually. Every bit of extra information helps, and just ignoring it because it is compiled by somebody else doesn't make sense to me.

    • Re:Is this really a GOOD idea? (Score:5, Insightful)

      by beh ( 4759 ) * on Monday April 12, 2004 @05:08PM (#8841458)
      (one minor thing I missed before:

      The advent of bayesian spamming brought spams that included whole paragraphs of random words - just so that your list would get more and more bloated...

      How long do you think it will take spammers to add dozens of valid - but in the context of the spam nonsensical - URLs just to fill up the black-list and make it useless?

      • The advent of bayesian spamming brought spams that included whole paragraphs of random words - just so that your list would get more and more bloated...

        It doesn't appear to have impacted baynesian scanners too adversely, however. I've been using a Baynesian scanner for ... I guess about a year now, and it's still working great (and the token list is big, but not unmanagable).

      • Unless you expect mail from these sites (and for me atleast, the vast majority do not send me mail); so who cares?

        Marking the odd legitimate mail as *not* spam should clue the filter in to those sites, and you only have to do this once per legitimate site.

      • Re:Is this really a GOOD idea? (Score:5, Interesting)

        by jelle ( 14827 ) on Monday April 12, 2004 @07:53PM (#8842949) Homepage
        Emails with paragraphs of random words are not very easy to distinguish from emails with paragraphs of actual language in nonspam emails. But emails with dozens of random links are better distinguishable from nonspam emails, so if the spammers start doing this, then you can filter out their spam even without having to check the SURBL by simply adding some points to the score of emails with a lot of links

        And if you use the auto-whitelist feature, then it won't increase the false-positive count, except for people who receive a lot of emails with lots of random links from lots of different people.

        Plus, the spam detection software may very well be capable of distinguishing between the decoys and the real spam-links by analyzing the context of the URI. At least that will be a lot easier than analyzing the grammar in an email and detecting the nonsensical paragraphs and the nonsensical/typo-ed words in spam.

        Sure, it's not the final battle, but it looks like a very promising improvement in the fight against spam.

        • Random words are -easy- to distinguish heuristically from actual text. It just takes a good bit of time.... Start with these simple rules:
          1. Alll sentences must contain at least one verb.
          2. All singular nouns other than proper nouns are almost always preceded by an article or a possessive noun/pronoun. There may be words in-between, but these will generally be adjectives (or adverbs modifying the adjectives).
          3. Prepositions are almost always followed by a noun within a handful of words, all of which are genera

          • Re:Is this really a GOOD idea? (Score:4, Interesting)

            by delstar dotstar ( 593915 ) on Tuesday April 13, 2004 @12:32AM (#8844690)
            Words that can be more than one part of speech are used fairly infrequently. Ten in a row is a pretty good giveaway.
            • that:
              1. adj (Not this one, that one)
              2. dem. pron. (Look at that )
              3. rel. pron. (birds that sing)
            • can:
              1. noun (a can of whoopass)
              2. verb (The boss is gonna can your ass)
              3. modal (I can swim)
            • one
              1. adj ( one fine morning)
              2. pron (the one that got away)
            • part
              1. noun ( part of speech)
              2. verb ( part the Red Sea)
              3. adj ( part man, part machine)
            • used:
              1. verb (I used a hammer on the kitten)
              2. adj (a used car)
            • ten
              1. adj ( ten fingers)
              2. pron ( ten in a row)
            • row
              1. noun (ten in a row )
              2. verb ( row your boat)
            • pretty
              1. adj (a pretty girl)
              2. adv (a pretty good giveaway)
            OK, that was a little snarky. Anyhoo, spammers can just extend the stream-of-random-words technique and create "sentences" that are syntactically kosher but semantically empty: Colorless green dreams sleep furiously. Hell, they don't even need to create sentences -- they can just pinch real, human-generated text from any old web site.
      • It's just another stage in the evolution of the economy.
        Pretty soon, the rich will employ servants to read their email, thus countering the whole offshoring thing.
        See now, it's all about finding that thread of silver in the crapheap...
      • I believe you can configure a Bayesian filter to only look at the first chunk of words. So if a spammer inserts random texts at the beginning, the commercial messages will be badly hampered.

        Similarly, if a spammer inserts a whole bunch of random URLs in the message, then the real URL will be diluted. (provided that the real URL is not standing out in any way, or if it does, the filter can identify it)

    • Re:Is this really a GOOD idea? (Score:5, Interesting)

      by acariquara ( 753971 ) on Monday April 12, 2004 @05:12PM (#8841509) Journal
      What ever happened to Bayesian Noise Reduction/Dobly algorythms? I was hoping for these to get more known and widespread...

      snip snip from their page

      Feb 24: We broke 99.984% today and caught up with CRM114 =). DSPAM is now around ten times more accurate than a human. According to a study by Bill Yerazunis (CRM114), a correspondence secretary is approximately 99.84% accurate at filtering spam. As of today, DSPAM has classified 3140 spams and 3457 nonspams in my mailbox with only 1 false accept and 1 false reject. The false accept was caused by a bug in the BNR code which was fixed, so depending on how you count it, I am getting either 99.968% or 99.984% accuracy. These are from real mailbox statistics, and not based on some 'test corpus' mail sent in. As spammers continue to try and evade filters, statistical filters such as DSPAM continue to adapt easily maintaining their high levels of accuracy.

      And no, I am not posting an URL. If you want to get to the page, google for "Dobly" (yes, that is the actual spelling) and go to the first page.

      • I had very high hopes for DSPAM. The installation was very easy, the CGI based web interface works pretty much out of the box and is very handy for the users.

        However, I had to abandon my DSPAM testing after a few weeks. The filter was *way* to slow to learn and in the process generated an incredible amount of false positives. With about 400 spams learned I still got around 29 false positives. And filtering accuracy according to its own built-in stats was less than 60%...

        Considereing that I get about 2500+

    • Re:Is this really a GOOD idea? (Score:4, Interesting)

      by DocSnyder ( 10755 ) on Monday April 12, 2004 @06:54PM (#8842431)
      Blocking URLs is an "ACTIVE" measure - and one that opens very bad possibilities for abuse.

      The SURBL is not blocking URLs but IPs where spamvertised URLs are hosted at. I've been doing this for about half a year, too - it's really effective in filtering spam as most spammers choose "bulletproof" ISPs whose netblocks are listed on SPEWS and SBL for that reason. Take Chinanet, for example - an email which is including a link hosted at Chinanet is almost always spam.

      I'd recommend not a single SURBL list but several ones, ranging from an in-progress DNSBL to a SPEWS-/SBL-like blacklist with the latter fed manually.

      If SURBL gains acceptance, spammers could choose bulletproof ISPs and have most of their spam emails filtered due to SURBL listings, or choose white-hat ISPs and don't get filtered but kicked.

    • Blocking URLs is an "ACTIVE" measure - and one that opens very bad possibilities for abuse.

      Absolutely, but that does not mean that a very restricted blacklist might not have a place.

      One of the frustrating things about the spam world is that every good idea gets grabbed by zealots who start to make a bigger nuisance of themselves than the spammers.

      It would be really good to have some mechanism that could used to protect people against phishing frauds. If some web site is pretending to be citybank or p

  • Time to dig out this old post. (Score:5, Funny)

    by Anonymous Coward on Monday April 12, 2004 @05:03PM (#8841403)
    This article advocates a

    (x) technical ( ) legislative ( ) market-based ( ) vigilante

    approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

    ( ) Spammers can easily use it to harvest email addresses
    ( ) Mailing lists and other legitimate email uses would be affected
    ( ) No one will be able to find the guy or collect the money
    ( ) It is defenseless against brute force attacks
    ( ) It will stop spam for two weeks and then we'll be stuck with it
    (x) Users of email will not put up with it
    ( ) Microsoft will not put up with it
    ( ) The police will not put up with it
    ( ) Requires too much cooperation from spammers
    ( ) Requires immediate total cooperation from everybody at once
    ( ) Many email users cannot afford to lose business or alienate potential employers
    ( ) Spammers don't care about invalid addresses in their lists
    ( ) Anyone could anonymously destroy anyone else's career or business

    Specifically, your plan fails to account for

    ( ) Laws expressly prohibiting it
    ( ) Lack of centrally controlling authority for email
    ( ) Open relays in foreign countries
    ( ) Ease of searching tiny alphanumeric address space of all email addresses
    ( ) Asshats
    ( ) Jurisdictional problems
    ( ) Unpopularity of weird new taxes
    ( ) Public reluctance to accept weird new forms of money
    ( ) Huge existing software investment in SMTP
    ( ) Susceptibility of protocols other than SMTP to attack
    ( ) Willingness of users to install OS patches received by email
    ( ) Armies of worm riddled broadband-connected Windows boxes
    (x) Eternal arms race involved in all filtering approaches
    ( ) Extreme profitability of spam
    ( ) Joe jobs and/or identity theft
    ( ) Technically illiterate politicians
    ( ) Extreme stupidity on the part of people who do business with spammers
    ( ) Dishonesty on the part of spammers themselves
    ( ) Bandwidth costs that are unaffected by client filtering
    ( ) Outlook

    and the following philosophical objections may also apply:

    ( ) Ideas similar to yours are easy to come up with, yet none have ever
    been shown practical
    ( ) Any scheme based on opt-out is unacceptable
    ( ) SMTP headers should not be the subject of legislation
    ( ) Blacklists suck
    (x) Whitelists suck
    ( ) We should be able to talk about Viagra without being censored
    ( ) Countermeasures should not involve wire fraud or credit card fraud
    ( ) Countermeasures should not involve sabotage of public networks
    ( ) Countermeasures must work if phased in gradually
    ( ) Sending email should be free
    ( ) Why should we have to trust you and your servers?
    ( ) Incompatiblity with open source or open source licenses
    ( ) Feel-good measures do nothing to solve the problem
    ( ) Temporary/one-time email addresses are cumbersome
    ( ) I don't want the government reading my email
    ( ) Killing them that way is not slow and painful enough

    Furthermore, this is what I think about you:

    (x) Sorry dude, but I don't think it would work.
    ( ) This is a stupid idea, and you're a stupid person for suggesting it.
    ( ) Nice try, assh0le! I'm going to find out where you live and burn your
    house down!

    • Re:Time to dig out this old post. (Score:5, Funny)

      by tds67 ( 670584 ) on Monday April 12, 2004 @05:26PM (#8841647)
      I think the preceding post:

      ( ) Was funny.
      ( ) Was informative.
      ( ) Was interesting.
      ( ) Was informative and funny.
      ( ) Was interestingly informative.
      (x) Was funny in an informative sort of way.
      ( ) Was rehash.
      ( ) Is itself spam.
      ( ) Is overrated.
      ( ) Gave me gas.

    • Re:Time to dig out this old post. (Score:5, Insightful)

      by interiot ( 50685 ) on Monday April 12, 2004 @05:33PM (#8841707) Homepage
      • (x) Users of email will not put up with it
      We'll see.
      • (x) Eternal arms race involved in all filtering approaches
      One of the few constants is that there will be way for money to get from the target back to the original spammer or seller. (well, it's possible something more complex is going on and that's not the real goal of spam, but at the least, it's something that's remained constant for years, which is notable in the world of spam). So "following the money" is really based on an acceptance of the above criticism, and a realization that the arms race can never get around the money stream.

      Filters may be lead to arms races, but does anyone NOT use them right now? There are few alternatives, namely things like making email non-anonymous / PKI, enacting large legal penalties along with huge international support, rejecting email from anyone you don't know, ....

      • (x) Whitelists suck
      Actually, it's a blacklist. Blacklists may suck, but it's possible they suck less than spam, and the proliferation of RBLs [rbls.org] kind of implies that.

      Sure, there might be a way to stop spam once and for all and then blacklists would be hated, but the very presence of a antispam-rejection-template implies that there won't be a magic bullet for a long time to come.

      • (x) Sorry dude, but I don't think it would work.
      The only way it CAN'T work is if money isn't the real goal of spammers, or if they make it hard enough to "follow the money" that other methods are easier/nicer.

    • Re:Time to dig out this old post. (Score:4, Interesting)

      by Carnildo ( 712617 ) on Monday April 12, 2004 @06:37PM (#8842283) Homepage Journal
      Additional problem:

      (x) The whitelist feature can be abused

      As anyone who's spent any amount of time reading Slashdot comments should know, there are open redirect URLs on a number of sites that would be whitelisted under this proposal. On Slashdot, they were used to hide references to goatse. In spam, they can be used to whitelist spam URLs.

  • The whitelist will always be limited (Score:5, Interesting)

    by Anonymous Coward on Monday April 12, 2004 @05:03PM (#8841407)
    There are millions of legitimate sites, and most of them aren't major sites like ebay, yahoo, etc. If I want to do a joe-job on an enemy small site, I can cause them a lot of pain by including their link. They'll have a dificult time someone wasn't spamming on their behalf.
    • Indeed... it creates a new kind of joe-job. One that not only inconveniences you for a couple of hours deleting the bounces, but possibly for weeks because your perfectly legitimate site is now on a blacklist.

      This is a non-solution, really.
  • The adjust the frequency of the phasers.

    I don't see this as the be-all, end-all for spam, but I do find it a very interesting and potentially very effective arrow for my spam-killer quiver.

    • I don't see this as the be-all, end-all for spam

      Either do I. In fact, the first thing that comes to mind is that the domains that start actually showing up in email will become random. This introduces a bit of additional cost to the spammer, but if that's the only way to survive, they'll do it.

      They could also use IPs, but this would become even more of a pain for them since it's harder to get IPs. If you start doing blocking for the random domain names by resolving the IP, and banning based on that, you'
      • Naw, all they got to do is get link results off google for random words for each email they send out, that way, each email is a little different, and nearly all the links are valid.
      • Another way to defeat this method would be to hack web servers, and put on files that redirect to the desired site. This has a lot of implications - legal and technical - but again gets into the same situation as before where blacklisting the site in the email would blacklist legitimate sites.

        You don't need to hack into them. I know that Yahoo has an open redirect URL -- it was used to disguise a link to goatse a while back -- and I suspect that most other major web sites have similar URLs.
  • three guesses and the first two don't count...

  • SURBL? (Score:5, Funny)

    by Mateito ( 746185 ) on Monday April 12, 2004 @05:06PM (#8841436) Homepage
    I mean, really, who comes up with the names for these things?

    Show me one self-respecting spammer who's going to quake in their boots at the threat of being hit with a "SURBL".

    ("Oh no.. please.. not the SURBL. Don't SURBL me.. Its too much... no.. No.. NOOOOOO!)

    Why not just call it a "NERF" and be done with it?

    I propose we come up with Spam deterrents with names like "Knuckle Duster", "Jagged Bottle", "Bloodied Crowbar" and "Bubba the Love Truncheon".

    • Re:SURBL? (Score:4, Funny)

      by Anonymous Coward on Monday April 12, 2004 @05:32PM (#8841701)
      I propose we come up with Spam deterrents with names

      Personally, I like BASTARD:

      Bad Ass Spam Threat And Reduction Deterrant

  • A plugin? (Score:4, Interesting)

    by Pranjal ( 624521 ) on Monday April 12, 2004 @05:09PM (#8841471)
    The only requirement to implement the SURBL is a plugin on your MTA such as spamassassin that can parse the body of each email.
    Anything which requires extra software on the MTA or client side is not a simple requirement as it cannotn be implemented universally. This is doomed to fail.

    • Re:A plugin? (Score:4, Funny)

      by Mateito ( 746185 ) on Monday April 12, 2004 @05:40PM (#8841780) Homepage
      > extra software on the ... client side.. not a simple requirement as
      > it cannot be implemented universally

      Bollocks. Send it to random users as an encrypted zip file with the key in an attached jpeg and a title like "returned mail" from a user called "hg477d762@hotmail.com", and enough people will install it to make it effective.

      Never underestimate the stupidity of end users.

    • Re:A plugin? (Score:3, Insightful)

      by interiot ( 50685 )
      So that explains why RBLs are so unpopular, right?

      ANY technical solution is going to require extra work on the client side, so rejecting this outright is kind of rediculous unless you're advocating a purely legal, market-based, or vigilante solution.

      Spam is getting to be such a problem that techies are setting up things like SpamAssassin for themselves and friends, and major ISPs are using RBLs. So this isn't really a problem.

    • Re:A plugin? (Score:3, Insightful)

      by MrChuck ( 14227 )
      You don't want MTAs to stop being MTAs. They really SHOULDN'T be looking at message bodies. And then making complex decisions (is http://1593985/ a decimal version of a member of the list?) or doing regexs.

      If you want bloat and dysfunction like this, look at Exchange or Notes or Gropewise. It's a GUI, it's a calendar, it's a database, it's an MTA! well, it doesn't scale and it tastes like floorwax.

      This is why sendmail developed the MILTER interface. Firewall-1 had a proprietary scanning interface (e

    • Why does it have to be implemented universally? If just 10,000 people use it and I do too, maybe I'll have less spam.

  • Whitelist maintenance? (Score:5, Interesting)

    by tepples ( 727027 ) * <tepplesNO@SPAMgmail.com> on Monday April 12, 2004 @05:10PM (#8841479) Homepage Journal

    From the article:

    This is a democratic effect, improved by manual de-selection of legitimate domains by SpamCop users when they submit their reports. More reports means more votes that a given site is indeed spam.

    Though the article's author feels that "most SC users probably make an effort to uncheck legitimate domains to prevent false reporting," I have read reports that some mail server admins claim that SpamCop's users are rather likely to mistakenly report ham as spam [emailaddresses.com]. So the domain whitelist [freeapp.net] becomes important, but what practices have the SURBL administrators put in place to prevent corruption with respect to sites reported to whitelist at surbl dot org?

  • Present problem. (Score:3, Insightful)

    by FreeLinux ( 555387 ) on Monday April 12, 2004 @05:12PM (#8841501)
    Presently the only problem with this is that there are no plug-ins for the MTAs themselves yet. The plug-in is for spamassassin. That means that the message has to be transfered and passed onto Spamassassin before it can be dropped or tagged whereas, the other RBLs allow you to drop the connection before the message is transfered. This problem will be solved once there are plug-ins for the MTAs themselves.

    But, I have to ask, why aren't existing RBLs like Spamhaus effective. They should be far more effective than the ~40% that I am experiencing.

    • >That means that the message has to be transfered
      >and passed onto Spamassassin before it can be
      >dropped or tagged whereas, the other RBLs allow you
      >to drop the connection before the message is transfered.

      if the RBL in question is based on parsing the text of the message, then I'm pretty sure you have to let the message be transfered anyway... So, not really a problem in this case.
    • But, I have to ask, why aren't existing RBLs like Spamhaus effective. They should be far more effective than the ~40% that I am experiencing.

      The answer is simple - many spammers are now querying the RBLs themselves and using the results to pick which proxies to send their spam through.

      If you run an RBL, I think that with some analysis you could determine when a spammer is querying your RBL by their traffic pattern - for instance, if a given source is consistently the first to query for a given IP addres

    • the other RBLs allow you to drop the connection before the message is transfered.
      How can URIs in the body of the message be checked before the message is transmitted? It's still going to use up bandwith, either way.

    • Re:Present problem. (Score:5, Insightful)

      by Phroggy ( 441 ) <slashdot3@@@phroggy...com> on Monday April 12, 2004 @09:18PM (#8843496) Homepage
      Presently the only problem with this is that there are no plug-ins for the MTAs themselves yet. The plug-in is for spamassassin. That means that the message has to be transfered and passed onto Spamassassin before it can be dropped or tagged whereas, the other RBLs allow you to drop the connection before the message is transfered. This problem will be solved once there are plug-ins for the MTAs themselves.

      Sorry, but that's not because it's a SpamAssassin plugin vs an MTA plugin. That's because the SMTP protocol doesn't allow for what you describe.

      Let's say I'm an MTA. When you connect to me, the first thing you do is introduce yourself, then tell me the envelope sender and envelope recipient of the message you're about to send, then give me the full message including headers and body. My options for blocking the message are:
      1. Before you even connect, your IP could be blocked at the firewall level, so I'd never see you.
      2. After you connect, before you introduce yourself, I have your IP address, and can check it against a blacklist and/or whitelist, and give you an error and disconnect if I don't like what I find. I can also do reverse and forward DNS queries on your IP to make sure they agree.
      3. After you introduce yourself, I can compare your greeting against your reverse DNS, since that's how you should be introducing yourself. I can give you an error if I don't like it.
      4. After you give me the envelope recipient, I can make sure that domain exists, etc. (Side note: Verisign wants to break this; ICANN is currently not letting them.)
      5. After you give me the envelope recipient, I can make sure that e-mail address is OK - if it's my domain name and the username is somebody I know I'll accept it, or if it's a valid domain name somewhere else and your IP is on my LAN I'll relay it. Otherwise I can give you an error.
      6. If we've gotten this far, I must now accept the entire message, including headers and body. If there's something in the headers I don't like, too bad! If there's something in the body I don't like, too bad! I have to let you send the whole message.
      7. After I've accepted the message, if there's a problem, I can generate a bounce message to send back to you, assuming the e-mail address you gave me actually works. If that fails, I'll send an e-mail to my postmaster explaining what happened. Or if that's too annoying, I could just delete your message and not tell anyone.

      Existing RBLs work at step 2. Filtering based on message content can't happen until step 7. You could build it into the MTA, but MTAs are complex enough as it is; using something else (SpamAssassin, Procmail, whatever) is a better idea.

  • Then what happens when .... (Score:5, Interesting)

    by Anonymous Coward on Monday April 12, 2004 @05:12PM (#8841505)
    Spammers could then post their web sites as search URL's on Google, MSN, etc.. If you block those URL's then lots of people would complain that they can't send Google entries. Even if you solved that, then what happens with sites like tinyurl.com? If you block them then you have liability and legal issues to think about. No doubt the spammers will script up a number of ways to cloak the marketeers site urls.

  • Works for me (Score:4, Interesting)

    by Frisky070802 ( 591229 ) * on Monday April 12, 2004 @05:12PM (#8841507) Journal
    I am continually adding certain domain names to my spam filter, if found in text. I'd love it for this tool to do it for me, as long as I can trust the low false positive rate.

  • DOSes and things outside of ones control (Score:5, Interesting)

    by Corvar ( 20297 ) * <corvar@nospAM.theonering.net> on Monday April 12, 2004 @05:12PM (#8841514) Homepage
    This type of system is very abusable.

    I know I have gotten spam reports from places like spam cop because people have included the URL of my website in their spam. My site had nothing to do with the spam other than the spammer was using an article on the site to back up his point of view.

    This type of system could very easily be abused to blackhole many mailing lists.

  • Spam is unavoidable (Score:2, Interesting)

    by Klatoo55 ( 726789 )
    We can't ever have a workable spam filter because of the adaptability of spam. However much you try, the spammers will come up with a way to circumvent your block. How long do you think that it would take for the spammers to figure out how to send emails that the whitelist software would mistake for legit? Nothing short of a trained monkey going through your inbox will sort this out effectively.

    • Re:Spam is unavoidable (Score:5, Insightful)

      by rw2 ( 17419 ) on Monday April 12, 2004 @05:37PM (#8841753) Homepage
      We can't ever have a workable spam filter because of the adaptability of spam.

      This is because the solutions of the day focus on content instead of anonymity.

      I've said it before, I'll probably say it again, get rid of unauthenticated email and the spam problem becomes a thousand times easier to fight. SPF and various RMX solutions exist in design today. If people want the spam problem to go away, that can be done today. Unfortunately people would rather piss and moan and call for legislation or perfect solutions than deal with these good ones today.

      In the case of spam the perfect is the enemy of the good enough. We should stop spam today.

  • sendmail internal RBL (Score:5, Informative)

    by mabu ( 178417 ) on Monday April 12, 2004 @05:13PM (#8841524)
    A good way to start if you're running your own mailserver is to use an internal IP-based blacklist such as the one found here [geocities.com]. It's incomplete due to Geocities limitations but send e-mail to that account and the guy running it will send you the whole file. It's a list that he's been compiling now for more than a year of IP blocks, mostly class Bs, that have virtually no useful SMTP traffic and should be completely cut off. This generally consists of the vast majority of Chinese, Korean and Brazillian DULs.

    We've been able to effectively stop about 50% of the spam using these lists and save resources and bandwidth. What's left is to start RBL'ing the domestic DUL IP space (Comcast, SWBell, Bellsouth, etc.) on a class B-level until the ISPs start cracking down on their rogue users.

  • so-called "remove me" links (Score:4, Interesting)

    by imroy ( 755 ) <imroykun@gmail.com> on Monday April 12, 2004 @05:13PM (#8841526) Homepage Journal
    I've been playing with a honeypot email account for the last couple of months. Those "remove me from your list" links sure are a good way to get more spam (Spammers are lying scum). I hope this SURBL suggestion doesn't get implemented at the ISP level. Then I wouldn't be able to go the spammers site (carefully editing the URL as needed, and with Mozilla) and sign up my honeypot account for more penis enlargement spam!

  • My proposed solution to spam (Score:3, Interesting)

    by GillBates0 ( 664202 ) on Monday April 12, 2004 @05:18PM (#8841567) Homepage Journal
    I don't know if anybody has tried this yet....and if not, why not:

    My suggestion is to present the user with those images containing a word (like the one used by Yahoo! etc during registration) everytime the user needs to send a mail (before clicking Send). This is a reasonably difficult Turing-type tests which could weed out a majority of automated scripts/spambots.

    An immediate problem with this scheme that I see is that for the words to be sufficiently random and crack-proof, they would have to be served in real-time to the mail program, and could need tweaks in current mail programs. A static list coded into the program might be too easy to break. This isn't too impractical, since an Internet connection is assumed during most email transactions.

    Another problem, ofcourse is that it will not work with text-based mailers like PINE, but as long as it weeds out all the spam sent from all the freebie mail accounts we could see an improvement.

    Comments/Suggestions?

    • Uh... No. (Score:3, Insightful)

      by FreeLinux ( 555387 )
      Sounds like a great idea especially for home users or some such but, as soon as you look at the bigger picture things start to break down. First of all, what about legitimate mailinglists? Some of them have hundreds of thousands of addresses. You want the administrator to have to go through and click a web page for each and every address on the list? Never gonna happen.

      What about corporate use? Many legitimate emails go to a dozen recipients almost like a mailinglist. Think of the lost productivity with th
      • I would suggest that this solution be implemented mainly by freebie "webmail" accounts rather than in the Corporate or other "trusted" environments. Most mailing lists would be sent out using academic or corporate accounts anyway - definitely not freebie accounts - atleast that's my assumption, though it's not too outstretched.
        • Most mailing lists would be sent out using academic or corporate accounts anyway - definitely not freebie accounts - atleast that's my assumption, though it's not too outstretched.

          These days, most spam isn't sent by freebie accounts, either. Most spam is sent from computers that have been hijacked using the latest batch of e-mail viruses.
    • See the above list? Your post fits into:

      (x) Requires immediate total cooperation from everybody at once

      Also, accessibility, custom SMTP clients, yadda yadda yadda... but you've already realized your mistake so I'll stop now.
    • This scheme doesn't work because:

      1. Spam isn't primarily coming from legitimate SMTP relays like Yahoo or Hotmail

      2. Ultimately to make such a system work, the mail would end up having to be flagged as "approved" by completing the process you suggest, which basically turns the scheme into a trusted-computing system (aka "whitelist"), and if you're going to go that route, you might as well call a spade a spade.

      And since we're calling spades a spades, the way to do it is to require all SMTP servers to have

      • Re:My proposed solution to spam (Score:5, Interesting)

        by hacker ( 14635 ) <hacker@gnu-designs.com> on Monday April 12, 2004 @05:58PM (#8841944)
        1. Spam isn't primarily coming from legitimate SMTP relays like Yahoo or Hotmail
        You're kidding [gnu-designs.com], right [gnu-designs.com]?

        At least 80% of our incoming spam, brute-force attacks, and other SMTP violations are coming from behind legitimate hosts like AOL, Verizon, Blueyonder, RoadRunner, and so on. Not forged IPs that pretend to be those hosts, but actual IPs that return to those MXs.

        Look at today's list of brute-force attacks so far [gnu-designs.com].. (as of Mon Apr 12 17:55:53 EDT 2004)

        Every single one of these lists gets collected and reported, per day, per provider, and to date, not a single one of them has done anything to stop the abuse. In fact, it keeps increasing every day. The more we block, the faster they come at us.

        • At least 80% of our incoming spam, brute-force attacks, and other SMTP violations are coming from behind legitimate hosts like AOL, Verizon, Blueyonder, RoadRunner, and so on. Not forged IPs that pretend to be those hosts, but actual IPs that return to those MXs.

          You just hit the problem on the head. Those aren't Hosts, they are companies or providers. A host is a single machine or "hostname"(multi-homed hosts and multi-ip load balancing notwithstanding). mail.example.com is a host. Most of your spam is pr
        • omr-blah is actually AOL's e-mail bouncing complex. That is why all the messages show up as having a null return path. No real spam filtering on those, as the messages aren't considered to have really been "sent" by AOL, someone else hit AOL with mail from fake addresses, to addresses they knew didn't exist.
    • My suggestion is to present the user with those images containing a word [...] everytime the user needs to send a mail (before clicking Send).
      [...]
      as long as it weeds out all the spam sent from all the freebie mail accounts we could see an improvement

      That wouldn't help one bit. Spammers may forge a freemailer's address, but in reality they use either open relays which are run by admins way too lazy to implement ANY contermeasures against spam or (as it's more and more common) they're using worms to infe
    • You are advocating a

      (x) technical ( ) legislative ( ) market-based ( ) vigilante

      approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

      (x) Users of email will not put up with it

      and the following philosophical objections may also apply:

      (x) Ideas similar to yours are easy to come up with, yet none have ever
      been sh
    • A) It's stupid.
      B) It's already been broken, by offering free porn to those who break the code
      C) It's stupid.
      D) What about spammers that run their own mail servers? Which is to say, nearly all spammers.
      E) It's stupid.
      F) It will break every mailing list in existence, or mailing lists will be set up by spammers.
      G) It's stupid.

      Did I mention, it's stupid?
    • check out the anonymous e-mail through www.icarusindie.com

      Instead of a picture I just present a riddle or other question.

      A human can search Google for the answer in order to be able to send their anonymous message. A program would need to be written and trained to be able to do that specifically for my web-site. I'm confident only someone with an academic interest in such a challenge would do it. And so far it hasn't been abused.

      I use the same type of challenge but render the text to an image and add

  • Yet Another Stupid Spam Idea (YASSI) (Score:4, Informative)

    by mabu ( 178417 ) on Monday April 12, 2004 @05:24PM (#8841616)
    Let's coin a new term: YASSI for yet another stupid spam-related idea.

    This boneheaded scheme falls into the same category as all content-based filtering systems: It doesn't address the most henous crime on the part of spammers, which is the consumption of bandwidth and network resources. And like other client-side/content-based filtering systems, the system will work about 12 minutes before the spammers figure out a way around it and then your system doesn't work. And of course, you'll have to constantly update it in order to make in effective, which means you have yet another piece of software that requires routine updating, slows down the mail service, your computer and everything in between. And after all that, you'll still get spam.

    The main reason spam is prevalent is because SPAMMERS STEAL BANDWIDTH WITHOUT PAYING FOR IT. When you force them to operate from a single location, then they have to act ethically and then they have to pay premium money to spam, and then they go out of business because it's only economical when they steal resources.

    You don't have content-based filtering on other primary methods of communication. It's a federal crime to go through mail; (at least before Patriot) you needed a court order to tap phones. E-mail should be an equally sacred communication medium that shouldn't be subject to "strip searches" before it hits your inbox. And this whole boneheaded scheme will NEVER stop spam in the first place, so let's stop pursuing these efforts.

    RBLs are most effective right now. The worm invasion is evidence of that, as spammers are finding less IP space to operate from so they're engaging in more aggressive tactics to take over peoples' machines, which, hopefully sooner-or-later, will land these sleazebags in jail.

    • You don't have content-based filtering on other primary methods of communication. It's a federal crime to go through mail; (at least before Patriot) you needed a court order to tap phones. E-mail should be an equally sacred communication medium that shouldn't be subject to "strip searches" before it hits your inbox.

      Ummmm, the hell? It's perfectly legal to go through mail. My own mail, naturally. And it's legal to tell someone else (say, a secretary) to go through your and filter it. Ditto phone calls

    • The main reason spam is prevalent is because SPAMMERS STEAL BANDWIDTH WITHOUT PAYING FOR IT. When you force them to operate from a single location, then they have to act ethically and then they have to pay premium money to spam, and then they go out of business because it's only economical when they steal resources

      So the RBL's keep them running from IP to IP, or serving spam off of compromised machines (Gotta love the spamhaus XBL). Personally, I view this as a progression of RBL's: make it so that, no

  • No system that uses the content of an email... (Score:4, Informative)

    by exp(pi*sqrt(163)) ( 613870 ) on Monday April 12, 2004 @05:26PM (#8841644) Journal
    ...to detect spam can be effective for long. Ultimately the mere fact that it's a system means that intelligent spammers can use the characteristics of the system to engineer a description of mail that isn't identified as spam and hence craft their own spam to fail to fit that description. There's probably some variant of Gödel's Theorem that makes this formal.

    There is already a cure for spam - give everyone unlimited email addresses, give out different addresses to different recipients, and delete any email that receives spam (along with possibly sending an email of complaint to whoever you originally have that particular address to). The whole thing could easily be built into mail clients and supported by mail providers. It works fine for me. It costs me $35 to buy my own domain and a one off payment of about $30 to zoneedit [zoneedit.com] to set up the mail forwarding. It works so well, and has worked for the least 3 or 4 years, that I almost suspect that there is some kind of conspiracy to overlook this method in order to promote other dubious methods.

    • Re:No system that uses the content of an email... (Score:4, Insightful)

      by interiot ( 50685 ) on Monday April 12, 2004 @05:56PM (#8841930) Homepage
      The unlimited-email-addresses basically comes down to requiring a password before someone can send you email (eg. if everyone accepted *@their.domain.com, spammers would just pick some random characters for the left side. So you have to have some sort of checksum or hash built into the characters, but if everyone uses the same algorithm, spammers would be able to generate their own random list again. So the only way to make it work universally is to salt the hash with something like a private key).

      If people have to get passwords from you before they can contact you, then... what do you do if you're an open source author... or if an ex from college wants to hook up again and googles you, and finds your website, but STILL can't contact you... or you want to sign up for match.com so that random women can email you.

      • Good questions.
        • what do you do if you're an open source author... I am and don't get spam this way. If I did I'd merely change the email address associated with the project. In the worst case scenario that someone needs to track me down from an ancient piece of code with an old email address they can use the method below...
        • or if an ex from college wants to hook up again and googles you, and finds your website, but STILL can't contact you... I have an email address on my website. It's in a form that can'
        • Could work I guess. There ARE a lot of disposable email address [google.com] services out there, and they seem to be all the rage [clickz.com] lately.

          It does put some burden on the sender as well though. These are the options I see:

          • The sender manages to keep your address confidential (meaning no catching outlook viruses, no cc'ing you along with 20 of their friends, no using it to send you stupid postcards). Not bloody likely for very many people.
          • All spams are responded to with a URL that users can use to get a new address
    • But if there are invariants associated with spam, then systems will be at least partially effective.

      Currently, spammers can create new spam relays only so fast.

      Currently, spammers want to receive money via credit card over the internet.

      Currently, it's hard enough to effectively spam that there aren't tens of thousands who are actively doing it, so blacklisting certain credit card vendor IDs could work.

      Currently, spammers want to make it harder to "follow the money" so they use crazy javascript stuff

    • It costs me $35 to buy my own domain and a one off payment of about $30 to zoneedit to set up the mail forwarding.

      Use a registrar like directNIC [directnic.com] that has $15 domains and free email forwarding.

      But note that you don't have to have your own domain to use that method. MTAs like qmail offer extension addresses (user-*@example.com). Also check out spamgourmet [spamgourmet.com] for a more advanced approach.
    • People want their old email addresses to continue to work, the way a physical address does.

  • Unfortunately (Score:3, Informative)

    by jeffster10304 ( 770716 ) on Monday April 12, 2004 @05:28PM (#8841661)
    Unfortunately I really don't see anything new or original in this idea. Until they start prosecuting these morons, we're not going to have a viable solution. By morons I mean the people that are BUYING this crap, rather than sending it ;)
  • Simple: have Sendmail, Postfix, etc, all stick in a sleep() call before they spit out the mail acknowledgement! No spammer could possibly hope to send out a million e-mails when they can't open that many sockets on their machine.

  • Counter-attacks are bad-- read this summary (Score:5, Informative)

    by joelparker ( 586428 ) <joel@school.net> on Monday April 12, 2004 @05:42PM (#8841795) Homepage
    Counter-attacks are bad--
    check this summary of spam methods.

    http://netextend.com/junkmail [netextend.com]

    ........

    Overview

    • What is Junk Mail?
    • Why Send Junk Mail?
    • How Bad is the Junk Mail Problem?
    • What is Needed?

    Solutions

    • Blacklists
    • Whitelists
    • Greylists
    • Adaptive Filters
    • Challenge-Response
    • Counter-Attacks
    • Tagging
    • Fake Honeypots, Tarpits, Spamholes
    • Sender Policy Framework (SPF)
    • Personal Digital Signatures
    • Internet Mail 2000 (IM2000)

    Conclusion

  • joe jobs (Score:2, Insightful)

    by selfabuse ( 681350 )
    One thing I've noticed a lot recently, is spammers including a big list of domains that have nothing to do with them in the text of thier junk mail. In the past 2 weeks, i've probally got about 10 spamcop reports for my customers, and in every case, my customer has had nothing to do with the junk mail, except for being listed in a list of about 15 URLs, that are not associated with the spammer. This system here says it has a whitelist for paces like ebay, paypal etc, but what about smaller people. They'd ge
  • Having just configured my email server with a ROBUST array of RBLs, I think this is a fantastic idea. I've been using the body_checks feature in postfix and manually adding violating URI's to create my own blacklist for several months now. I would love to benefit from a shared list. I don't care much for the white-list feature, that seems to me to create a backdoor for the spammer. Combined, RBL and private blacklisting (IPs and Addys) allow me to block 6000 plus spam A DAY. That's for a mere 150 plus user
    • P.s. We have been successfully blocking encrypted URI's for months now. It's an easy rule to set up...

      Care to tell us what the setup is?

      • Sure! These are things you can do if you are using Postfix as your MTA.

        In your main.cf file include this at the bottom

        body_checks = regexp:/etc/postfix/spammerbodies

        Learn more here about main.cf and other cool spam protections here:
        http://www.afp548.com/Articles/mail/spam2.html [afp548.com] including a really great RBL configuration.

        Create a spammerbodies file and include this line
        # various encoded URL formats. if they're trying to disguise the URL then they're up to no good /(ftp|https?):\/\/([^\/]*@)
  • How is this anything other than just another test for spamassassin? The SURBL lists all seem to be based on OTHER RBLs. While I don't deny that it's a great tool for filtering, it has the same problems as any other filter:
    1. The spam still has to be received in order to filter it. So bandwith, storage, and processing costs are still levied onto the receiver.
    2. It does nothing to stop the spam being SENT. It just hides it from the user.In other words, automated "JHD" ("Just Hit Delete").
    3. It's potentially easily
    • It might have a use for spoting mail that gets through SBL/XBL/DUHL/SPEWS/&c.lists, but worthless as a first line of defense.

      That's right. It's not intended to be the first line of defense. It's intended to help spot spam that gets through your IP-based blacklists.

      And for that purpose, it looks very promising.
      • Then I agree, it does look promising, and it is a very good idea. If I understand correctly how it works, it provides some more blocking for spam support services as well. If a spammers' site is on shared hosting, and the plugin resolves the host names, then emails containing any url that resolves to that host will be blocked too. Personally, I don't see that as a problem, but some of the other whiners will. AFAIC, that's what they get for giving their money to a spammy provider.

  • First, there is no spam magic bullet. There never will be.

    This is very similar to what SpamPal [spampal.org] along with the URLBody [noctua.org.uk] plugin does. (Client-side, Windows-only, also not a magic bullet.) The only difference being that this checks URLs against existing DNSBLs, and this is a new DNSBL specifically for this purpose.

  • Already in use by MT-Blacklist (Score:5, Informative)

    by santiago ( 42242 ) on Monday April 12, 2004 @06:25PM (#8842188)
    This exact method is the basis of the MT-Blacklist [jayallen.org] comment-spam prevention system for Movable Type-based blogs. It works wonderfully, as it identifies spam on the basis of the one feature it must have to be successful--a link back to the spammer's site.

  • This isn't new (Score:4, Informative)

    by KalvinB ( 205500 ) on Monday April 12, 2004 @07:32PM (#8842766) Homepage
    I've been doing this with my e-mail server (link in sig) for at least a year now. You can view the entire list of domains I filter at the Indie-Mail site. I even have a right up describing the why and how of this system on www.icarusindie.com called "An Analysis of Spam." And this is probably the 20th time I'm hawked this method up on Slashdot.

    The process is mostly automated but when it comes to blacklisting a domain, it's manual. You cannot automate it fully because legitimate domains make it into spams. yahoo, msn and w3c.org are the most common. Even without it being intentional on the spammers part. The automated part rips through e-mail logs pulling out who it's to, from and the subject and then all the urls. I can then clear out any entries that are going to account that aren't mine. And from there I go through and make sure the ones that do get added are actually spam domains.

    A computer can't really tell the difference between a spam domain and a legitimate domain. Humans can.

    Spam domains are blatently labeled like "medsforyou.com" contain random letters and numbers or have the spams images linked in the root. 8000hosting.com/ad.jpg is a big giant clue that this is a spam domain. I've seen links with 6 or 7 subdomains tacked on. I manually remove all the subdomain garbage and block the main one.

    The link ripper not only yanks out the root domain (and any subdomains) but also the exact URL of what it was pointing to.

    The main problem with anti-spam tools is that they rely on computers to find patterns. Spammers are not computers. They're idiots but not computers. And you can't get around the fact you need humans to be effective without causing colateral damage. Spammers do not always use computer identifiable patterns.

    The other "problem" with this method is that it only says 50% of the bandwidth cost at max since the server has to recieve the message for parsing. So it's only good for people offering e-mail services like myself who can't risk being over zealous in fighting spam which could result in losing other people's e-mail.

    ISPs are forced for the sake of bandwidth to use IP blacklisting while this sort of method would work as a secondary filter.

    Again, there is no silver bullet. You cannot just rely on one form of spam protection if your goal is irradication. This method is just the least error prone when done properly. IP blacklisting can be like nuking a small villiage to kill a fly. This is a highly focused and reasonably sized flyswatter that may occasionally flak off some paint if swung too hard.

    And never underestimate the number of domains spammers own. I get a dozen or so new domains to filter out ever few days. I may get spam but at least it's costing them real money to get it to me.

    Ben

  • Could be good, could be bad. (Score:5, Insightful)

    by autopr0n ( 534291 ) on Monday April 12, 2004 @11:03PM (#8844155) Homepage Journal
    I see one major problem with this, which is that Spammers might now be able to cause problems for legitimate websites simply by including their URL in the a Spam.

    I'm a little sensitive to this since a spammer is actually Jo-jobbing one of my domains (not autopr0n), and I get hundreds of "user unknown" messages every day, along with a handful of messages telling me "my" email was blocked. It's really irritating.

    But, if it's done right, it could work out pretty well. In fact, this would actually be effective against a lot of the current Spam out there, and kill Spam with off-site images.

    Anyway, let me throw one countermeasure out there. Suppose spammers start including commonly mailed URLs (such as those on hotornot, yahoo, etc) in their spams in order to decrease the usefulness of these things. If this thing gets popular, expect to see a lot of Spam include a lot of random URLs the way they now include lots of random words. You'll also start to see things like "Javascript decryption" and other techniques to prevent machines from figuring out which, exactly, URL it is that is being advertised, rather then random noise.

Slashdot Top Deals

A morsel of genuine history is a thing so rare as to be always valuable. -- Thomas Jefferson

Close