Forgot your password?
typodupeerror
Spam The Internet Your Rights Online IT

Maintaining a Publicly Available Blacklist - Mechanisms and Principles 89

Posted by samzenpus
from the it's-not-me-it's-you dept.
badger.foo writes "When you publicly assert that somebody sent spam, you need to ensure that your data is accurate. Your process needs to be simple and verifiable, and to compensate for any errors, you want your process to be transparent to the public with clear points of contact and line of responsibility. Here are some pointers from the operator of the bsdly.net greytrap-based blacklist."
This discussion has been archived. No new comments can be posted.

Maintaining a Publicly Available Blacklist - Mechanisms and Principles

Comments Filter:
  • by Gothmolly (148874) on Sunday April 14, 2013 @05:24PM (#43447531)

    Use greylisting as a first defense - easily configurable in postfix, and it reduces the amount of spam dramatically. This relies on the behavior of the sender, rather than someone else's opinion of them.

    • by Anonymous Coward

      Use greylisting as a first defense - easily configurable in postfix, and it reduces the amount of spam dramatically. This relies on the behavior of the sender, rather than someone else's opinion of them.

      Problem is, reducing 90% of spam isn't good enough, you need to remove 99%+. Even if just a handful get through, it is and underestimated productivity killer, and.. and increasing amount of spam also contain malware. I have been on the wrong end of a blacklist because we had an open relay, so know how frustrating it can be. We are currently using a service called Norman SecureTide that do block 99%+. Not sure which blacklists they are using, if any (might only be scanning and their own reputation) but damn

      • by 1s44c (552956) on Sunday April 14, 2013 @05:34PM (#43447577)

        If you ran an open relay you were on the right end of a blacklisting.

        • by PNutts (199112)

          If you ran an open relay you were on the right end of a blacklisting.

          I wish I had mod points for you.

        • If you ran an open relay you were on the right end of a blacklisting.

          Right, because although we're after the content of these e-mails, guilt by association is a perfectly valid technique for eliminating spam. Just like bombing a city to get rid of the army in it is totally okay... nevermind the civilians.

          • by Anonymous Coward

            If you ran an open relay you were on the right end of a blacklisting.

            Right, because although we're after the content of these e-mails, guilt by association is a perfectly valid technique for eliminating spam. Just like bombing a city to get rid of the army in it is totally okay... nevermind the civilians.

            Your misunderstand (probably intentionally, this is Slashdot after all).

            Let's say Adam makes a list of gun stores that verifiably have provided guns without background checks, in a state where such checks are voluntary, to people who use them to commit crimes.

            Bob is placed on the list. Bob complains: "I should not be on the list because I did not know the guns I provided would be used for crimes".

            Adam responds appropriately: "But that's not what I am making a list of. It doesn't matter whether you knew. To

            • by cstacy (534252)

              Your misunderstand (probably intentionally, this is Slashdot after all).

              Let's say Adam makes a list of gun stores that verifiably have provided guns without background checks, in a state where such checks are voluntary, to people who use them to commit crimes.

              Just so you know, there are no stores (in the U.S.) where guns can be purchased without a background check; it's not voluntary..

          • by Luckyo (1726890)

            Not sure if serious. I don't think even the most hardcore spammers who rage on spamhaus ever tried to compare blacklisting open relays to mass murder.

            Terrorism, yes. Mass murder, not even they have sank that low. Congratulations, you sank lower then spammers!

    • Re: (Score:2, Insightful)

      by KiloByte (825081)

      ... and all mails you get will be delayed by an hour or more, pretty unacceptable when you get an urgent complaint that something is down. And even in not work-related matters, making people wait for no reason is rude.

      There are many spam fighting techniques without such flaws. And other than gmail, server admins are generally smart enough to handle failures properly (ie, with instant notification that something went wrong).

      • by ShanghaiBill (739463) * on Sunday April 14, 2013 @05:57PM (#43447689)

        ... and all mails you get will be delayed by an hour or more, pretty unacceptable when you get an urgent complaint that something is down. And even in not work-related matters, making people wait for no reason is rude.

        Simple solution: Use a whitelist first. If the email is from some on your family/friend/co-worker/customer list, or someone you have corresponded with in the past, then you see it immediately. Anyone else can wait.

        • wrong tech. (Score:4, Insightful)

          by buss_error (142273) on Sunday April 14, 2013 @07:30PM (#43448107) Homepage Journal

          Better solution: Stop trying to force email to be a reliable and concurrent source of information. It has never been reliable nor has it ever been concurrent protocol. Check the default settings for sending email - try every hour for up to 5 days before giving up. Wait one day before sending a trouble report.

          That email now generally DOES deliver results in almost real time is no excuse to think it will ALWAYS deliver in real time. If your communication either critical and/or time sensitive, then email is the wrong tool to use.

      • by jhoegl (638955) on Sunday April 14, 2013 @06:00PM (#43447707)
        Email is not a priority notice system.
        If it is so urgent, pick up the phone.
        • by KiloByte (825081)

          A phone doesn't queue, can't handle you being in the loo, and so on. And for dealing with live people, it's usually "the program left a crash dump here and there, just mail it to me" -- ie, the mail is in addition to the phone call. And at that point, the live human already knows you are aware, so you're adding an hour to your response time for no gain (greylisting's 90% rejected spam doesn't add up with most other techniques).

          • by thegarbz (1787294)

            A phone doesn't queue, can't handle you being in the loo, and so on.

            Send a txt message? Leave a voice mail? Seriously there are many ways to contact people for something urgent and email is not necessarily considered one of them.

            • by KiloByte (825081)

              A text message can't include any relevant data, a voice mail takes ages to listen to. Email suffers no such flaws, is reliable and instanteous (the last part, in absence of greylisting).

              • If email were supposed to be instant, nobody would have invented Instant Messaging. Email is designed to be reliable instead of instant. That's exactly why instant messaging was invented 15 years after email was, because email was not, is not, and is not designed to be instant. It's designed to be efficient and reliable. Read the protocols some time. Have a look at how send mail works. Queues to send, queues to relay, queues to receive.
            • SMS messaging is not and never has been a guaranteed delivery system. I have been told (but cannot find a source right now) that networks can and do silently drop text messages.

              Like it nor not, Email may not initially have been an instantaneous guaranteed messaging service but that is what is expected of it now.

              • by mwvdlee (775178)

                Like it nor not, Email may not initially have been an instantaneous guaranteed messaging service but that is what is expected of it now.

                Then expectations need to be managed, because email technically isn't capable of doing what these people expect.
                Just because some people don't understand email doesn't mean email will magically change to be what they expect it to be.
                If you want instant messaging, use something like... I dunno... instant messaging.

      • Re:Greylist instead (Score:4, Informative)

        by nabsltd (1313397) on Sunday April 14, 2013 @06:20PM (#43447811)

        and all mails you get will be delayed by an hour or more, pretty unacceptable when you get an urgent complaint that something is down.

        In a correctly configured greylist, only the first e-mail ever received from a particular IP address will be delayed. Once you know an IP addresss follows the RFC and retries, then you know that even if they do send you spam, delaying it won't change that. In order to allow for the actual machine behind an IP address changing, instead of a permanent whitelist, you pick a timeout that is long enough but not too long. I use 40 days, which allows a once-monthly mailing list to not be delayed (since the timeout is reset each time you receive an e-mail from an IP). You also pre-load the database with whitelists for Google, Amazon, Yahoo, etc.

        I also set just a 4 minute delay, which means that the one e-mail is rarely even delayed by 10 minutes. I could probably get by with as short as one minute, since that would still handle the spambots that try all MX records but never try again.

        Last, since I already have a database, it makes it really easy to build my own "IP address reputation" based on the incoming e-mail, which allows me to do things like temporarily blacklist an IP that has sent a lot of spam recently, etc.

      • by dskoll (99328)

        and all mails you get will be delayed by an hour or more

        Only if you have a broken and/or stupid greylist implementation. A correct implementation will refrain (for a few weeks) from greylisting an IP address once it notices that it does retry. That makes the initial delays quite tolerable.

        • by KiloByte (825081)

          So you get breakage notifications more often than once per few weeks, for every source you have? Impressive.

          • by dskoll (99328)

            I'm not sure what you mean by "breakage notifications"

            If you're referring to automated alerts and the like, obviously you don't greylist those because you know (or should know) either the originating IP address, the envelope sender, or both.

      • by mrmeval (662166)

        Use a whitelist or better make the submitter have cryptographic credentials that your system can validate before the message gets through. If that fails have them pick up a phone.

        • by KiloByte (825081)

          So you say I should take elaborate steps just to work around damage done by greylisting? So what about just not installing the damn thing in the first place?

    • by khasim (1285)

      First off, because spam is so bad (80% of messages by some counts) just about ANYTHING that ANYONE does will reduce their spam (ignoring false positives).

      Secondly, READ YOUR LOGS!

      There are broad categories of how different groups use email (and their email infrastructure). So what works great for one group sucks for a different group.

      So I recommend something like SpamAssassin where you can tweak the settings to what works for your specific circumstances (and the people/groups that you send/receive email wit

      • Greylisting is great, except when you try to greylist gmail servers.

        What is special about gmail servers that would stop greylisting? Do they really not retry mail transmission?

        • by khasim (1285)

          What is special about gmail servers that would stop greylisting? Do they really not retry mail transmission?

          The message gets bounced to a different server that tries delivering it. Since it is a different IP address it also gets greylisted.

          So it bounces the message to a different server (probably not the first server) and tries again. And gets a different server greylisted. And so on and on and on.

          After X failures the gmail system gives up and returns the message as undeliverable.

          A lot of the big sites (hot

          • by jaseuk (217780)

            Most sane grey-listing implementations will at least assume that a resend from the same class C network is OK. This tends to work around these problems.

            Jason.

  • by magic maverick (2615475) on Sunday April 14, 2013 @05:32PM (#43447565) Homepage Journal

    And while we're at it, some hints on using a public blacklist with regards spam. The correct way is not to trust the blacklist 100%. Instead, you use it as one part of a comprehensive scheme (part of this complete breakfast). So, you may use a dictionary, and for every word in the dictionary you add 10 points (viagra, v1agra, v14gr4, etc.). You can use SPF [wikipedia.org] and if it doesn't match, then that's worth 50 points, and if it's not there, maybe 20 points. And if the domain or IP address is on a blacklist, maybe 40 points. You assign the points as you like. Then, if you hit 100 points, you mark the email as "probably spam".

    But you never reject or mark an email spam just because it's on some blacklist. That's just stupid. Now I'm off to RTFA.

    ----

    OK if you have your own blacklist (perhaps a list of domains or IP addresses that have sent email to a catch-all, or that have fallen into a honeytrap), then you do what you want. But you probably should date entries and remove old ones (if they do not misbehave again), in case a legitimate user is now at that location.

    • by PNutts (199112) on Sunday April 14, 2013 @05:58PM (#43447693)

      I don't disagree with your premise. I work in a health based organization and the SPAM and "dirty word" lexicons block legit e-mails. I've also found that for receiving e-mails SPF and most other common sense checks block too much legit mail. God forbid businesses configure their hosts / gateways correctly. And don't get me started on third party mailer services. It makes an impossible job more impossibler.

      • by danomac (1032160)

        Ditto - and the interesting thing is that most of the time when something is wonky with the hosts/gateways it's almost always the government. Go figure.

    • Another board I frequent, using the Drupal blogging software, is currently being overwhelmed with spam.

      Our beloved webmaster is experimenting with Mollum spam retarding software [mollom.com].

      This software does have its faults, as it is hindering the posting of links by some of our most informative posters. A blogsite's "good folk" need to be whitelisted so they can post links unhindered. More often than not, the most informative content of a post is a link.

      Anyone else having a blogsite overrun with crap migh
    • What you describe is SpamAssassin. The scores are learned by feeding a lot of ham and spam and finding the right balance. Of course you can (re)train the scoring with your own ham & spam, and add your own rules, etc.
      • Yes. SpamAssassin is one way to do it. The main point is, it doesn't matter how you do it, just don't ever trust a public blacklist 100%, because you'll block legitimate users. Perhaps they are at an ISP that has hosted spammers. Perhaps they are currently in a country (like the USA, I personally block all email from the USA as most of it is spam*) that spams a lot. Perhaps they pissed off the blacklist maintainers. Whatever the reason is, a blacklist is not 100% reliable.

        Also, use a whitelist. If an email

      • The problem with SpamAssassin (at least, as I've seen it deployed), is that it only determines if something is spam after the MTA has sent an acknowledgement that it has been received. At this point, you have to deliver it to the user's spam folder, because if you don't then neither the sender nor the receiver knows if a non-spam mail is accidentally dropped because of a false-positive. The nice thing about DNSBLs is that you can reject the mail early, so the sender gets a reject message. They can then t
  • by girlintraining (1395911) on Sunday April 14, 2013 @06:01PM (#43447709)

    . Your process needs to be simple and verifiable,

    The process can't be simple because spammers are endlessly creative with how they try to get past the filters. And if it was verifiable, that would mean published -- and once published, becomes useless. Spammers can simply test their latest creation against your filter, and now you effectively have given them a way to bypass your entire process, making it worthless.

    and to compensate for any errors, you want your process to be transparent to the public

    The administrative process can be transparent, but the technical process, as outlined above, cannot.

    with clear points of contact and line of responsibility.

    The problem here is; how do you tell the liars from the rest? Responsibility is fine, clear points of contact are fine, but what's the criterion for delineating between 'spam' and 'marketing'? How about between 'spam' and 'opt-in' that the user no longer wants? How about between... you get the idea. There is some grey here, and odds are good you're going to find someone doing something with a legitimate and ethical reason, that by all appearances... isn't. And then you're going to make a decision based on those appearances (because what else can you go on?) and then you're going to burn a bridge down.

    These problems can't be solved with a handwave and a post on an internet forum.

    • by MeNeXT (200840)

      .... but what's the criterion for delineating between 'spam' and 'marketing'? How about between 'spam' and 'opt-in' that the user no longer wants?

      SPAM is very clear any and all email that I did not subscribe to which is soliciting me. Now I subscribe to lists and once in a while marketing comes through some lists which I therefore unsubscribe. If it continues coming it is SPAM. Very clear to me.

  • by alen (225700) on Sunday April 14, 2013 @06:09PM (#43447741)

    most of your spam problems will be solved by simply blocking all email from those countries except for your business partners

    • by Anonymous Coward

      The easiest way to block such email is to use geoip information on the envelope sender ip address. With sendmail you need to create a milter so that the email is not accepted - accepting the email then later processing unfortunately gives the sender the impression that their email is getting through and therefore they keep sending. Rejection based on geoip means that the sender might get the message. One can also arrange for such a milter to pause for a long time before issuing the rejection ... helps slow

  • by Anonymous Coward

    in addition to making sure your data is accurate.

    • by unitron (5733)

      "
      Make sure your grammar are correct (Score:0)
      by Anonymous Coward on Sun Apr 14, '13 06:19 PM (#43447801)

      in addition to making sure your data is accurate."

      Well played, sir or madam, well played indeed!

      (even if you did beat me to it--grumble, grumble, grumble)

      Remember boys and girls, data 'R' plural, just like media.

  • by dskoll (99328) on Sunday April 14, 2013 @06:51PM (#43447973)

    ... though they are not publicly-accessible; only accessible to our customers. Here's how they work:

    Using our reputation-collection protocol [mimedefang.org], we receive a constant stream of events from our customers. An "event" is something like "IPv4 address x.y.z.w sent to a nonexistent recipient" or "IPv6 address abcd::1234 sent something that a human voted as spam"

    Currently, we have a database of just under two billion events. Once an hour, we go through our database and categorize IP addresses as:

    • Greylist Stumblers: Machines that seem to have trouble passing the greylist hurdle.
    • Dictionary Attackers: Machines that seem to send to a lot of nonexistent addresses.
    • Spam Sources: Machines that send a lot of spam.
    • Mixed: Machines that send a lot of spam, but also a lot of ham (think Yahoo's servers, for example.)
    • Good: Machines that aren't on any of the other four lists and that seem to send a lot of ham

    The whole system is 99.99% automated. The only manual intervention is when some requests delisting. If it seems that someone was the victim of a compromise and has now cleaned up his/her machine, we delist it for 45 days which is long enough for all events from that IP to expire. Then it goes back into consideration for automatic listing.

    This system works really well. We have about 3.75 million IPv4 and 3300 IPv6 addresses on our lists; those are machines for which we have confidence that there's enough data to categorize them.

  • 0. Find a system that makes their blacklistings publicly available.
    1. Send it SPAM.
    3. See what gets through, send more of that from those IPs.
    4. Tweak the stuff that didn't get through until it does.
    2. V1AGR4 !!
    5. Rotate IPs from your pool of thousands that aren't blacklisted.
    6. Prophet.
    7. GOTO 0.

    Protip: Your public blacklist is part of the fucking problem, fool. Either use a whitelist if you can (+trust graphs), or if you can't then let those blacklisted contact you if they care.

    • by Anonymous Coward

      Botnets generally don't use IP addresses, but host-domain names instead: Why? For the purposes of "fastflux" botnet construction

      So - what's that? Well, put it THIS way:

      The "infamous they" (law enforcement or other authories online etc.) take 1 out?

      Well, no big deal!

      Just "jump" to another node on your botnet in some 'enslaved' system(s) you have in it! This is done @ the botnet C&C (command & control) server master level.

      (Which of course, your botnet's infestors on clientrigs in it also has the abili

  • milter-greylist (Score:2, Insightful)

    by Anonymous Coward
    Six years ago, I wrote milter-greylist [hcpnet.free.fr]. At that time I thought some kind of distributed spam traps would be useful. I wrote software for a P2P network of mail servers that exchange signed information on messages reaching spam traps. The thing turned to be useless: greylisting alone was enough. Today, greylisting with variable delays depending on sender reputation from various DNSRBL is still enough, even is the DNSRBL information is not very reliable: an error just means an extra delay in delivery.
  • I'm running a physical server in a colo. Unfortunately about 6 months back my server sent a burst of about 5,000 spam messages. I was getting bounce messages on my admin account but with no information as to which account was breached in the bounce message, I'm scrambling about on my system, first shutting down mail, then trying to figure out if I was even sending it or just a victim of a one of the spam tricks. I did see entries in my logs, but I wasn't able to track it down to a specific account. During a

We have a equal opportunity Calculus class -- it's fully integrated.

Working...