Maintaining a Publicly Available Blacklist - Mechanisms and Principles 89
badger.foo writes "When you publicly assert that somebody sent spam, you need to ensure that your data is accurate. Your process needs to be simple and verifiable, and to compensate for any errors, you want your process to be transparent to the public with clear points of contact and line of responsibility. Here are some pointers from the operator of the bsdly.net greytrap-based blacklist."
Re: (Score:1)
Re: (Score:3)
The nice thing about yelling at yourself is you can be pretty sure someone is listening, even if they don't agree with you.
Re: (Score:1)
You end up losing mail and who is it for someone else to filter what I can and can't see. There is a delete button for a reason. Use it.
Says the Anonymous Coward.... (Whom probably has an agenda, and is too afraid to post whom he is)....
Spam lists are not just for email. If you are forum admin (or a slashdot admin), spam posts are the bane of many a website.
A good, verifyable, public, and accountable spam list means admins of such sites can spend more time on being productive, rather than wasting time deleting hundreds and hundreds of useless posts and users.
Considering this is slashdot, and the majority of people here prefer technology to
Re: (Score:1)
No it doesn't. System administrators shouldn't be filtering the mail. It's not up to you do hand-choose what mail goes into your customers mail boxes.
Some people like myself always post AC. You don't have to have an agenda any more than anybody else. You clearly have an agenda and aren't posting AC. Your probably a system administrator who feels overwhelmed by financial constraints, work, bitching from end users, etc. Spam filtering makes you more productive only because there is no better system in place.
Re: (Score:2, Funny)
Your post advocates a
(X) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
(X) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the mone
Re: (Score:2)
At least one thing positive:
(X) Microsoft will not put up with it
Isn't all that relevant anymore.
Let's update it;
(X) Google will not put up with it
Re: (Score:1)
You end up losing mail and who is it for someone else to filter what I can and can't see. There is a delete button for a reason. Use it.
Unless you are actually concerned that other people won't see your SPAM, then your concern that you won't see someone else's email because it was sent from a SPAMmy provider is adequately addressed by your choice of ISP.
So pick an ISP who doesn't filter; I'd suggest Network Solutions, since they are now charging extra for SPAM filtering because it's a valuable commodity. Don't pay the extra, and you can hit delete all you want.
Re: (Score:1)
blacklists are awesome and who gives a shit about the occasional lost mail every now and then? neurotic obsessed people, that's who.
Re: (Score:2)
blacklists are awesome and who gives a shit about the occasional lost mail every now and then? neurotic obsessed people, that's who.
Businesses, for one. You wouldn't like to lose a sale because you didn't get the request for a price offer, for instance.
Not all email is snail-social-network garbage; some people have useful things to say.
Though in reality, using a set of blacklists instead of a single one pretty much nullifies any mistakes in the individual lists.
Re: (Score:2)
I used to receive some 20K mails a day.
When unfiltered, the inflow is faster than my ability to manually delete mails.
Greylist instead (Score:3)
Use greylisting as a first defense - easily configurable in postfix, and it reduces the amount of spam dramatically. This relies on the behavior of the sender, rather than someone else's opinion of them.
Re: (Score:1)
Use greylisting as a first defense - easily configurable in postfix, and it reduces the amount of spam dramatically. This relies on the behavior of the sender, rather than someone else's opinion of them.
Problem is, reducing 90% of spam isn't good enough, you need to remove 99%+. Even if just a handful get through, it is and underestimated productivity killer, and.. and increasing amount of spam also contain malware. I have been on the wrong end of a blacklist because we had an open relay, so know how frustrating it can be. We are currently using a service called Norman SecureTide that do block 99%+. Not sure which blacklists they are using, if any (might only be scanning and their own reputation) but damn
Re:Greylist instead (Score:5, Insightful)
If you ran an open relay you were on the right end of a blacklisting.
Re: (Score:2)
If you ran an open relay you were on the right end of a blacklisting.
I wish I had mod points for you.
Re: (Score:3)
If you ran an open relay you were on the right end of a blacklisting.
Right, because although we're after the content of these e-mails, guilt by association is a perfectly valid technique for eliminating spam. Just like bombing a city to get rid of the army in it is totally okay... nevermind the civilians.
Re: (Score:1)
If you ran an open relay you were on the right end of a blacklisting.
Right, because although we're after the content of these e-mails, guilt by association is a perfectly valid technique for eliminating spam. Just like bombing a city to get rid of the army in it is totally okay... nevermind the civilians.
Your misunderstand (probably intentionally, this is Slashdot after all).
Let's say Adam makes a list of gun stores that verifiably have provided guns without background checks, in a state where such checks are voluntary, to people who use them to commit crimes.
Bob is placed on the list. Bob complains: "I should not be on the list because I did not know the guns I provided would be used for crimes".
Adam responds appropriately: "But that's not what I am making a list of. It doesn't matter whether you knew. To
Re: (Score:2)
Your misunderstand (probably intentionally, this is Slashdot after all).
Let's say Adam makes a list of gun stores that verifiably have provided guns without background checks, in a state where such checks are voluntary, to people who use them to commit crimes.
Just so you know, there are no stores (in the U.S.) where guns can be purchased without a background check; it's not voluntary..
Re: (Score:2)
Not sure if serious. I don't think even the most hardcore spammers who rage on spamhaus ever tried to compare blacklisting open relays to mass murder.
Terrorism, yes. Mass murder, not even they have sank that low. Congratulations, you sank lower then spammers!
Re: (Score:2, Insightful)
... and all mails you get will be delayed by an hour or more, pretty unacceptable when you get an urgent complaint that something is down. And even in not work-related matters, making people wait for no reason is rude.
There are many spam fighting techniques without such flaws. And other than gmail, server admins are generally smart enough to handle failures properly (ie, with instant notification that something went wrong).
Re:Greylist instead (Score:4, Insightful)
... and all mails you get will be delayed by an hour or more, pretty unacceptable when you get an urgent complaint that something is down. And even in not work-related matters, making people wait for no reason is rude.
Simple solution: Use a whitelist first. If the email is from some on your family/friend/co-worker/customer list, or someone you have corresponded with in the past, then you see it immediately. Anyone else can wait.
wrong tech. (Score:4, Insightful)
Better solution: Stop trying to force email to be a reliable and concurrent source of information. It has never been reliable nor has it ever been concurrent protocol. Check the default settings for sending email - try every hour for up to 5 days before giving up. Wait one day before sending a trouble report.
That email now generally DOES deliver results in almost real time is no excuse to think it will ALWAYS deliver in real time. If your communication either critical and/or time sensitive, then email is the wrong tool to use.
Re: (Score:2)
I don't think that "concurrent" means what you think it does.
Rgds
Damon
Re:Greylist instead (Score:5, Insightful)
If it is so urgent, pick up the phone.
Re: (Score:2)
A phone doesn't queue, can't handle you being in the loo, and so on. And for dealing with live people, it's usually "the program left a crash dump here and there, just mail it to me" -- ie, the mail is in addition to the phone call. And at that point, the live human already knows you are aware, so you're adding an hour to your response time for no gain (greylisting's 90% rejected spam doesn't add up with most other techniques).
Re: (Score:2)
A phone doesn't queue, can't handle you being in the loo, and so on.
Send a txt message? Leave a voice mail? Seriously there are many ways to contact people for something urgent and email is not necessarily considered one of them.
Re: (Score:2)
A text message can't include any relevant data, a voice mail takes ages to listen to. Email suffers no such flaws, is reliable and instanteous (the last part, in absence of greylisting).
No, it's reliable, not instant. Instant Messaging (Score:2)
Re: (Score:2)
SMS messaging is not and never has been a guaranteed delivery system. I have been told (but cannot find a source right now) that networks can and do silently drop text messages.
Like it nor not, Email may not initially have been an instantaneous guaranteed messaging service but that is what is expected of it now.
Re: (Score:2)
Like it nor not, Email may not initially have been an instantaneous guaranteed messaging service but that is what is expected of it now.
Then expectations need to be managed, because email technically isn't capable of doing what these people expect.
Just because some people don't understand email doesn't mean email will magically change to be what they expect it to be.
If you want instant messaging, use something like... I dunno... instant messaging.
Re:Greylist instead (Score:4, Informative)
and all mails you get will be delayed by an hour or more, pretty unacceptable when you get an urgent complaint that something is down.
In a correctly configured greylist, only the first e-mail ever received from a particular IP address will be delayed. Once you know an IP addresss follows the RFC and retries, then you know that even if they do send you spam, delaying it won't change that. In order to allow for the actual machine behind an IP address changing, instead of a permanent whitelist, you pick a timeout that is long enough but not too long. I use 40 days, which allows a once-monthly mailing list to not be delayed (since the timeout is reset each time you receive an e-mail from an IP). You also pre-load the database with whitelists for Google, Amazon, Yahoo, etc.
I also set just a 4 minute delay, which means that the one e-mail is rarely even delayed by 10 minutes. I could probably get by with as short as one minute, since that would still handle the spambots that try all MX records but never try again.
Last, since I already have a database, it makes it really easy to build my own "IP address reputation" based on the incoming e-mail, which allows me to do things like temporarily blacklist an IP that has sent a lot of spam recently, etc.
Re: (Score:3)
and all mails you get will be delayed by an hour or more
Only if you have a broken and/or stupid greylist implementation. A correct implementation will refrain (for a few weeks) from greylisting an IP address once it notices that it does retry. That makes the initial delays quite tolerable.
Re: (Score:2)
So you get breakage notifications more often than once per few weeks, for every source you have? Impressive.
Re: (Score:2)
I'm not sure what you mean by "breakage notifications"
If you're referring to automated alerts and the like, obviously you don't greylist those because you know (or should know) either the originating IP address, the envelope sender, or both.
Re: (Score:2)
Use a whitelist or better make the submitter have cryptographic credentials that your system can validate before the message gets through. If that fails have them pick up a phone.
Re: (Score:2)
So you say I should take elaborate steps just to work around damage done by greylisting? So what about just not installing the damn thing in the first place?
Knobs! (Score:3)
First off, because spam is so bad (80% of messages by some counts) just about ANYTHING that ANYONE does will reduce their spam (ignoring false positives).
Secondly, READ YOUR LOGS!
There are broad categories of how different groups use email (and their email infrastructure). So what works great for one group sucks for a different group.
So I recommend something like SpamAssassin where you can tweak the settings to what works for your specific circumstances (and the people/groups that you send/receive email wit
Re: (Score:2)
Greylisting is great, except when you try to greylist gmail servers.
What is special about gmail servers that would stop greylisting? Do they really not retry mail transmission?
Re: (Score:3)
The message gets bounced to a different server that tries delivering it. Since it is a different IP address it also gets greylisted.
So it bounces the message to a different server (probably not the first server) and tries again. And gets a different server greylisted. And so on and on and on.
After X failures the gmail system gives up and returns the message as undeliverable.
A lot of the big sites (hot
Re: (Score:2)
Most sane grey-listing implementations will at least assume that a resend from the same class C network is OK. This tends to work around these problems.
Jason.
Using a blacklist ... (Score:5, Interesting)
And while we're at it, some hints on using a public blacklist with regards spam. The correct way is not to trust the blacklist 100%. Instead, you use it as one part of a comprehensive scheme (part of this complete breakfast). So, you may use a dictionary, and for every word in the dictionary you add 10 points (viagra, v1agra, v14gr4, etc.). You can use SPF [wikipedia.org] and if it doesn't match, then that's worth 50 points, and if it's not there, maybe 20 points. And if the domain or IP address is on a blacklist, maybe 40 points. You assign the points as you like. Then, if you hit 100 points, you mark the email as "probably spam".
But you never reject or mark an email spam just because it's on some blacklist. That's just stupid. Now I'm off to RTFA.
----
OK if you have your own blacklist (perhaps a list of domains or IP addresses that have sent email to a catch-all, or that have fallen into a honeytrap), then you do what you want. But you probably should date entries and remove old ones (if they do not misbehave again), in case a legitimate user is now at that location.
Re:Using a blacklist ... (Score:4, Insightful)
I don't disagree with your premise. I work in a health based organization and the SPAM and "dirty word" lexicons block legit e-mails. I've also found that for receiving e-mails SPF and most other common sense checks block too much legit mail. God forbid businesses configure their hosts / gateways correctly. And don't get me started on third party mailer services. It makes an impossible job more impossibler.
Re: (Score:2)
Ditto - and the interesting thing is that most of the time when something is wonky with the hosts/gateways it's almost always the government. Go figure.
mollom spam (Score:2)
Our beloved webmaster is experimenting with Mollum spam retarding software
This software does have its faults, as it is hindering the posting of links by some of our most informative posters. A blogsite's "good folk" need to be whitelisted so they can post links unhindered. More often than not, the most informative content of a post is a link.
Anyone else having a blogsite overrun with crap migh
SpamAssassin just called... (Score:2)
Re: (Score:2)
Yes. SpamAssassin is one way to do it. The main point is, it doesn't matter how you do it, just don't ever trust a public blacklist 100%, because you'll block legitimate users. Perhaps they are at an ISP that has hosted spammers. Perhaps they are currently in a country (like the USA, I personally block all email from the USA as most of it is spam*) that spams a lot. Perhaps they pissed off the blacklist maintainers. Whatever the reason is, a blacklist is not 100% reliable.
Also, use a whitelist. If an email
Re: (Score:2)
Not realistically achievable (Score:4, Insightful)
. Your process needs to be simple and verifiable,
The process can't be simple because spammers are endlessly creative with how they try to get past the filters. And if it was verifiable, that would mean published -- and once published, becomes useless. Spammers can simply test their latest creation against your filter, and now you effectively have given them a way to bypass your entire process, making it worthless.
and to compensate for any errors, you want your process to be transparent to the public
The administrative process can be transparent, but the technical process, as outlined above, cannot.
with clear points of contact and line of responsibility.
The problem here is; how do you tell the liars from the rest? Responsibility is fine, clear points of contact are fine, but what's the criterion for delineating between 'spam' and 'marketing'? How about between 'spam' and 'opt-in' that the user no longer wants? How about between... you get the idea. There is some grey here, and odds are good you're going to find someone doing something with a legitimate and ethical reason, that by all appearances... isn't. And then you're going to make a decision based on those appearances (because what else can you go on?) and then you're going to burn a bridge down.
These problems can't be solved with a handwave and a post on an internet forum.
Re: (Score:2)
.... but what's the criterion for delineating between 'spam' and 'marketing'? How about between 'spam' and 'opt-in' that the user no longer wants?
SPAM is very clear any and all email that I did not subscribe to which is soliciting me. Now I subscribe to lists and once in a while marketing comes through some lists which I therefore unsubscribe. If it continues coming it is SPAM. Very clear to me.
block russia, china and india (Score:3)
most of your spam problems will be solved by simply blocking all email from those countries except for your business partners
Re: (Score:1)
The easiest way to block such email is to use geoip information on the envelope sender ip address. With sendmail you need to create a milter so that the email is not accepted - accepting the email then later processing unfortunately gives the sender the impression that their email is getting through and therefore they keep sending. Rejection based on geoip means that the sender might get the message. One can also arrange for such a milter to pause for a long time before issuing the rejection ... helps slow
Make sure your grammar are correct (Score:1)
in addition to making sure your data is accurate.
Re: (Score:2)
"
Make sure your grammar are correct (Score:0)
by Anonymous Coward on Sun Apr 14, '13 06:19 PM (#43447801)
in addition to making sure your data is accurate."
Well played, sir or madam, well played indeed!
(even if you did beat me to it--grumble, grumble, grumble)
Remember boys and girls, data 'R' plural, just like media.
We run DNS-based lists (Score:5, Interesting)
... though they are not publicly-accessible; only accessible to our customers. Here's how they work:
Using our reputation-collection protocol [mimedefang.org], we receive a constant stream of events from our customers. An "event" is something like "IPv4 address x.y.z.w sent to a nonexistent recipient" or "IPv6 address abcd::1234 sent something that a human voted as spam"
Currently, we have a database of just under two billion events. Once an hour, we go through our database and categorize IP addresses as:
The whole system is 99.99% automated. The only manual intervention is when some requests delisting. If it seems that someone was the victim of a compromise and has now cleaned up his/her machine, we delist it for 45 days which is long enough for all events from that IP to expire. Then it goes back into consideration for automatic listing.
This system works really well. We have about 3.75 million IPv4 and 3300 IPv6 addresses on our lists; those are machines for which we have confidence that there's enough data to categorize them.
How to get your botnet around a Blacklist. (Score:3)
0. Find a system that makes their blacklistings publicly available.
1. Send it SPAM.
3. See what gets through, send more of that from those IPs.
4. Tweak the stuff that didn't get through until it does.
2. V1AGR4 !!
5. Rotate IPs from your pool of thousands that aren't blacklisted.
6. Prophet.
7. GOTO 0.
Protip: Your public blacklist is part of the fucking problem, fool. Either use a whitelist if you can (+trust graphs), or if you can't then let those blacklisted contact you if they care.
Ever heard of "FastFlux" botnet design? (Score:1, Insightful)
Botnets generally don't use IP addresses, but host-domain names instead: Why? For the purposes of "fastflux" botnet construction
So - what's that? Well, put it THIS way:
The "infamous they" (law enforcement or other authories online etc.) take 1 out?
Well, no big deal!
Just "jump" to another node on your botnet in some 'enslaved' system(s) you have in it! This is done @ the botnet C&C (command & control) server master level.
(Which of course, your botnet's infestors on clientrigs in it also has the abili
milter-greylist (Score:2, Insightful)
How to get off? (Score:2)
I'm running a physical server in a colo. Unfortunately about 6 months back my server sent a burst of about 5,000 spam messages. I was getting bounce messages on my admin account but with no information as to which account was breached in the bounce message, I'm scrambling about on my system, first shutting down mail, then trying to figure out if I was even sending it or just a victim of a one of the spam tricks. I did see entries in my logs, but I wasn't able to track it down to a specific account. During a