University Capitulates, Switches Off Spam Filters 470
Heraklit writes "As reported on German news site Heise, the system administrators of the Technical University of Braunschweig have temporarily given up the fight against spam. Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!), they decided to switch off all filter mechanisms. Before, the 20 servers dedicated to processing e-mail alone had been breaking down under a load of 100000 unprocessed mail messages, ca. 98% of which had been spam or viruses. ... A similar e-mail jam occurred recently at the IT central of the German Federal Government.
Is this the beginning of the end of e-mail?" (The Fish may be useful.)
Question? (Score:5, Insightful)
Re:Question? (Score:4, Interesting)
Re:Question? (Score:4, Insightful)
I'm sure that's not the point - it's easy to deliver 100K mails, but the problem is that you've got to manually check for false positives and un-mark them as good email.
Re:Question? (Score:5, Interesting)
No one (sane) *manually* checks for false positives, just the end user. You do need manual personnel to follow up on end user inquiries, but it should be moot. If you have the right spamblocking service/setup, you're not going to get false positives...
Re:Question? (Score:4, Insightful)
Hence, a difficulty for the end users to mark themselves the false positives....
Re:Question? (Score:4, Insightful)
Re:Question? (Score:4, Interesting)
I work at a UK university and we're introducing a new system to deal with spam. We've already got an in-house product, MailScanner [soton.ac.uk] which does the detection job pretty well, but our mail servers are quite loaded with junk.
We're about to offer a "delete at gateway" option, so our users don't have to filter their email and lessen the load on the mail servers at the same time. This service is optional, so our users can choose whether they want it, but we'll be strongly encouraging them to use it.
Additionally, they can set their spam threshold, so they can delete most spam, but review the borderline cases.
Re:Question? (Score:5, Informative)
You can increase the threshhold at which you declare spam to be spam. Allows for more misses, but reduces the false positives to, essentially, nothing.
Or, you can just tag likely spam with ***SPAM*** in the subject and let the user deal with it.
Or even better, you can direct likely spam into a specific IMAP folder on the server that the user's client can subscribe to and they can glance at their personal SPAM folder on the server whenever they want without having to download all the bodies.
As someone who personally uses postfix+procmail+spamassassin+razor and recieves 4,000 emails per day, I am currently filtering out 98% of the spam on the server and have had ZERO false positives in two years and 2.9million messages.
Statistically, you will eventually get some false positives - especially if you have a large userbase (as opposed to just one or two accounts). But if one out of every few million messages isn't acceptable, you can just use one of the previously suggested methods.
The worst you can do is nothing at all.
No false positives? (Score:5, Insightful)
Have you personally reviewed the 2.9M messages which were filtered out... if you have then i'd question the value of your filtering.
I know i've occasionally had false positives and i get nowhere near your message volume. My personal favorite is the UK paypal-esque service NoChex which sends emails with the subject line "YOU'VE GOT CASH!!"...
Re:No false positives? (Score:4, Interesting)
Anyway, if an amateur could do that well, I'm sure close enough to 100% accuracy *can* be achieved by a professional solution. In fact, it's made me wonder why some solutions don't perform better than they do.
It's a moving target (Score:5, Insightful)
Re:Question? (Score:5, Funny)
They had a team of 20 monkeys that would read the emails and determine if they were spam. Unfortunately, the monkeys are easily distracted, so anytime they got spam about banannas, they would lose focus. This lead to the backlog.
What? you have never gotten bananna spam before?
Re:Question? (Score:4, Funny)
Yes, but the emphasis wasn't so much the banana as it was where it was inserted.
Re:Question? (Score:4, Funny)
No but I often get asked if I'm satisfied with the size and/or performance of my ba|\|a|\|a
Re:Question? (Score:4, Funny)
White listing + Auth tokens (Score:3, Interesting)
Byebye, spam.
Byebye, email.
Don't forget the other problem... (Score:4, Interesting)
Re:Don't forget the other problem... (Score:4, Interesting)
--
New deal processing engine online: http://www.dealsites.net/livedeals.html [dealsites.net]
Re:Don't forget the other problem... (Score:3, Interesting)
Assist, not preempt, the user. (Score:4, Interesting)
If required they can also set a spam level on the mail server in a MySQL user/account database to automatically delete mail over the specified threshold (for accounts receiving oodles of obvious spam).
It has a nice balance between performance, security, and leaving most of the control in the hands of the users. We haven't faced extreme loads but it hasn't even raised an eyebrow over the load so far. Most importantly, no unhappy usres complaining of missing emails...
Q.
Re:Don't forget the other problem... (Score:4, Interesting)
All of them. Don't process them, just ban them.
If you want to send a file, use ftp or send a link to a read-only http or smb/nfs share.
Using email server blacklists instead of filters on the spam will work a lot better too. Hopefully they aren't using a brain damaged email server like Exchange.
Finally (Score:3, Funny)
Finally, I can get my "male enhancement" emails again.
I wonder... (Score:5, Funny)
Re:I wonder... (Score:5, Funny)
Same thing.
Spam And Viruses (Score:5, Insightful)
Re:Spam And Viruses (Score:5, Interesting)
And that warning is so useful. Who do you send it to?
These messages are a waste of everyone's time. I get hundreds of worms daily...but I never see them, because they're easy to filter. What I do see are these damned "helpful" messages that "I" sent someone a virus. Those are much harder to filter.
Much better way: reject viruses in the SMTP transaction. The SMTP client is then responsible for notifying the sender. If that client is a virus or worm, it will do nothing; no one is bothered. If it's a false positive, the sender will get the bounce. Reliable, unobstrusive.
If you want to filter email politely, you must follow these rules [advogato.org]. People who don't cause the rest of us constant headaches. The worst thing is that they don't even realize it.
Re:Spam And Viruses (Score:5, Interesting)
I once confronted a sysop about this and they told me "if we don't email them back people won't know the message was rejected". Apparently the idea of checking while reading the message never crossed his mind.
As another poster suggested I just filter out all "warning" emails as junk which helps.
Tom
Re:Spam And Viruses (Score:4, Informative)
There are many cases where email is relayed before being sent to a system that does virus scanning. (Consider what happens when you use sendmail aliases and virtual domain entries that contain somthing on the order of "user: user@someotherhost.com".)
Your SMTP 5xx error will cause the relaying server to generate a bounce. The bounce will go to the person listed by the forged "To" headers, and will even include a copy of the Virus.
The proper way to deal with email worms is to quietly delete them.
Re:Spam And Viruses (Score:4, Interesting)
I would question quietly deleting such mails. Most of the worm/virus ridden mails that I get come from people who have infected systems and where I am in their address book. They need to know they have an infected system.
I quarantine all the worms/viruses sent to my system. I look through the quarantine directory about once a week. On ONE occacion (out of a few hundred virus laden messages) I was able to determine who was sending the virus. The vast majority of the time the viruses don't leak any information about the system, and they come from dynamic IP addreses. Delivering the virus, or a "user X sent you a virus" message to the user is useless. I've never once had a false positive (and I believe the chance of false positives is about zero).
Delivering the virus laden email is just stupid. The reasons deleting it, or quarantining it far outweigh the reasons for delivering it. I'm pretty good about being able to track where a virus came from and I was only able to track down one virus origin. End users are going to have zero ability, and zero interest in doing do. They'll actually send out false "you've got a virus" reports to their friends (who don't actually have a virus, the from address was just forged).
Re:Spam And Viruses (Score:4, Insightful)
One small quibble about a final point in those rules:
It's indirect? What's a good way to transfer binary files that is both direct and secure? ... and archived with a personal note. One handy thing I do for large attachments is to upload them to a http server and send the link. But this is a pain in the ass for anything other than the biggest files. What are the good options otherwise?
Re:Spam And Viruses (Score:5, Informative)
Two things:
The approach that we take is the following: We mark virus messages with a special header and deliver them in a dedicated folder in the user's mailbox. Most users simply delete all messages in this folder, but then it is their choice, we abide to all laws and do not generate bounce messages.
Sebastian
Mirror (Score:3, Informative)
Akamai Mirror [akamaitech.net].
Probably a better alternative... (Score:4, Insightful)
What about network load? (Score:3, Interesting)
However, I work in a large organisation, and with a 98% spam ratio, the mail infrastructre would need to be much larger (and more expensive!) than it actually neeeds to be. Let alone the (*&&^$@# junk traffic and bounces caused by auto-responses to forged addresses. Plus we have a significant number of staff who are clueless who would be excluded from communicating effectively because they do not have the time or skills to learn
Re:What about network load? (Score:4, Interesting)
Plus we have a significant number of staff who are clueless who would be excluded from communicating effectively because they do not have the time or skills to learn how to train a spam filter. in such a situation, no-one could no-longer *rely* on email to contact/inform our staff, reducing its value as a tool.
True, I also work in a large international organisation, but our Spam/Ham ratio in "only" about 40%..
I am handling the Spam problem and we have been running SpamAssassin, as a pilot project, for the last year.
The SpamAssassin project almost got replaced by a commercial solution when people started asking themselves, "what good is it if we still deliver, the Spam to the users inboxes ?". Our users may be experts in other fields, but for many, computers are not their thing.
Some commercial solutions have "Quarantine" system where you can send a report once a day to the recipients, with a list of all spam they received the day before, with a link for each email the user can click if he wants it delivered to his inbox.
It took me 4 days, but I wrote my own Quarantine system that does exactly that, and got permission to release it under the GPL.. [biodef.org]
That way the Spam doesn't constantly flow in the user's inboxes and takes up the users time. (And, 'no' manually creating a filter rule for thousands of users is not an option)
It's done. (Score:4, Interesting)
Yes. When one university decides to stop filtering SPAM the entire world's infrastructure has effectively been shut down. Oh wait... no.
My UIC account gets NO spam (because I don't give it to anyone
Anyway I don't see anyone stopping you from using your own SPAM filter. Let's not blow this out of proportion, please.
Re:It's done. (Score:4, Funny)
Re:It's done. (Score:5, Funny)
Exactly! That's why I require all my users to use multi-case letters, symbols and numbers as their email address. I also require them to change the address every couple of weeks to a value different than any previous value (in case some spammer has managed to brute force it, or the user has leaked it). This has practically eliminated spam and reduced the mail server's storage usage by 99.9% (though the mail server still has to work really hard sending all those 550's).
Re:It's done. (Score:5, Funny)
Have you had your coffee today?
Don't be so naive (Score:3, Interesting)
In addition, I have two accounts that I use regularly -- one that I give to everyone (web registration forms, etc) where I don't care about spam, a
blacklists (Score:4, Interesting)
Re:blacklists (Score:3)
The Delivery Obligation Is Their Problem (Score:3, Insightful)
Better to just not deliver ANY mail than to deal with that requirement.
Re:blacklists (Score:5, Insightful)
You may think it is okay to block email from China or even the whole Asia because you don't know some Asians in person, but please check again where your RAM, mobo, anime etc come from... A lot of companies and university have collaborations overseas as well...
We don't really have much options left... Basically, you will have to blacklist all the high boardband provider's IP range (rr, earthlink etc)... Sorry, geeks, your email server will no longer work... It is not really an ideal solution. The other idea is kind of similar to secured DNS, ie, mail server retrieves "good IPs" from a central server. Email originated elsewhere are assigned with very low priority or filtered out altogether.
Everyone needs to be registered with their mail server with the governing body (similar to the domain name idea), say for $100 per IP. It is not that expensive if you really need that... But, prohibitive for spammer... Yes, it makes home run email server more expensive... But, you cannot get a domain name for free anyway. Why should we expect email server to be free? It may be the solution to get the economy of spamming right again.
Re:blacklists (Score:5, Insightful)
As someone who lives in China I get more than a little tired of being filtered out because of the continent I live in. (Especially since the vast majority of spam I get is selling products from America, regardless of what server they're sending them through.) And in this particular case, being a university it's very likely that they have a sizeable number of students from China, and many staff with academic links.
Re:blacklists (Score:5, Funny)
Re:blacklists (Score:3, Insightful)
Then bitch at the Chinese ISPs who allowed the problem to exist in the first place.
Re:blacklists (Score:5, Insightful)
Your faith is touching. Was it Nixon who started the first "war on drugs"? How's that going?
end of email? (Score:4, Interesting)
Of course more bandwidth is wasted on spam mails, but since I don't see much of it, it doesn't bother me so much.
What do you propose to use instead of email? instant messaging? Talk about waste of time
Re:end of email? (Score:5, Insightful)
Afterall, the "from" field is a total free-response section in SMTP with no need to authenticate that you're really associated with the address you claim to be. That and other weaknesses are why spam is so hard to kill in the first place.
We'd be in a much better place if our e-mail system at least had a trustworthy traceback facility so that we affirmatively know who sent the message by default.
Re:end of email? (Score:3, Insightful)
Once this new email2 protocol is invented, how long would it take to be implemented around the world by every admin?
What happens when that protocol gets hacked (probably by the spammers)?
I think its the right direction to make an email2 protocol but it wont be easy.
Securing the entry point (Score:3, Insightful)
I agree that SMTP needs a makeover, but what to replace it with is still very much an open question.
Re:end of email? (Score:5, Insightful)
What's OK for you may not be OK for other people. Personally, I get about 200 spams a day, versus about 1-2 real e-mails. When the ratio of spam to good mail is 100:1, it gets hard to implement spam filtering that's accurate enough to do the job. And are you under the illusion that you aren't paying your ISP for the bandwidth they waste dealing with spam?
There are some basic problems here:
Re:end of email? (Score:4, Funny)
the word(s) spamassasin literally...
Comment removed (Score:4, Insightful)
Re:It'll never die. (Score:5, Insightful)
I would agree, but only on a few stipulations. E-mail as we know it will almost certainly die sooner or later, to be replaced with something else that better fits our future needs. Like gopher and http, smtp, pop, and imap will all sooner or later be replaced by another set of protocols. Perhaps they will require something like SPF to reduce spoofed "From" headers. Perhaps they will support or even require encryption? Face it. Sooner or later, e-mail as we know it will die, but only when something else is able to take its place.
20 servers for only 100,000 messages? (Score:5, Interesting)
Re:20 servers for only 100,000 messages? (Score:5, Funny)
Re:20 servers for only 100,000 messages? (Score:5, Informative)
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 06:46:39 +0200 (METDST)
Re:20 servers for only 100,000 messages? (Score:4, Informative)
Re:20 servers for only 100,000 messages? (Score:3, Interesting)
Re:20 servers for only 100,000 messages? (Score:4, Insightful)
Re:20 servers for only 100,000 messages? (Score:5, Funny)
1 server processes spam, 1 processes viruses, 1 is a DNS server. The other 17 process data for the SETI@home [berkeley.edu] German team.
Re:20 servers for only 100,000 messages? (Score:3, Funny)
Another riduculous law! (Score:3, Interesting)
Is it just me or is this another ridiculous law? The University is providing free email services to those that are students at this establishment and they obviously need to filter out spam in order to be able to offer this service with there current hardware requirements. Spam is a legitimate problem and people that are offering free email services should be allowed to attempt to filter it as it can be extremely taxing on a busy mailserver. They can filter the spam without being intrusive or breaking privacy laws so I see no reason that it should be prevented by law.
Re:Another riduculous law! (Score:3, Insightful)
Re:Another riduculous law! (Score:5, Informative)
Real Time Blackhole Lists (Score:5, Insightful)
Personally, if it were my universtiry, I would prefer they started to use a RTBL. The fact of the matter is, if the likely spam isn't sorted out first, I have to try to discern the stuff entirely by hand. And although I can easily pick out Viagra ads, I have relatives and the occasional acquaintence who send mail that looks awfully like spam. Didn't want to type a subject. Used "hello" as the subject. Didn't configure their mail client properly, so their "replyto" looks crazy. Without some initialy spam filtering, I would miss at least some of these -- in fact, I'd probably miss more mail with no filtering than with a judicious blackhole in front of me.
Love or hate SPEWS [spews.org] and other kinder [spamhaus.org], gentler RTBLs [spamcop.net], they're better than the present choice. It would certainly reduce the load of these email servers to where it could be more easily handled. And, if nothing else, they couldbe used to prioritize mail. Use Spam Assassin or something else to do some initial tag and filter so that mail coming from Asian IPs or originating from mail servers on cable/ADSL networks gets put into the "slow" processing queue while everything else gets sent down the faster pipe.
</spouting with little to no knowledge>
Client Side Filters (Score:3, Insightful)
It's not the end of the world. There's a few good spam filters for outlook and outlook express, and some really awesome free ones for linux/unix.
Self-Destructing E-Mail helps (Score:5, Informative)
Beginning of the end? (Score:5, Insightful)
Something has to be done soon or email just wont be practical to have. Between Spam and viruii its overloading a lot of comanines network feed and servers..
And don't forget the cost of having to maintain antispam and antiviral solutions..
I know personally where I'm at, we are hitting over 2/3 of all email is spam/virus. ( i hear we drop 10k a day from the black hole list alone )
At home its 98%...
Parasites (Score:3, Interesting)
I mean, even parasites usually try to not kill the host.
*sigh*
Re:Parasites (Score:4, Insightful)
Even if the spammers band together and make a big organziation to self organize and police, spammers by almost by definition dishonest (no honor among theives!), and as soon as one realizes that he can make more money by ignoring the organzation (i.e. almost immediately), he will.
Reject at SMTP time solved the problem.. (Score:4, Interesting)
Rejected mails thus don't generate any undeliverable bounce messages to fill up the local mail queue, and the sender gets an immediate response.
In tomorrows news (Score:3, Funny)
No filter day (Score:3, Interesting)
Reverse DNS to MX record checking.... (Score:4, Informative)
Re:Reverse DNS to MX record checking.... (Score:4, Insightful)
FUD ALERT, FUD ALERT (Score:3, Funny)
Wish my university would get rid of filters (Score:5, Interesting)
Yet this same university loves to publish my email address on the web; ensuring I get tons of spam(some even in Chinese!)
I hate when the community at large has to pay for the transgressions of a few slimeballs and the idiocy of some(not even most) gullible windows users.
Ideas for a new email protocol... (Score:5, Insightful)
I would say this is probably not the end of email, nor is it the end of the Internet as a whole. However, it is probably the end of the protocols currently used to send and receive email.
I believe that spam is ultimately a security issue, because it slows down systems and creates problems for users and system administrators. Sometimes, security problems are caused by buffer overruns and other programming errors. However, in this case, I think the entire protocol is faulty. It may have worked wonderfully before spammers, but it's time to introduce something new that will make it extremely difficult to send spam.
I don't know exactly how the new protocol needs to look. But I have some ideas. Paying for "postage" is not one of them, as I think it is a very bad idea. Unless some payment system could be set up whereby the recipient of the mail receives the payment, not some 3rd party, like Microsoft, which would profit incredibly from garbage spam mails going all over the place. In fact, if that were the setup, then each recipient could state a price per email and/or per kilobyte of the mail message for receiving an email from a source, which the source would pay to the recipient as postage. A whitelist could be set up to allow certain senders, like one's friends, family, coworkers, etc., to send emails without paying the recipient. A blacklist could be set up to disallow all emails from specific senders and/or domains, as we have today, and if you read further in this post, you'll see my ideas for making sure that addresses are not spoofed. But I digress...
Perhaps first of all, the mail headers need to include digital signatures based on the source and destination domain names, email addresses, and other identifying information that is unique to each email sent. To avoid address spoofing, for example, people sending junk with a 'yahoo' or 'hotmail' address, when in fact it originates elsewhere, each such domain would have a private key, which upon sending, would be used in the computation. A valid signature could not be computed when the address is spoofed, and so all spammers would need to use their own valid domain name. Further, the need to make computations would make it more costly for spammers to send mail in high volumes. The algorithm should be designed so that recipients of email will have a much lower cost to verify the key. Further, the signature system could, should, and would be used to verify that each bit of the contents of the email, including all attachments, arrived correctly and without being tampered with or corrupted in transit.
Something is not right (Score:5, Insightful)
But there are some other issues you need to look at, with these emails not being scanned - do you know how much of storage you need to have online to have a mailstore this size and developing by the hour at 100k msgs ? not everyonce will use pop3 to get their emails, and not all the users will check email every day. Were talking about a very very large and very well setup Mail Store for this kind of volume. What about network bandwidth ?
A few basic things can reduce the work of those servers : Duplicacy level across these emails is going to be very high - all 100k emails per hour cannot be unique, there are going to be loads and loads of dupes, that dont even need to be scanned.
Creating a small database in-house with bad MailSender's list ( kind of like an in house RBL ), and flushing that list on an 6 hour interval will slow the inflow as well to quite an extent - in some tests done, i have seen it go down by almost 15 - 18% when there is a heavy load. Since most 'real' mailservers tend to retry, even if a genuine mailserver is blacklisted for 6 hours - it wont make much of a difference, however most 'hijacked PC's sending spam' dont have any retry or resending mechanism - and will just not be able to send into your server.
Another issue that helps stem the tide of bad email is to check for Virus infections before checking for spam. A lot of cases the tides of mail coming in can be virus infections ( which are easier and faster to check against - compared to rules + logic based spam checkers ).
However, all this is said and done without knowing of what system and what kind of a setup they use, there is no way anyone can really know what happened and why.
In the end, classic case for Linux and Unix based technologies to come into the frame I think.
Solution: (Score:5, Interesting)
Joking aside, it boils down to economics. Spam is profitable. If something is profitable, people will do it. Selling drugs is profitable, and the war on some drugs hasn't changed that. The answer to spam (and drugs) is not to try and stop them, but to make doing them unprofitable.
What makes spam profitable is the presence of people on the internet who are SO incredibly stupid that they fall for it. (See Junkie loves his spam [slashdot.org]) Remove them, and you shoot spam through it's purtid heart. I can think of several methods of doing do:
- If you respond to spam, you've probably got shitloads of viruses on your computer. Beyond any shadow of a doubt some of them are spamming people. If you ISP detects lots of mystery traffic from you on known virus ports, you're given one warning. Then you get kicked off without ceremony and not allowed to reconnect until you can prove to them that all computers using your connection are malware-free. No more malware, no more spam zombies.
- (You, the ISP) Send test-spams. Specify in the header that it is NOT a real spam so you don't get blacklisted. Anyone who responds to them loses all services except port 80 until you prove to a professional who visits your house that you know enough not to buy from spam. Do it again and you will never be allowed to use your ISP's mail servers again.
Neither of these can possibly be routed around or hacked by spammers, because they are not involved in any part of the process. If you are not in the habit of perpetuating malware with your computer, you needn't worry of getting caught up in it all. Neither of these requires a major invasion of your privacyDisc space vs. CPU (Score:3, Informative)
Good luck with that approach! If their primary constraint is budgetary, as it would seem, it would make more sense to invest *more* in filtering so that the crap didn't get to users' mailboxes where it will doubtless stay indefinitely in some cases.
Note: I'm assuming that, because they have some apparent requirement that all mail gets delivered, that they cannot effectively enforce email quotas that would result in non-delivery of email.
Easy Solution... (Score:4, Funny)
Report that all emails are stored in an infinitismally small location that only future, advanced technologies will be able to restore email upon request. Requests will be queued until the technology has been developed.
OpenBSD has a Good Solution: spamd (Score:5, Interesting)
Re:OpenBSD has a Good Solution: spamd (Score:5, Informative)
SMTP Tarpits are another powerful tool (Score:4, Insightful)
Spamd and other means for "tarpitting" the calling SMTP are another great tool to be used in combination with RBLs and bayesian filters.
It's a strategy in layers:
I wonder (Score:5, Insightful)
No one has to or could guarantee anything for email. With the amount flowing because of SPAM the dropped packets must be astronomical.
Won't Last (Score:5, Interesting)
Being the person that blocks spam is a lose/lose situation. They don't understand how bad the problem is when you do your job right. They complain when spam gets through and complain when legit email gets blocked, but don't want you wasting all your time on it.
I predict that this school will be forced back into filtering spam by their students (customers).
[rant]See, 3 years ago, as spam was beginning to get bad, I began filtering spam on the email system I manage. Over 2.5 years, I developed a rather intensive filter, but since I knew I was not perfect, I had to scan blocked email for false positives. It got to the point I was spending 25% of my time scanning for false positives and the boss didn't like that. He also didn't want me to spend time trying to figure out how to set up Spam Assassin. (I'm not a Linux guru. Sorry!) The board didn't want to spend the money on a purchased system and didn't want me wasting my time with spam. They didn't think it was a problem so they told me to just stop blocking spam. My boss told them that spam was a BIG problem, but they never saw it so they didn't believe him. I asked my boss 10X "Are you sure you want me to stop blocking spam? They won't like the results." He confirmed. I stopped blocking spam and about 50,000 additional spams per week came flooding into the system. The 50,000 were what was being blocked previously. I was flooded with phone calls until everyone realized what was happening. Then, just 2 weeks ago, I was instructed by the board to go back to my filtering, but only spend 30 minutes a day on it. RIIIIGGGHHHHHTTTTT! Ever try scanning for legit email among the trash, adjusting filters to make it better and taking calls and emails from people that want you to be sure an email is blocked and only spend 30 minutes a day on it? I managed to put together a Spam Assassin box and it blocks 10,000 per week, but there's a lot that doesn't get blocked. I don't know enough about it to make it better.[/rant]
dsbl.org (Score:4, Informative)
-John
less centralized servers (Score:4, Informative)
It's the same reason users of major ISPs are more likely to be probed for vulerabilities.
I've found the method of filtering based on the "Click-Me" domains to be the most effective with virtually no false positives (zero is a realistic number).
I've found that setting up a secure public mail system is cake. Mercury Mail is free and handles well. A single check box set by default is all it takes to keep it from being an open relay. Students of the university could probably do rather well offering their own e-mail services to students. Mercury Mail's filtering system is quite robust.
MM supports IMAP/POP3/SMTP and alternate ports as well as SSL on all them. Adding a web-based front end also isn't that difficult if you know what you're doing. There's actually one built in and a more robust version coming.
I already have a few hundred users on Indie-Mail [icarusindie.com] and the amount of bandwidth used per day is pretty negligable.
Ben
Perhaps they need some Canadian help... (Score:5, Informative)
The measured UBEs over a 3 moth period were 172,887 - only for their top-25 most spammed employees!
Centralism has its costs (Score:4, Insightful)
Take a university that has thousands of people actively using email, and thousands of computers, probably a hundred of which function as mail server. Now, decide that "we need a central mail server to filter viruses and spam". Take a few useless machines lying in the computer center, and make them the mail server that's supposed to replace the hundred you had previously. Then slow down the new mail server by applying every concievable virus and spam filtering.
What do you get? Incredibly slow service (sometimes mails get stuck for hours or more in the queue), single point of failure, and officially-mandated false positives (noone in the university can avoid them). AND, you still get a lot of spam.
Computer centers must know that if they want to centralize a service that was previously decentralized (different departments and individual running their own mail servers and filters), they must be prepared. Prepared to handle the load (Google had to buy 100,000 machines to handle their load!), prepared to handle the humans who use their service, and prepared to handle exceptions (a person or department that doesn't want the centralized filtering). Often, these computer centers don't think of these issues in advance, causing things like described in this article.
Re:First Post (Score:5, Informative)
1: They refused to use blacklists to cut the load.
2: They refused to publish SPF records and use SPF to block all the email forged to look like it's from their domain, significantly cutting the spam load.
3: They used one of those "commercial-grade" virus/spam mail scanners that's designed to use entirely Bayesian scanning without ever setting time-outs on the generated rules, and which was written for "completeness", not speed.
4: They forgot to set up a honeypot machine to auto-block spam domains.
6: They underbudgeted for the servers to actually do the mail handling, forgetting to set up up appropriate MX records with good fallover behavior, so when any of their served domain's MX record listed machine blinked that entire domain went offline.
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
No, sendmail (Score:5, Informative)
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
Nah, it's sendmail:$ dig -t MX tu-bs.de
[...]
tu-bs.de. 172738 IN MX 10 rzcomm5.rz.tu-bs.de.
$ telnet rzcomm5.rz.tu-bs.de smtp
Trying 134.169.9.40...
Connected to rzcomm5.rz.tu-bs.de.
Escape character is '^]'.
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 04:00:51 +0200 (METDST)
Re:No, sendmail (Score:5, Insightful)
Re:translation (Score:3, Informative)
I hope they
Re:translation (Score:5, Informative)
No, sendmail (Score:5, Informative)
by marnanel (98063) on Monday May 24, @12:04PM (#9234290)
(http://marnanel.org/)
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
Nah, it's sendmail:
$ dig -t MX tu-bs.de
[...]
tu-bs.de. 172738 IN MX 10 rzcomm5.rz.tu-bs.de.
$ telnet rzcomm5.rz.tu-bs.de smtp
Trying 134.169.9.40...
Connected to rzcomm5.rz.tu-bs.de.
Escape character is '^]'.
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 04:00:51 +0200 (METDST)
Must be using SpamAssassin (Score:5, Interesting)
I have also seen situations where SpamAssassin was not correctly respecting the maximum child spawn limit. Since spamd is a fairly heavyweight process, the server started swapping and throughput plunged.
Such heavy overhead is not a essential part of anti-spam software. Something NOT written in Perl nor any "interpreted" language, something with a smaller footprint, will be much, much faster. I wonder how many people have switched to dspam for this reason?