MIT Spam Conference Conclusions 388
RT Alec writes "The 2003 Spam Conference has concluded, reports InfoWorld. (related read: abstracts of the conference discussions). I was unable to attend the conference, but it appears all that was discussed was filters (client and server). I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25 (you do have permission to use that SMTP server, don't you?). It is not too tough to set up an SMTP server to require authentication, or at a minimum to run off a different port. I am suprised that this is never mentioned as a cure for spam. If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive). I was pleased to see that Barry Shein, president of The World (a Boston based ISP) was included in the talks. I am not sure by the abstract (see link above) posted if he mentioned blocking port 25. In a recent interview he did not mention it."
filters (Score:2)
Seriously, who doesn't block AOL?
Re:filters (Score:2)
Re:filters (Score:2)
Me: How much do I have to pay not to get 3 free hours of AOL?
Checker at CompUSA: That's $86.37.
Me: I actually offered to pay them $10 a month if they'd shut down their Internet connection. But they didn't go for it.
Checker at CompUSA: Really? That sounds like a pretty good deal.
My conclusion is that CompUSA is hiring slightly sharper people than they used to.
aol addresses (Score:3, Insightful)
The problem is that they use an AOL connection to get online, then spoof through a korean SMTP sever.
I like the idea. But, also do it for most of the dial-up services. Cable and DSL does provide a way back to the spammer's home.
Re:filters (Score:2, Informative)
I think AOL is really being blamed for a lot of spam it shouldn't be. Lots of common spamware forges aol.com at various places in the headers. Real mail sent through an external mail server while signed onto AOL has an "X-Apparently-From:" header inserted by AOL. That header contains the actual AOL screen name of the account being used to send the mail. Ergo, AOL isn't really a good choice for spammers to begin with.
-MFS
AOL the source? I think not. (Score:5, Informative)
Send spam using AOL's e-mail client and your account is nearly-instant toast, thanks to automated rate-limiting software.
AOL set up rate limiting sometime around 07/98 [google.com]. Yes, it was THAT long ago. Note, as another poster has said, this wouldn't stop someone from using AOL as their ISP and connecting to another SMTP server for spamming purposes, but considering how slow (not to mention expensive) AOL-provided net access is, I doubt any real spammer would use it for even that.
Since most of the
I don't. (Score:3, Interesting)
I really don't like the idea of ISPs blocking ports. That should be the responsibility of the end user.
Instead of blocking ports why don't they force users to sign an agreement that they won't send spam and if they do they'll pay each recipient $50/incident.
Then if a bonehead sends spam they can go after them and enforce their TOS. I believe AOL requires a valid credit card number to even do the free trials, but I'm just guessing.
I tried blocking ports. (Score:2)
I use Charter for cable internet access. The day they start blocking ports is the day I leave them.
Re:I don't. (Score:2)
The fact is, almost all ISPs have anti-spam provisions in their contracts (even SpewSpewNet [uu.net].) Deleting an account is easy; they'll just signup for an other one. Fining them is easy, in theory; in practice, good luck getting a spammer to pay up. Cleaning up after the fact is difficult and time consuming.
In retrospect, I'm gonna blame the sales people who are too stupid or too blinded by their commision check(s) to realize they are selling an OC3 to a spammer. Really, how much spam is done by dialup these days? They either use broadband or pay some nuts to spam for them.
Re:filters (Score:4, Insightful)
Antivirals! (Score:2)
Re:Antivirals! (Score:5, Informative)
I had to stop using Eudora [eudora.com], because I had so many filters (400+) to kill my spam that it took, literally, 5 minutes for my mail to appear in my inbox, which, needless to say was very frustrating and annoying.
Anyhow, I have been using Spamnet for about 7-8 months and, depending upon the time of day that I check my email it correctly blocked between 60% - 95% of my spam.
For example, since it is a peer based spam detection system [slashdot.org], so the more users that vote that email from a particular sender is Spam, the more likely you will get it blocked. Eventually, it maps out and makes blacklists based on overall stats.
The point is, I took 2 days off for Xmas and when I checked my mail on the 27th, it filtered out about 295 of about 300 spam messages.
Re:Antivirals! (Score:2)
Spamming vs. sending legit mail. (Score:5, Interesting)
Are you saying that if I want to run my own mail server, I should get in touch with the mail admins of every single mail server of everyone I might ever want to send an email too so that I can send it on another port?
That's ridiculous. I shouldn't need to subsidize MX providers.
Otoh, a good solution might be traffic shaping, or even a sort of intelligent traffic shaper that limits the number of actual emails per day.
Personally, I think SMTP is just obsolete. Schlepping anti-spam mesures onto it is like trying to put copy protection on CDs. It's just not going to work. What we need to do move to new protocols. Ideally two separate ones. one for personal mail, and one for commercial/bulk mail. The personal system would make it difficult to send out tons of mail, but easy to get into people's boxes, while the commercial system would make it hard to get into the box (i.e. you need to be pre-authorized) but, by definition, you could send out as much as you want.
Digital certificates and encryption would be helpfull, for one thing
Re:Spamming vs. sending legit mail. (Score:2)
It has nothing to do with contacting every other mail server and everything to do with j-random dialup ISP not allowing mailservers on 56K modems.
It's entirely logical and doesn't involve any changes to the protocol at all. And it would put a huge damper on spammers' abillity to scan for and exploit servers off in some backwater country.
I mean really.. what logical reason do AOL and friends have have for allowing customers on a $10/month disposeable account to connect *directly* to other people's mail servers?
Re:Spamming vs. sending legit mail. (Score:2)
It doesn't seem fair that I should have to pay for the actions of some spammer who uses the same ISP as I do. It would be like if I wasn't hired on the grounds that I graduated from the same school as some former employee who hurt the company somehow...
Re:Spamming vs. sending legit mail. (Score:5, Insightful)
I work for a small company that offers web hosting. Along with the web hosting, we give the customer mail accounts, with SMTP, POP and IMAP access. We have had numerous complaints from customers that were unable to connect to the SMTP server because thier ISP blocks port 25. Why shouldn't they be able to connect to any server they like? This is certainly legitimate traffic but it is being blocked because some jackasses send spam and other jackasses run open relays. Why should my users be blocked because of the actions of other users?
All I want from an ISP is an unfiltered network connection. Once the ISP starts filtering the service it is unlikely to stop. What is the next service to go? Surely people don't need to connect to IMAP or POP servers that are not on the ISP's network. Block 110 and 143. Better block 6346 while we're at it, as it cuts into the pocketbooks of our partners. Don't forget 22, it allows people to work on VIRUSES without the ISP being able to detect it! Pretty soon the network connections ISPs provide will be nearly nonexistent. Port 80 will be open to sites on the whitelist, and you can get a connection on 443 to sites that have registered with the ISP (and paid their tax to Verisign) but all other ports will be blocked. After all, why would anyone need to connect to any service that is not web-based? As everyone knows, 'the internet' == 'www' and connections to other services are not needed.
If I pay for internet access, I don't think it is unreasonable to expect access to all available services. Instead of harrassing the ISPs into degrading my service, how about harassing the mail server vendors to produce products that connot be configured as open relays?
Re:Spamming vs. sending legit mail. (Score:2)
What's needed is some kind of "digital postage stamp", voluntarily issued among ISPs and users (not the postal authorities, so please don't bring up mythical "Bill 602P"), which has to be there before mail gets relayed or, more importantly, *accepted*. No stamp, no receipt. Every retail ISP user will get hundreds of stamps a month, and bulk users can buy them (say, for a corporate email gateway) by the myriad, for something in the penny order of magnitude. That wouldn't be noticeable to anybody but a spammer, who depends on extreme volume.
The trick is to make it work securely without too much of a performance hit.
Nah (Score:2)
You have a Certificate Authority, say your ISP, VeriSign (gag), Me, whoever and when you send an email you digitally sign it, and send a copy of your public key (to verify), which in turn has been signed by the CA. If I trust the CA, then my mail server will accept your mail. Otherwise, bouncy bouncy...
If a CA gives out a lot of certs to spammers, they'll get taken off the list of valid CAs.
Re:Spamming vs. sending legit mail. (Score:3, Insightful)
This will stop most luser spam, because most lusers don't have fixed IP internet connections. Whether it's an idiot running an open poxy or a moron who responsed to an ad in the Weekly Saver for "MAKE $75/HR WITH YOUR COMPUTER!", at least this will get rid of the harder to trace stuff.
The real problem is ISPs that just don't fscking care. The ISPs who would go out of their way to block port 25 for fixed IP customers were probably not the ones with much of an outbound spam problem in the first place.
Re:Spamming vs. sending legit mail. (Score:2)
I suggest that 90% or more of people who read slashdot from home are on cable/DSL modems with non-fixed IPs.
The solution here isn't to just block ports and pretend there's no problem, but to enhance or re-create the mail protocol so forged message headers addresses are impossible so spam filtering can be more effective.
In the meantime, having ISPs introduce statistical filters on email to block spam would also be a great benefit.
I've recently installed POPFile on my system, and with a couple weeks of training, it's in the 97% accuracy range blocking spam. Another couple weeks and it should be nearly perfect.
Re:Spamming vs. sending legit mail. (Score:5, Insightful)
This will stop most luser spam, because most lusers don't have fixed IP internet connections.
Oh, that's nice of you to pass value judgements based on people's IP addresses.
I am not a "luser" (I have probably forgotten more about computing than you know), but I have a dynamic IP address simply because I don't feel like giving ATTBI another $50/month to get a static one. I also have a reason to send mails out on port 25 - I don't use my ATTBI e-mail address, I use my business one. Thus, I send my e-mail through my company's SMTP servers. I certainly have permission to do this, and a legitimate reason, so why should I be punished? I also run an SMTP server (authenticated). Sure people try and send spam though it (every day my syslog is full of Relaying Denied messages), but they fail. When they fail, their address gets blackholed (by me), and passed on to all my friends to be blackholed too).
Now, if what you meant to say was "port 25 blocking should be instituted for people on dialup addresses", I might be slightly more inclined to agree with that. There's a lot less accountability with dialup (read: modem) addresses (due to free trial accounts) than there is with cable or DSL. AT&T Worldnet, for example, drops any outgoing packets on the floor destined for port 25 on a machine other than mailhost.att.net Most of the relay attempts I see in my logs are from dialup pools.
So what is the solution? Certainly any time you institute a widespread "solution" (blackholing, port blocking, etc), innocent folks are always going to be punshed. There's lots of chatter about creating a new protocol, but guess what? If it ain't supported by Outlook, you're SOL. Whether you like it or not, no ISP is going to switch from SMTP to a protocol that will alienate a large portion of their clients. And, guess what, MS isn't going to switch from SMTP. Why? Well, at the spam conference, they said they had found the perfect algorithm to filter spam. Of course, they declined to tell us what it was...
Re:Spamming vs. sending legit mail. (Score:3, Insightful)
However, I currently do need to reconfigure the mail server because some lame ISP does block port 25, so I have to use their mailserver (which, naturally, I can't access if I'm not using their connectivity).
Port 25 filtering is an idea I've only come across recently, and appears to affect a lot of legitimate use without bothering spammers who use lax ISPs anyway.
The people who make money sending spam will pay to get to ISPs who will allow them to do so, but legitimate private users are greatly inconvenienced by ill-informed choices such as interfering at the level of packet filtering in what is a high-level protocol problem.
Re:Spamming vs. sending legit mail. (Score:2)
The best anti-spam method I've seen, bar none, is a friend of mine's opt-in method. His filters indicate the email addresses of people whose mail he's willing to accept, and dumps the rest in his spam folder. Just like call display- the messages that matter get to him and the junk gets junked. For personal email, I think this is the answer-- people with important personal things to say don't rely on email to do it, so if you miss a few, that's ok. Business-related email, is of course, another story, but I figure if SPAM were really a problem for businesses, it would already be solved.
Re:Spamming vs. sending legit mail. (Score:2)
The fact that you don't figure spam is a problem for business shows how little you know about this subject. I've seen estimates indicating that spam costs US businesses alone a few billion a year (in lost time reading the spam and in server/network capacity to store and receive the spam).
Re:Spamming vs. sending legit mail. (Score:4, Informative)
I hope your friend isn't on a mailing list and ever wants help. If people reply directly to him, they may directly land in his spam folder. Ok, I'm exaggeriting, this can be solved with filters also.
A very annoying method people use is filters which auto-reply if your email is not in a positive list, giving you instructions how you should resend your mail.
You sometimes get these messages when replying to list-messages and cc'ing the original sender. Since I'm not on this world to accomodate these people's mail-filters, I just killfile them.
Oh please don't do that. (Score:5, Interesting)
I'm already annoyed at being collatoral damage in the war against SPAM. I use mutt as my e-mail MUA, which is not an MTA and doesn't support use of an SMTP server. No problem; use sendmail or exim on my macine to actually *send* the mail. Except that I find out that some of my mail is bouncing, because my cable modem is in a blacklisted range (the range that includes "all cable modems"), and therefore being rejected by some SPAM filters. I don't run an open relay, I'm just using a program to send mail from my computer in the way that it is designed.
Very annoying.
So I have to configure my MTA to forward to a gateway SMTP server which won't be on the various RBL lists. A pain, but fine, I can do that. I've managed to get that set up... but I'm not using Comcast's SMTP server. Maybe I should, but after briefly using @Home's mail services, I've leanred simply not to trust the cable modem ISP services for anything. I've got web hosting outfits I pay for, so I can use those SMTP servers, configuring my exim to forward to them and use SMTP AUTH. But if Comcast starts blocking port 25, then *that* won't work, and I'll be stuck again. (And, of course, "getting another ISP" isn't an option, because where I live, the cable company's got a monopoly as far as broadband access goes. I *do* have another ISP I pay for for things like news and mail, on top of the cable modem. But, unlike where I used to live, I don't have the option of going with DSL and choosing the ISP to use with it.)
Let's please not put forward this idea. There's enough collatoral damage as it is. And it won't really cut back on the spam, either. It's very very fuzzy logic to assert that since 50% of the spam now comes from AOL customers, that shutting that down would cut spam by 50%. The spammers out there will just find other places to spam. Going after the spammers themselves, and not just some of the tools they use, is the only way to stop spamming. Anything else only temporarily inconveniences them, and meanwhile greatly inconveniences innocents.
-Rob
Re:Oh please don't do that. (Score:2)
say you get 100 transmissions per hour on pt 25 without penalty; then any more than that are allowed at 50% reduced bandwith, and the next 50 at 75% reduced, and so forth; this would make spam all but impossible over 25 ( I don't mean limiting point to point connections, but ALL connections on 25 would eb considered in aggreate)
If you need to send bulk mail (mailing lists, and so forth) you should be connecting to the network which hosts the SMTP anyway - or do it via a VPN setup. There really isn't a reason to be transmiting traffic across a wan in bulk - personal use (which would rarley exceed 100/hr, and if it did, wouldn't be hurt by the 50% slowdown.)
I agree that penalizing acceptable use to fight spam is worse than spam itself - but this seems like a good idea to fight the problem without creating more problems
Re:Oh please don't do that. (Score:2)
I do however offer a suggestion I've seen that might actually work out, however according to the person who started this thread it might not directly for him:
idea 1) force all smtp servers (recieving) to query back to the original sender of the e-mail to confirm that the user exsists on their system
note: this isn't perfect it might work, there is a good chance it doesn't though
idea 2) reject e-mail who's sender 1) doesn't match the domain it's coming from or 2) doesn't have a fully qualified domain
THIS should stop a lot of spam, as a lot of it will fail on one or both of those. I have been running into this more and more recently, and am going to install filters and such that match this
idea 3) arm everyone with shotguns, have them spy on their neighbours. If they find that their neighbours are spammers, they must shoot the kneecaps off their neighbours and bring them before the world for trial. if they are found guilty..... may the world have mercy on them. If not they get free knee surgery to have the poor things replaced and all.
this would work maybe... except that most e-mail comes from Asia, and their neighbours probably do it too... ohhh well
but seriously idea 2 really would stop a lot of the spam, look through the headers and you will see what I mean.
Also isps really need to take action against spammers, this is one of the reasons it persists. Talk to you ISP, tell them what you think.
Re:Oh please don't do that. (Score:4, Insightful)
I don't run an open server (I test this whenever I make any significant changes to my configuration) and certainly don't allow spam. I'm so anal that I have a filter that bounces subjects which contain "fwd: fwd:". That caught my mom and sisters a few times.
Since I have a dynamic ip address, I use a service to deal with that (along with a 15 minute cron job to make sure my domain and ip address are synced). Unfortunately, some of the more "religious" antispammers block the entire dynamic dsl range, so there are a few places that refuse mail from me (very rare, fortunately).
Preventing private email servers is just plain stupid. Just because some people are abusing this doesn't mean everyone must be punished. That's the equivlent of saying, "some people print child porn, therefore we must outlaw all private publications."
Re:Oh please don't do that. (Score:2)
No, you're *owned*.
If you were right, there wouldn't be a reason for ISPs to consider blocking 25. But there is. Unrestricted broadband connections are widely used to originate spam. Saying it isn't so doesn't make it true.
.
I don't get it. (Score:2)
Re:I don't get it. (Score:2)
Advantages? You know exactly how many mails each customer is sending so it's easier to detect a spam run and the spammers get a massive reduction in the rate they can send at since the now can't connect to 50 other mailservers and just toss in a large recipiant field per message.
It basically renders your ISP useless to spammers and thus reducing creditcard fraud and the support costs of dealing with spam complaints.
Re:I don't get it. (Score:2)
Re:I don't get it. (Score:2, Insightful)
The ISP does not block port 25 for traffic coming into their customer's systems, they block it for traffic coming out of them.
Their customers must relay their outgoing email through the ISP's mailservers.
Messages relayed by the ISP's mailservers can include header info that ensures that the originating customer can be determined. Then, if a complaint is sent to the ISP, they can decide which customer to deal with.
This only has to be done for customers that use dynamic IP addresses - when fixed IP addresses are used, that is adequate to identify which customer sent the message.
Of course, this will only be done by those ISPs that believe in being a good netizens.
Open Relays vs. Spam friendly ISPs (Score:3, Insightful)
Does anyone have an idea how much spam comes through open relays vs. spam friendly ISPs?
Blocking port 25 is terrible! (Score:5, Insightful)
Saying "oh, just run it on a different port" is not as simple as it sounds to us geeks. Sure, we offer SMTP on another port to get around those ISP's, but your typical nontechnical user doesn't even understand the problem, much less know how to apply the workaround. And during the time they can't send mail, they're blaming you. They're blaming your "broken" mail service, because the mailbox their ISP provided them with is working just fine.
So you set up the nonstandard port and tell them "point it here." Now you're wasting untold amounts of tech support time on the phone with the nontechnical users -- you have to figure out what operating system and e-mail client they're using, and hopefully it's a setup that someone in your tech support organization is familiar with. Then you have to walk them through the process of setting up SMTP on a nonstandard port, and setting up authentication if necessary. During that time, you've spent enough tech support time to make that account unprofitable this month, and the spammers have found some other way to deliver their mail anyway.
Blocking egress on port 25 is not a good solution.
Re:Blocking port 25 is terrible! (Score:2)
It's not that hard.. I've also had to walk non technical customers though that problem and I still wish more isps would block 25.
Re:Blocking port 25 is terrible! (Score:2)
AOL does block port 25 (Score:5, Informative)
AOL does block port 25. More specifically, they highjack all outbound port 25 traffic to go through their own SMTP server (you didn't know they had an SMTP server, did you?), sorta like some web proxy servers intercept all port 80 traffic.
Try connecting via AOL, pull up your favourite piece of software that sends e-mail via SMTP and you'll find that it doesn't matter what SMTP server you specify, your mail will always go through. Look at the headers of the mail when it reaches it's destination and you'll see references to AOL's servers rather than the SMTP server you though you used.
Presumably they have limits on these servers that prevent spamming because I've never seen them on headers from spam
Re:AOL does block port 25 (Score:2)
Re:AOL does block port 25 (Score:4, Informative)
Of course, the down side is that you can't use AOL to, say, make a secure ESMTP connection to your office's outbound mail server while on the road, and business travel is one of the more popular non-novice uses for AOL. Dunno if they've worked around that yet, sinced I left 18 months ago. Another down side is that there is now no way for a receiving site to distinguish between those who try to send direct-to-MX spam and those who simply fire up Outlook Express while connected to AOL.
If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive)
Examine a little more closely, timothy. It ain't there.
Jay Levitt
Former Chief Mail Systems Architect, AOL
Port 25 blocking is a dumb idea (Score:5, Interesting)
I note that most Windows mail clients don't let you choose a port, at least not easily (in a setup-menu-visible manner). I've just checked Eudora and LookOut Express, two very common ones, but frankly I don't know of any that do. In Unix, sure, a clever sysop-type using classical tools like fetchmail can probably pipe anything anywhere, but that doesn't address everyone either. I also note a need for external servers at times. Not every account even has a mail relay. And some users want to use their office relay (presumably with authentication) or a third-party service. After all, scummy ISPs like VeriZontal Online don't let you put your own bloody address in the From: field, requiring you instead to advertise them, even if you normally use a private or third-party domain! So those users have to find external relays.
Even if it were trivial to pick a port, it would be a dumb idea. Spammers simply find the open relay and open originating-ISP wherever it is. Close 90% and spam falls by, oh, approximately 0%. Only once close to 100% are closed will spam drop off much. It's like Star Wars (the Reagan-Bush defense-contractor-welfare plan, not the movie) -- it's insane to think that blocking an ICBM nuclear attack is workable, because a 90% hit rate will still pass enough bombs to blow everything up.
Also, port blocking at the ISP increases the load on the routers, compared to fast-path switching on IP only.
Re:SMTP Port Selection (Score:2, Informative)
I might not like LOE, but give credit where it is due.
Here's why blocking 25 isn't useful... (Score:3, Interesting)
A lot of people, not too miraculously, made the pertinent remark that blocking a port would only make people move to different ports.
Well, I won't hold judgement on the statement, but you have to realize you are putting yourself in the exact same boat as they were. There's lots of things that aren't too brilliant of such a scheme, mainly the fact that 25 is actually used for something... and also that putting restrictions on 25 will just make branches in standard internet protocols.
Not quite (Score:5, Insightful)
No, the key problem is ISPs that don't disconnect spammers and charge them for violating the AUP, as well as ISPs that don't even have anti-spam AUP's. Open relays are next on the list. True, blocking outgoing port 25 traffic on the routers might eliminate a lot of spam (not a significant amount: in my experience the majority of spams I get are from various Asian countries, though configuring Postfix to reject connection attempts from a dozen or so subnets in China has cut down drastically), but then again, dropping every packet would solve the problem even more effectively, because:
As soon as an ISP blocks port 25, any spammers using that ISP will run their spammachines off of different ports. If an ISP requires SMTP AUTH connections to their mailservers, how long before spammers start relaying through their own ISP servers? Ultimately, blocking port 25 will have no measurable effect on spamming, because if the ISP provides a means around it for sending legitimate mail, it will be abused to send spam. All your proposed remedy will do is make life difficult for those who run legitimate mailservers.
Re:Not quite (Score:2)
If the only way around that is using the isp's mail server the spam suddenly gets easier to trace and the server admins will notice if someone suddenly saturates their servers with a ton of bulk email. Even if they don't it will slow the traffic because spammers usually take as many open relays as they can find and cram them so full they end up not being able to get legitemate email(including complaints) until 4 days later.
Re:Not quite (Score:2)
On the other hand, once you realize an ISP is totally black hat *cough* *coretel* *cough*, you can blackhole their entire IP block and cut off all future spam from them.
They can't change their ports (Score:2)
spambayes? (Score:5, Informative)
While they started out with the bayesian algorithm described by Paul Graham [paulgraham.com] they quickly discovered that the effectiveness of his algorithm tends to depend on the values of some quite sensitive tuning parameters and that diffrent people can get wildly differing degrees of success depending on their configuration and the types of spam/ham that they receive. Gary Robinson wrote an interesting critique [weblogs.com] of Paul's algorithm and helped the spambayes team incorporate his so-called chi-squared combining scheme (which apparently isn't bayesian at all) which doesn't seem to depend so much on 'magic' numbers and their testing framework showed that it works surprisingly well for both small and large sets of messages.
It's still under active development although most of the ongoing work is centered around the user interface components (POP proxies, Outlook plugins, etc...) whereas the actual spam classifier hasn't changed much in a while.
Well worth looking into if you're getting too much spam. Who isn't?
Tarpit! (Score:5, Interesting)
Excellent description of how to use it with your own self generated blacklist at http://www.benzedrine.cx/relaydb.html.
Unfortunately, it's only on OpenBSD so far. Can some one please port this to Linux by tomorrow?
Temporary rejection - but only temporarily (Score:5, Interesting)
NO! (Score:4, Insightful)
It is critical for the future of the Internet that ISPs provide unmolested IP service. When ISPs are permitted to filter anything, for any reason, you start down a slippery slope. As soon as ISPs start trying to prophylactically control what goes on through filtering, they will find new things they need to control, for "security" or "liability" reasons. This will screw the end users by changing the 'net from its current state to a choice of which ISP's walled garden you want to be trapped in -- which ISP's filtering and censoring you want to pay for the privilege of being subjected to. It also screws the ISPs -- technologically it's expensive, it creates new problems for their customer service to deal with, draws the ire of some of their customers and civil liberties types, and the more they try to filter/control/censor, the more ISPs will be legally required to (the principle behind common carrier -- if I provide a neutral and blind service, I can be exempted from being required to control many things, but if I provide a controlled service where I can know what's going on, then I'm required to use my control and knowledge to prevent certain things or I can be held as aiding those things being done)
And it won't stop the bad guys. The worst thing about the spammers is that they're just smart enough that whenever any effective anti-spam measure comes around, they just find a way around it. Yes, AOL filtering outbound port 25 today will stop a lot of spam TODAY. And guess what? The spammers will just do something else. Open -- or cracked -- proxies are the up and coming new spammer tools. Please explain to me how cutting off outbound port 25 solves that problem. Please explain to me why spammers will just go away and stop spamming because you're blocking port 25 as opposed to finding some other way to spam.
This is a solution where the users lose because they lose functionality and are likely to lose more with it as precedent. It's a solution where the ISPs lose because they incur new costs and liabilities while only temporarily slowing down spam. It's a solution where the spammers lose least of all, they've been shut out of ISPs before and they've been blocked in various ways before and they already know how to do their deeds differently if they need to.
This is a really bad idea.
I am disturbed that a bunch of supposedly clueful folks came up with this.
Blocking 25 too heavy handed (Score:2)
A better idea might be a DNS hack. if the reverse lookup of the sending server's address doesnt include mx.domain.tld, require AUTH. It is less of a problem than blacklisting all cable modems or blocking outgoung traffic to SMTP and will do at least as much to kill off spam.
Pointing a domain to your Broadband or dialup address is easy, but adding a PTR record to your ISP's server is hard. Hopping from colo to colo is a lot harder than getting a new dialup every other day.
Re:Blocking 25 too heavy handed (Score:2)
Re:Blocking 25 too heavy handed (Score:2, Informative)
ssh -L25:remotemachinename:25 remotemachinename
Works like a champ. I tunnel my IMAP and SMTP connections this way.
Barry Shein's modest proposal. (Score:5, Interesting)
Barry's proposal for that last point was a fundamental change in the economics of spam, as follows:
Basically, it boiled down to "Spam is currently in a gray area legally, so let's legitimize spam in order to divide the spammers into legal spammers (who pay handsomely for the privilege) and illegal spammers (who do hard time, just like people who cheat a utility company).
Challenging proposal, and great fun to hear him speak.
Re:Barry Shein's modest proposal. (Score:4, Informative)
I also kind of got the impression that he thought the rate for this should be prohibitively high (did he say something like a penny per message, or am I making that up?). The point being, to put a system in place so that you are ABLE to charge for it so the magnitude of the problem is more clearly discernable.
Barry also mentioned many other "features" of spam from an ISP's point of view, not the least of which is that naive people hold their own ISP responsible for the mail they get, which is sometimes pornographic and exposed to children. I don't think he was seriously suggesting ISPs should let this go and furthermore profit from it, but rather that, if they were authorized and able to charge for it, they could flip the spammer's economic model and improve relationships between ISPs and their clients.
Active Spam Killer / TMDA not mentioned (Score:3, Interesting)
Since two months, I've been using the Active Spam Killer (ASK) [paganini.net] now, and this has been mostly successful. In short: If a person writes me an email, they will have to confirm the mail, unless they are on my whitelist or the email contains a magic key (which is included in my sig and will thus be included in a reply). Confirmation also places a person on the whitelist, automatically. Since most spammers forge the From: address, they are not able to confirm their mail, even if they wanted... -> Pretty much no spam (dropped from approx. 20-30 spam-messages per day to 1-3 per week). Sure, if you order a book at amazon, their computer might not confirm. Thus I look into the confirmation queue from time to time whether anything in there is legitimate. Thus far it has not yet occurred that a person would not confirm his/her email, by the way. ASK is well documented, written in python and easy to setup.
There is another similar system (which I haven't checked out): TMDA [tmda.net].
I am wondering why big corporations, universities, ISPs are not providing such a (preconfigured) system as an option in their email packages
Re:Active Spam Killer / TMDA not mentioned (Score:3, Interesting)
Most of them focused on statistical methods, primarily Bayesian ones, actually. And yes, sometimes even a well-trained Bayesian filter will result in a false positive sometimes.
One presenter made an excellent point, though: you can easily say "I've never had a false positive" if you just don't filter very much. So, I'm glad your system hasn't been tagging your good messages as bad; how effective is it at getitng rid of the bad ones, though?
Paul Graham's presentation revolved around a Bayesian algorithm he'd devised which put more weight on features in the headers, as opposed to the bodies, of email; he claimed something like 99.5% effectiveness with only something like 5 false positives in 4000 emails sorted.
The really interesting part was the nature of the 3 false positives that he showed. Two of them were mailing lists that he "didn't care much about anymore," and the other was a note in all caps from a person in egypt requesting some info on one of Graham's academic projects. In other words, they all *did* resemble unsolicited mail.
Recordings of the conference? (Score:2)
Lets get to the meat of the matter ... (Score:5, Informative)
http://spamconference.org/
The talks are online.
Read-Only Internet Access Is Bad (Score:2)
"I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25 (you do have permission to use that SMTP server, don't you?). It is not too tough to set up an SMTP server to require authentication, or at a minimum to run off a different port. I am suprised that this is never mentioned as a cure for spam. If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive)."
In my opinion, this is a terrible idea, for a number of reasons.
The first reason is the First Amendment of the U.S. Constitution. This would inhibit free speech by anyone who wants to send mail to anyone else. You know how you love to have port 80 blocked to your computer, don't you? This would continue the terrible trend of allowing read-only Internet access. You can read all you want, but if you want to upload anything or enjoy the pleasure of having unfettered bidirectional Internet access, you are going to have to pay $10 a month for an IP address, plus a BS charge of $300.
In it's most expensive form, an IPv4 address from ARIN costs about 7 cents per month. Granted you have to buy in bulk, but all ISPs do. So why can't you have a routed allocation if you meet the requirements for BCP12/RFC2050? Network operators are lazy and arrogant -- I know, I used to be one. I used to be an engineer at Global Center and GlobalCrossing.
It is absolutely not an ISP's responsibility to filter packets or frames based upon any protocol or service -- that is your job. Furthermore, most Internet routers simply could not perform with such requirements. If you want to pay your ISP to waste clock cycles and memory to block ports for you, you may ask. Or maybe just you could just get a firewall instead.
The reason that your suggestions are never mentioned as a cure for spam is because they would not work.
If you want to isolate yourself from the Internet and prevent yourself from ever being able to run your own DNS, STMP, HTTP, IMAP, and other servers off of your Internet connection (like I do), you may do so upon your own discretion. But please don't give the (dis)service providers any new ideas. Things are bad enough as it is.
Re:Read-Only Internet Access Is Bad (Score:2, Informative)
Sorry, the First Amendment says CONGRESS shall pass no law... A private ISP can restrict your speech as much as they want when you use their service (within the bounds of contracts, etc).
That's great, if I can... (Score:3, Insightful)
It IS a terrible idea; if you want to offer a public data service, then that's what you offer. You don't get to make exceptions just because you feel like it, unless you are declaring, in essence, that you are providing the service of selectively restricting traffic. And in that case, you become liable for every judgement you make about who to service and who not.
A bar/pub/saloon can restrict you all sorts of ways just because they feel like it. But this doesn't give anyone the right to stop you from getting drunk, trying to pick up strangers, or making a fool of yourself in public. A public communications service is different, and for a very good reason. bars and saloons are primarily there to provide a space for private associations; a communications infrastructure is there to provide a public infrastructure. and the internet points this out very well; it's public, accept the fact or build your own fucking internet.
It comes down to this; you are advancing the idea that the primary argument is "it's mine, i can do whatever i want with it". but in the interest of creating a just society (one where few people have an interest in destroying it), we recognize many "level playing field" exceptions to this. separate water fountains, "whites only" policies, etc. tell me who and what you are and i'll tell you how you depend on this fundamental fairness. i'll also point out that the internet isn't yours and if you can't play by its fundamental rules of openness then you have no business connecting to it.
Re:That's great, if I can... (Score:2)
Also, you have to consider - are they restricting speech in general, or are they restricting the content of speech? Forcing you to make less than 15 calls a month could be considered a restriction of speech, but again is not an abridgement of that right. Preventing you from complaining about the RIAA and the MPAA would be a restriction of content and thus a violation of their common carrier status.
Spam tracking (Score:4, Insightful)
Re:Spam tracking (Score:2)
I do something similar albeit a bit fancier. But I learned something though: Just because you use a different address with every place you go doesn't necessarily mean that all the junk mail you get there is the fault of the place you signed up with. Your email can be posted somewhere on the web and it'll somehow get captured.
I did an experiment on Slashdot where I made my email address available without 'spam armor'. Before long, I had all KINDS of unsolicited mail. However, I do not believe
I learned a lesson in doing this. I came down hard on a dude once because I got an unsolicited mail from him. Turns out, somebody 'volunteered' my address to him. (The email wasn't a solicitaion, it was a notification... it'd make sense if you saw it why it wasn't SPAM.)
So I guess my point is: be careful if you decide to give anybody shit over it.
Anti-spam (Score:4, Interesting)
Your email provider delivers an email to you only if
it has a "Reply-To" field in the header AND
the Reply-To value has been accepted as a valid email address by another customer.
So in order for a person that just created an email address to email you, they would have to get their new address validated first and would receive a message to that effect the first time they tried to email you. They would have to get in touch with you or someone else under your email provider to get validated.
If you get some spam, you report it to your email provider and the ISP deals with the customer who validated the "Reply-To" address.
Email providers would set up peering relationships wherein they can share validated email addresses.
If the Reply-To value is faked, it would have to point to a validated email address and would probably bring severe damage to that email account. This method would push spammers into using this strategy, but it would certainly get them into more trouble that they currently get into.
I'm sure there are holes in my idea, so shoot away and educate me.
Yeah, blame the ISPs... (Score:5, Insightful)
- I think the key problem is ISPs that do not block egress traffic on port 25
And think a big part of the problem are the nuts who think filtering port 25 network wide is a viable option. Here are some real world numbers...Router #1:
30 second input rate 21782000 bits/sec, 6210 packets/sec
30 second output rate 12294000 bits/sec, 4651 packets/sec
Router #2:
30 second input rate 7543000 bits/sec, 2133 packets/sec
30 second output rate 12182000 bits/sec, 3183 packets/sec
(and that's business traffic at 0030ET Sunday -- it goes a lot higher during business hours.)
Routers have a lot of work to do already without having to look for spam. Devices along the lines of a Packeteer could be used to perform in-line packet inspection, but that'll get old real fast.
Yes, it's perfectly doable to filter dialup users either at the ppp line or the next hop router by either explicit blocks or redirection. Many ISPs already do this. (UUNet requires it, oddly enough.) But an equal many don't. Plus, there's a growing amount of broadband in the world.
Most companies buying network connectivity and hosting their own email systems expect them to have direct control over those systems and the routing of their email in both directions. It's a simple task to set a mail server to use a "smart host", but then one is at the mercy of those controlling that server(s).
Oh, and just how exactly will this stop them from sending spam? Exactly. Simply put, it won't. It just changes the origin of the spam and maybe speed up the response time for blocking it and dealing with the user. HOWEVER, it introduces a much larger annoyance: blacklisting of the ISP server(s) and thus hundreds or thousands of companies and/or users.
Next I suppose the ISP should be looking at the email to judge it's spamliness? Well, I'm gonna have to play my lawyer card on that bit of stupidity. The instant an ISP begins any type of content filtering, most of the protective provision of various laws cease to apply. In the eyes of the law, this would be exactly the same as the post office opening all of your mail to determine and discard what they feel is "junk mail".
In the end, spam is what it is because of the [censored] creatans who think they can make money by participating in any of a growing number of scams. Basically, technology cannot protect the internet from stupid people. (esp. when the standard was constructed in a "stupid people" void. I guess we've bred better idiots.)
If anything (Score:3, Funny)
Eventually, if spam is allowed to proliferate, we will all live in a world with lower APR on our credit cards, countless anonymous women in love with our cocks are that have grown 4" bigger guaranteed.
Enough of this conservative conspiracy.
On a serious note, I hate arbitrarily blocking ports. It won't do shit to stop spam, it's more about the ISPs wanting to block all the ports possible, to reduce the amount of traffic an end user can have.
powerful penis enlargement lobby (Score:2)
The PEL you say? why of course the Penis Enlargement Lobby! Read this: Anti-Spam Legislation Opposed By Powerful Penis-Enlagement Lobby [neurofractal.org]
An argument FOR egress blocking port 25 (Score:3, Insightful)
Port 25 egress blocking is a good start to the spam problem for two reasons: First, it prevents a spammer from signing up and just doing direct-to-MX spam from that throwaway account. Not many spammers do this anymore, because its easily tracead and bigger ISPs kick those accounts fastest. Second, it limits a spammer's ability to abuse open proxies and relays on a network. Say clueless users are running a WinGate open proxy or an open sendmail relay on an older default Linux/BSD install on their cable or DSL line. A spammer could try to relay spam through it, but the egress block would stop it.
I see alot of complaints here about how such a block prevents you from running a mail server on your broadband line. People, this is residential service you are getting here. If you need to run your own mail server you need to find out about that when you sign up for service. A typical residential user never needs to connect to any SMTP relay except the ones the ISP provides. These users are also more likely to cluelessly leave their computers open to abuse. If you're responsible enough to run a mail server, and you really NEED one, get a real account.
Another option is to relay your mail over a non-standard port through a third-party email provider, if you really loathe your ISPs relays. This is my situation, and I use Lux Scientiae [luxsci.com]. They run a SMTP AUTH relay on a secondary non-standard port. It's locked down to prevent abuse, and SMTP AUTH lets them track down any of their users that abuse it. They don't accept incoming mail on that non-standard port, only relay for users, so it's not like they're re-defining SMTP to use a different port.
Of course, there will always be those ISPs that really don't care about preventing abuse. This is why blocklists even exist, to allow users to shut out the bad neighborhoods on the net. It would be nice if all those residential broadband users' computers couldn't be hijacked by spammers. As it stands, they are, so one way or another port 25 traffic is blocked.
Naive Bayesians probably don't work in long run (Score:3, Interesting)
a) short messages don't get caught- no words that are going to be blocked, just a URL. The URL doesn't match because it's several words stuck together without spaces.
b) misspelt words don't get caught. If the spammer deliberately misspells the key words, then it goes through.
c) common words- if the spammer only uses common words, it is unlikely that the spam can get caught; the spammer can check all the words he uses for being common before he sends it.
d) pictures- if the spammer sends his advert in a GIF, the Naive Bayesian can do nothing.
Overall, I am pessimistic about whether filtering will work in the long run, but in the short run it works pretty good.
Re:Naive Bayesians probably don't work in long run (Score:2)
Something *slightly* different (Score:3, Interesting)
As long as the mail server doesn't do anything more agregious to the mail than add a Received: header, I find it unlikely that any legitimate complaints could be made about this practice. It's certainly a much more gentle answer than simply blocking port 25 egress completely. At least this way it's more or less invisible to the end-user.
You can't fix all ISPs, but their users can. (Score:3, Interesting)
I'd propose a slight change to SMTP servers so that they automatically block incoming mail from other servers that act as an open relay. It would not discriminate against open relays when sending mail, however.
What this does is effectively drops all users of open relays off the map. Once enough servers out there start doing this, all the open relays start getting fixed, because their users demand mail to stop bouncing. Open relay spam ceases to annoy everybody behind a protected server immediately, however, and you don't really care when or if those servers get fixed.
This isn't going to fix the general spam problem, where valid addresses are used for spam, but at least you can block domains that annoy you.
But the truth is, spam will never calm down until every unsolicited/untrusted message costs a nominal sum, which curteous people return in the form of a reply from valid messages.
The real solution is to do the opposite (Score:2)
Here's what you do instead: you configure all the email servers to take the FROM address specified in the SMTP exchange itself, then look up all the MXes for the domain the FROM address claims to belong to, then compare the actual address the connection is coming from to the list of addresses you just got back. If you don't get a match, you drop the connection right then and there.
End result: anyone who is running their own domain or who is using a legitimate mail server is able to get through, and nobody else is. Suddenly most open relays become totally ineffective. Spammers now have to go to the trouble of acquiring a domain and setting up MX records, and if they don't have a static IP then they'll have to use a dynamic DNS service. End result: killing a spammer is as simple as telling their dynamic DNS service to shut them down.
If there needs to be a way to differentiate between email receivers and email senders, then define a different type of MX record for email senders and do a lookup on them as well.
Thoughts?
Re:The real solution is to do the opposite (Score:2)
I think the parent post could have been worded better, as I get the impression he wasn't talking about the From: header that mailers add, but the actual SMTP connection sequence (HELO and all that).
New mail protocols needed (Score:3, Interesting)
Legislation is not the answer. We know how tech-savvy politicians are. Do laws stop corrupt CEOs from plundering corporate pensions or cooking the books? Do laws solve problems?
Terrorizing spammers is not the answer. Again, this is not solving the problem. Pestering less than intelligent people who exploit less than intelligent methods of mass communication does not solve the problem. It might be a thrill short term, but there are too many people who will spam if the current mail protocols persist.
So what is the problem? Strangers send me e-mail I don't want. What is the solution?
I won't pretend to be an expert. I'm not. However, I'm surprised better men and women have not come up with something, ANYTHING, to solve the spam problem. I am NOT suprised to see 90-100 unsolicited e-mails (from strangers) in my inbox every day. Somebody needs to come up with something. So here goes...
First, classify e-mail accounts. Home/personal accounts should be bulletproof. You only receive messages from people you have on your list of acceptable senders, your "inner circle." Shopping/e-commerce accounts: you can receive messages from merchants who register with some central agency/server. Business/work accounts: I dunno. Ideas? How should we handle mailing list type accounts? Second, every e-mail sent has something solid identifying it with a sender included. The identification is sent to the recipient. If the recipient has this identification in his list and it matches 100%, then the recipient fetches the message from the sender. So instead of the sender wielding the power, the potential recipient makes the call. Why allow just anybody to send an entire friggin' message to scores of people? Messages go no where until the recipient says so.
Finally, and this is where the law comes into play, if someone manages to fake out your list by saying he is someone he is not, sic the prosecutors on him. That's identity theft, pal. As it is now, e-mail headers are raw schitzophrenia.
So step one, classify e-mail accounts. Different classifications have different list of people you are willing to accept mail from. Step two, the sender sends his identification and maybe a subject header to the recipient. Step three, the recipient accepts the senders request and fetches the message himself, rejects it outright, or adds the sender to his list and fetches the message.
I don't know 90 people whose mugs I'd piss on if they set themselves on fire. Why should any of these rat bastards be able to dump a second or third bit in my inbox?
Web of Trust (Score:2, Interesting)
Someone using a signature for spam would see himself removed from the web of trust, and those that verified the person as a non-spammer.
Just don't ask me how somebody that doesn't know anybody else with an e-mail account gets somebody else to vouch for him. (Maybe your ISP will vouch for you if you verify yourself with a CC or something?). Any thoughts?
Faulty Logic (Score:2)
If it did reduce spam by 50%, it would only be for a very short time. The vast majority of spammers that would be foiled by this would simply find another way to send their spam. So in the end you'll have pretty much the same amount of spam, but you'll inconvenience all of the people who had legitimate reasons for using port 25. Doesn't sound like a great idea.
My notes for the proceedings (very long post!) (Score:5, Interesting)
Turning my notes for the day into something vaguely coherent, here are some hightlights from the proceedings. There are a couple of speakers that I didn't write anything down for, but from mid-morning on this should be pretty comprehensive. Apologies in advance if my notes lead me to attribute certain comments to the wrong speaker -- if anyone notices any mistakes please feel free to add corrections:
Bill Yerazunis - CRM114 & MailFilter
Because Perl "freaks him out", Yerazunis came up with the CRM114 minilanguage (points for anyone that gets the joke in the name without googling for it :), then wrote MailFilter in CRM114 as an implementation of a filter that can be used with Procmail or SpamAssassin or what have you. The basic idea is to decompose a message into a set of "features" composed of various permutations of single words, consecutive words, words appearing within a certain distance of one another, etc, such that the set of features N is very much bigger than the set of words X. You then analyze the features in various ways and if you get above a certain arbitrary threshold, you flag the message as spam & handle it accordingly.
He claimed that with this software he could get better than 99.9% accuracy in nailing spam, and a similar percentage in avoiding "ham" (the term everyone was using for false positives -- legit mail that was falsely identified as spam). One of Yerazunis' observations is that the best way to defeat the spam problem is to disrupt the economics: if a 99.9% or better filter rate were to become the norm, then the cost of delivering spam can be pushed higher than the cost of traditional mail and the problem will naturally go away without requiring legislation (which would be nice anyway, but we can't count on it).
The drawback of CRM114/MailFilter is that it can only handle about 20k of text per second, so it's not appropriate for large scale use yet. Still an interesting project to watch though: crm114.sourceforge.net [sourceforge.net]
John Graham-Cumming - POPfile
Most of his very entertaining talk was about the ingenious tricks that spammers resort to to obfuscate spam against filters, including most diabolically one example that placed each column of monospace text in the message into an HTML column, so that the average HTML-capable mail client would render the message properly, but it would be absolute gibberish to most mail filters. The ultimate lesson was that any good filter has to focus not on "ascii-space" (the literal bytes as transmitted) but the "eye space" (the rendered text as seen by the user), which by extension may mean that any full scale spam parser/filter could also have to include a full-scale HTML & Javascript engine. Yikes!
As for Graham-Cumming's software, it's a Perl application, available for all platforms (Windows, Mac, & of course Linux) that allows users to filter POP3 mail. Interesting stuff if you're a POP user: popfile.sourceforge.net [sourceforge.net]
John Draper - ShopIP
Most of Draper's work seemed to be focused on profiling spammers, as opposed to profiling spam itself, by throwing out a series of honeypot addresses & using data collected to hunt down spammers. spambayes.sourceforge.net [sourceforge.net]
Paul Judge, CipherTrust
Judge's big argument, which no one really disagrees with, is that spam has become not just a nuisance, but an actual information security issue. To that end, he is advocating much more collaborative effort to address the problem than we have seen to date: conferences like this, mailing list discussions, better tools, and public data repositories of known spam [and ham]. To that last point, one of his observations (which others made as well) was that there are no universally agreed on standards for what qualifies as spam, so repositories for spam will not be accurate for all users (spam for your programmers will be the bread & butter of your marketing department, etc). Plus, there are obvious privacy issues in publishing your spam & ham for public scrutiny. And to add another wrinkle, one danger of public spam/ham databases is that spammers can poison them with false data, screwing things up for everyone. That said, he encouraged users to help out with building spamarchive.org [spamarchive.org].
Paul Graham
The man who organized the conference and kicked everything this week off with his landmark paper from last fall, A Plan for Spam [paulgraham.com]. Graham's spam filtering technique famously makes use of Bayesian statistics, a technique popular with nearly all of the speakers. The nice thing about a statistical approach, as opposed to heuristics, simple phrase matching, RBLs, etc, is that they can be very robust & accurate; the down sides are that they have to be trained against a sufficiently large "corpus" of spam (most techniques have this property though) and they have to be continually retrained over time (again, this is common). Graham was too modest to produce numbers, but subjectively his results seemed to be even better than what Yerazunis gets with MailFilter, by an order of magnitude or more.
Like other speakers, he predicted that spammers are going to make their messages appear more & more like "normal" mail, so we're always going to have to be persistent about this -- as one example, he showed us an email he received IN ALL CAPS from a non-English speaker asking for programming help, and although it was legit, the filters insisted otherwise. "That message is the one that keeps me up at night."
Everyone interested in the spam issue should go read Graham's paper immediately.
Robert Rothe, eXpurgate
Rothe works for Eleven, an ASP company from Berlin selling a spam management service/application called eXpurgate. His talk was short on details about how the tool worked (mainly that it searches for bulk mail), focusing instead on the high level functionality it provides to users -- basically, they classify mail as safe, questionable, or dangerous, and let the users handle them accordingly. Another speaker that sees spam as a network security issue, so they built their system accordingly, with privacy of the client's mail content in mind etc.
Like many speakers, he warned about the dangers of an anti-spam "monoculture": that Bayesian techniques might be great, but if that's all anyone uses then spammers will catch on and adjust their messages to look more like normal mail, to the point that Bayesian filters won't work anymore. As a result, we're going to need to attack the problem from several angles, using different techniques, to keep the spammers off balance as much as possible.
Matt Sergeant, SpamAssassin
SA is a well known Perl application for heuristically profiling messages as spam, adding headers to the message saying for example "I am 72% sure this is spam because it has X Y Z", and passing off the message to procmail or whatever to be handled accordingly. SpamAssassin can handle a message throughput great enough that it can be deployed at the network level (whereas some of the others, which might have somewhat better hit rates, are still too inefficient at this point). Deployed this way, the differences in effectiveness for single vs. multiple users becomes very apparent, as 99% effective rates fall down into the 95-80% range. This happens because, again, different users define different things as spam, so mapping one fingerprint to all users can never work quite right. For an example of a tool that your company can deploy right now & get fast, decent results, SA looks like a good choice; but for the long run it looks like a Bayesian technique is going to get better performance, and SA is adding a statistical component to its toolkit. Good talk.
Barry Warsaw, Python Labs
This was another example of the "monocultures are dangerous" philosophy, as Warsaw explained how he is helping to use a variety of anti-spam techniques -- from clever Exim MTA configuration to good use of Spam Assassin & Procmail to fine tuning of the MailMan mailing list engine -- to work together to manage the spam problem for all things Python (Python.org, Zope, many mailing lists, a few employees, etc).
He pointed out that some very simple filters can be surprisingly effective: run a sanity check on the message's date; look for obviously forged headers; make sure the recipients are legit; scan for missing Message-Id headers; etc. In response to the person that originally posted the article, yes, he did mention blocking outgoing SMTP as an effective element of a many tiered spam management approach.
Among other tricks for getting the different filtering tiers to play nice together, they make heavy use of the X-Warning header so that if an alarm goes off in one tier of their mail architecture, other components can respond appropriately. Cited projects included ElSpy [sourceforge.net] and SpamBayes [sourceforge.net].
Barry Shein, founder & CEO of The World -- or as he laughingly put it, "President of the World". Har har har
This talk was mostly a let down for me -- Shein has made his views very well known [internetwk.com], and his ranting, rambling talk didn't really introduce any new ideas for anyone that had read that interview (some good jokes & quotes though).
His core argument is that spam is "the rise of organized crime on the internet", that filters are nice but that the mail architecture itself is fundamentally flawed, and that ISPs like his -- in 1989, The World was the world's first dialup ISP -- are being killed by the problem. Shein was very annoyed that all these talented people are having to clean up a mess like this when we should be out working on more interesting stuff, and not having to worry about this issue. His big hope seemed to be that legislation will someday come to the rescue, but he sounded very pessimisstic. (Others in the room seemed to feel that this was a very interesting machine learning problem, and weren't really fazed by his pessimism -- but then most of the people in the room don't run ISPs.)
He also suggested that we need to find a way to make spammers pay for the bandwidth they are consuming (rather than having users & ISPs shoulder the burden) but didn't seem to know how we might go about implementing this. At all.
Fun rant to cheer along to, but for me it wasn't very constructive in the end.
Jean-David Ruvini, eLabs SmartLook
This was an interesting product. Ruvini's company is developing an extension to Outlook 2000 & XP that will watch the way users categorize messages into folders, come up with a profile for what kinds of messages end up in which folders, and then try to offer similar categorization on an automatic basis. Think of it as Procmail for Outlook, without having to mess with (or even be aware of!) all the nasty recipies.
Obviously if you have a spam folder, then spam will be one of the categories it looks for, but more broadly it will try to categorize all your mail as you would ordinarily categorize it. This makes SmartLook a broader tool than "just" a spam manager.
SmartLook is another statistical filter, though it uses non-Bayesian algorithms to get results. eLabs' tests suggest that the product is able to properly categorize messages about 96% of the time, with no false positives, and (for their tests, mind you) that it performed better than Bayes filters over three months of usage.
One nice property of this tool was that it works well with different [human] languages -- some strategies fall apart &/or need retraining when you switch from English to some other language. For certain markets (eLabs seems to be a European company, perhaps French?) this is a crucial feature, and having a tool that works with one of the biggest mail clients out there (most people don't use Mutt or Pine, sadly enough) can be very valuable. Very clever -- watch for the inevitable embrace & extend three years from now.
Eric Raymond
He didn't say anything about guns, but he did try to correct one of the other speakers for misusing the term "hacker."
Like Graham, ESR is a Lisp fan, but he knows that the vast majority of people aren't, and he also knows that the vast majority of people need to be using something like Graham's spam software. So on a lark, he came up with a clean version in C, named it BogoFilter, and put it on Sourceforge, where a community sprung up to, well, embrace & extend it.
As good as Graham's Bayesian algorithm is, ESR felt -- as did many of the other speakers -- that the nature of your spam/ham corpus is much more significant than the relative difference among any handful of reasonably good algorithms. (Back to the often repeated point about how corpus effectiveness falls apart when used for a group of users, as opposed to individuals.) To that end, he strongly feels that the best way to deal with the spam problem is to get good tools into the hands of as many people as possible, and to make them as easy to use as possible (ahh, the old "open source UIs always suck" argument :). As an example, one of the first things he did was to patch the Mutt mail agent so that it had two delete keys: one for general deletion, one for "get rid of this because it's spam." That second key, and interface touches like it, seem like the way to get average people to start using filters on a regular basis.
Joshua Goodman, Microsoft Research
Unlike ESR, Goodman felt that algorithm selection does make a big difference, but this being Microsoft he refused to disclose what algorithms his team is working with -- except to say that, when delivered, they will be more accessible for average users than SpamAssassin, Procmail recipies, or Mutt :)
Microsoft has been working on the spam problem since 1997, but because of how big they are they've had unique problems in bringing solutions to market. As a case in point, they tried to introduce spam filters to a 1999 Outlook Express release, but were immediately sued by email greeting card company Blue Mountain because their messages were being inaccurately categorized as spam. With that in mind, they have been very reluctant to bring new anti-spam software out since then because they would like to see legislation protecting "good faith spam prevention efforts."
As a very large player, Microsoft faced certain difficulties in developing useful filters -- it may make sense for you as an individual to filter all mail from Korea, but this doesn't work so well if you are trying to attract customers *from* Korea :). This has forced them to put a lot of work into thoroughly testing different strategies before offering them to the public.
In spite of what millions of webmail users may have expected, Hotmail & MSN are currently being filtered by Brightmail's service, and plans are underway to reintroduce spam management features to client side software again. (Just imagine how bad it would be if they weren't paying someone to filter for them! Unfortunately, no hecklers piped up to ask if they are really selling Hotmail's user database to spammers, and if that is a source of annoyance for his team.)
An interesting barrier his group has had to grapple with was what he called the "Chinese menu" or "madlibs" spam generation strategy: that it's easy to come up with a template for spam -- "[a very special offer] [to make your penis bigger] [and please your special lady friend all night!" vs. "[an exclusive deal] [for genital enlargement] [that will boost your sex life!]" etc -- and have a small handful of options for each 'bucket' multiplying into a huge variety of individual messages that are easy for a human to group together but almost impossible for software to identify.
Michael Salib, extremely funny MIT student
Unlike nearly all other filter writers of the day, Salib's approach was heuristic: find a handful of reasonable spam discriminators, throw them all against his mail, and see how much he can identify that way. "It's sketchy, but this is a class project. I don't have to be realistic. [...] These results may be completely wrong."
Much to his surprise, he's trapping a lot of spam. He pulls in a little bit of RBL data ("the first two or three links from Google, whatever"), looks for some patterns and so on, and then churns it through LMMSE, an electrical engineering technique that as far as he can tell doesn't seem to be known in other fields. Basically this involves running the messages through a series of scary-but-fast-to-calculate linear equations). It turns out that he can process this much faster than a Bayes filter, to the point that customizing his approach for each user in a network would actually be feasible.
For a small spam corpus, he got results better than SpamAssassin did, though for a large corpus his results were worse; he couldn't really account for why this would be the case, or predict how things would scale as the corpus continued to grow.
When questioned about the RBL tactic by a member of the audience [who was apparently familiar to Salib -- I don't know who it was] about whether authenticating remote users might be the answer, Salib's response was "yes, I agree, but then you *do* work for Verisign, who is in the verification business, so you would say that."
Right on, Salib -- his talk was easily the funniest & breezy of the day :)
David Lewis, general researcher
The core of Lewis' argument, as ESR said earlier in the day, is that for any machine learning technique the quality of the learning corpus is much more important than the algorithm used. Bayes is one such algorithm, but there are many other good ones in the literature. In a dig at Goodman's refusal to disclose algorithms, Lewis pointed out that all of this has been publicly discussed since the first machine learning paper was published in 1961.
Observations: "lots of task inspecific stuff works badly, but task specific stuff helps a lot." It is important to use different corpuses [corpi?] for training and for general use, so that you don't train your machine to focus too much on certain types of input (this is a point that Microsoft's Goodman made as well).
As Graham did, Davis emphasized that spam is going to slowly start looking more like natural text, and we're going to have to deal with this as time goes on. www.daviddlewis.com/events/ [daviddlewis.com]
Jon Praed, Internet Law Group
To a burst of tremendous applause, this talk began with the sentence "my name is Jon Praed, and I sue spammers."
He brought a legal take on the "not everything is spam to everybody" angle, emphasizing that we need a precise definition of what qualifies as Unsolicited Commercial Email (UCE). In particular, it has been difficult trying to pin down if the mail was really unsolicited, as this is where the spammers have the most wiggle room. However, if you can track down the spammer, they have to date rarely been able to verify that the user asked for mail, and so Praed has been able to successfully prosecute several spammers on this angle. He doesn't expect this to work forever though.
According to Praed, "laws against spam exist in every state, and more are pending", but he doubts that a legal solution will ever be completely effective as long as spam is lucrative. By analogy, he pointed out that people still rob banks and that has never been legal.
Praed informed the audience that there are several ways to get back at spammers, including injunctions, bankruptcy, and contempt, and all of these can be very effective. He pointed out that, to be blunt, a lot of these people are desperate low-lifes, and spam has been their biggest success in life. After these legal responses, their lives all get much worse. It hadn't occured to me to see spammers as pitiful before, but I can now. Most importantly, Praed stressed that these legal remedies can be very effective, and he strongly warned against taking vigilante action. This is almost always worse than the spam itself, and it only serves to get you in even deeper trouble than the spammer.
Identifying the sources of spam, most comes from offshore spam houses, abuse of free mail accounts (Hotmail & Yahoo, free signups at ISPs, etc) and bulk software (which may apparently soon become illegal in certain areas, provided that a law can be found to ban spam software while allowing things like MailMan or MajorDomo). Interestingly, he questioned the idea that header spoofing is a big problem, and claimed that in every case he has dealt with he has been able to track down the messages to a legit source sooner or later.
Suggestion: if you get a spam citing a trademarked product [e.g. Viagra], forward it to the trademark holder and they will almost always follow up on it. Suggestion: be fast in trying to track down spammers, as some of them have gotten in the habit of leaving sites up long enough for mail recipients to visit, but taking them down before investigators get a chance to take a look. Legal observation: spam is almost always fraud, and can be prosecuted accordingly.
Praed wrapped up his talk by citing the encouraging precedent that the famous Verizon Online vs. Ralsky case set: [a] that the court is interested in where the harm occurs, not where the person doing harm was when causing it (so if you send spam to someone in Alaska and spam is a capital offence in Alaska, you can be tried as a citizen of that state even if you caused the harm from somewhere else), and [b] it is assumed that you have to be familiar with a remote ISPs acceptable usage policies, and ignorance is no defence (just as you can't say "I didn't know it was illegal to shoot someone", Ralsky couldn't say that he didn't know Verizon prohibits spam -- (he had to have known that the AUP wouldn't allow what he was doing, so he deliberately didn't read it)). That precedent makes future prosecution of spammers much more encouraging. While, again, legal solutions may never eliminate the spam problem, a precendent like this can be an important supplement to filtering efforts (the stick to the filter's carrot, or something -- my lousy analogy, not Praed's).
David Berlind, ZDNet executive editor
His talk was primarily about how he receives a huge quantity of email from ZDNet readers, and he can't afford to use any spam filtering solution strategy that would allow *any* false positives. As one of the speakers said -- sorry, I forget who (Microsoft's Goodman?) -- getting a 0% false positive rate is easy: just classify nothing as spam. Getting a 100% hit rate is also easy: just classify everything as spam. Any solution besides those two is always going to have some degree of error either way, and determing how much of what kind of error you want to accept is up to you. Most users will tolerate a moderate false negative rate (some spam gets through) if it means that the false positive rate (legit mail is deleted) is very low. In Berlind's case, the false positive rate has to be vanishingly small, because reading all customer mail is a critical sign of respect for him.
Further, his business is also a legitimate mass emailer, sending out millions of free newsletters to users every day, and if Shein's proposal to bill bulk mailers were to catch on then even a very low rate would quickly put his company in the red. One obvious solution, which wasn't mentioned: start charging a subscription for these mailings, and make them profitable. I don't want to see this happen but if it did then the economics would tilt back toward making things feasible again.
Berlind is appreciative of the anti-spam work that is being done, but at the same time is skeptical of how pragmatic most of what is being proposed can really be. He feels we need a massive effort to rework the way mail is handled [Y2K anyone? It could get IT people back to work...], and to that end hopes ZDNet can help promote such a cooperative effort between the parties working on this. They don't want to be involved -- they are journalists & publishers, not standards developers -- but they are eager to get things going & want to cover the story as it progresses.
Like Shein said, he feels it's a waste for all these talented people to be working on combating penis enlargement offers, and hopes that we can find a way to get past this and work on real problems, "like world peace." This comment got a chuckle from the audience, but he seemed like the kind of guy that really meant that, and more importantly, he was right. A smart guy like Paul Graham or Bill Yerazunis shouldn't have to waste time tinkering with how many Viagra offers he can automagically delete when there are more fun things to be doing.
Ken Schneider, Brightmail
As mentioned earlier, Brightmail provides an ASP service for real time filtering of both incoming & outgoing mail. As would perhaps be expected, bigger ISPs and networks attract larger amounts of spam: 50% of mail coming into big ISPs and 40% coming into big companies is now spam. Brightmail offers the Probe Network, a <slashdot-killfile-term>patented</slashdot-killfil e-term> system of decoy honeypot addresses that gather data for analysis at their logistics center, which in turn distributes spam filtering rules to their clients where a plugin for $MTA (using the open source or proprietary MTA of the client's choice) can act on the database.
An interesting property of their system is that they have a mechanism for both aging out dormant rules as well as for reactivating retired ones, so that the currently active ruleset can be kept as lean & effient as possible. A big source of difficulty for them is legitimate commercial opt-in lists, because things have gotten more shady & blurry over time and it's now hard to tell this mail from much of the spam out there. Whitelists help here, but the problem is still difficult.
After each speaker had his turn, there was a panel discussion, but not much really happened there, and the moderator cut things short after only a couple of minutes. The original plan was for everyone to go out for Chinese food afterwards and continue the discussions over dinner, but when 580 people signed up that plan obviously fell apart. :) And so, here ends the notes...
Port 25 blocking? (Score:3, Insightful)
I think the key problem is ISPs that do not block egress traffic on port 25.
No.. the ISPs that block port 25 already care about spam, they just block it to reduce their administrative load. It reduces the spam cases they have to deal with - but they still cut off spammers. If they didn't block 25, they'd still cut off the spammers. The actual problem is ISPs that don't care about spam. These ISPs don't deal with their spammers so how can you expect them to block port 25?
If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive)
Funny, I base this statistic on the fact that you pulled it out of your ass. AOL has had spam problems, but they do deal with their spammers. It's ludicrous to suggest that they are responsible for half of all the spam on the Internet.
Tell me "RT Alec," how is port 25 blocking going to deal with rogue ISPs, who have a bulletproof connection through Verio? How about the clueless open relays that dot the maps of China, Brazil, and Argentina? What about for users of business DSL? Do we say, "you can't use your own corporate SMTP server, because you could be a spammer and we don't want to bother to deal with it?"
Declare foreign spam to be terrorism... (Score:3, Funny)
Oh sure, (Score:3, Interesting)
BTW: Cause I run my own port 25 and have a static IP and a domain name, I get hardly any spam, personally. Why? Because I give out a different novel seperate address to everyone, and keep them all aliased to forward to my main account. If one becomes contaminated by spam, I simply delete it. If it actually was an address I gave to a correspondant [and not to some website, which is almost universally is] I only have to inform one person of a new address... come to think of it, that's only happened once...
Conference Notes (and a comment on blocking port 2 (Score:3, Insightful)
My notes on the conference can be found at http://commons.somewhere.com/buzz/2003/Technology. Notes.from.th.html [somewhere.com]. The really quick summary--everyone's got content-filtering fever, and I think they are nuts. You're trying to filter something that is NP-complete (Javascript email) and then do natural language understanding on it? I don't think so. Just as an example, consider the following three spams I've received recently.
Content filtering is doomed.
Oh yes, about blocking port 25. This is always followed by "and then your sysadmin can run SMTP on a different port so that you can connect to it via that." And if this becomes common, how long do you think until the spammers start scanning for alternate SMTP ports and doing direct delivery? In any case, it's moot. 90% of your spam isn't being sent from this country anyway. You're not going to persuade those remote sysadmins to block outbound port 25 any more than we've managed to get them to close their open relays. This is big business and big bucks.
You're being naive, good sir (Score:4, Troll)
As Paul Graham writes, "A few simple rules will take a big bite out of your incoming spam. Merely looking for the word "click" will catch 79.7% of the emails in my spam corpus, with only 1.2% false positives.
One idea that I haven't tried yet is to filter based on word pairs, or even triples, rather than individual words. This should yield a much sharper estimate of the probability. For example, in my current database, the word "offers" has a probability of
Re:You're being naive, good sir (Score:2)
Sigh... But I liked to be able to check my email via telnet...
Re:You're being naive, good sir (Score:2)
Re:You're being naive, good sir (Score:2)
I read this post, but I'm not quite sure why it's been modded as troll. Am I being naieve?
Graphic spam (Score:2)
Re:One person's treasure is another person's junk. (Score:5, Insightful)
First of all, CRM114 is just a language. Bayesian filters could just as easily be written in Perl or C. The language makes no discrimination whatsoever.
Secondly, the very point of Bayesian filtering is that it learns what you consider trash and what you consider treasure. You start with a training set of several hundred "legit" messages and several hundred spams, and it goes from there.
The reason it works so well on a person-by-person configuration is that certain phrases (eg, email addresses of people you know in the "From" header) correlate very strongly to good mail, while phrases like "click here" and "enlarge your" are almost certainly spam indicators. Everything between is personal; if you're on a BDSM list, your filter will learn that you like that stuff. Given a training set with your personal tastes, rates well in excess of 95% are possible.
Incidentally, this is why Bayesian methods aren't that great for site-wide filtering (that, and they would be tremendously slow); it's much harder to establish what a *group* of people considers to be "not spam."
Re:One person's treasure is another person's junk. (Score:3, Insightful)
So if your personal mail often has "penis" in it, the filter learns that it is not a good indicator for spam.
I use POPFile http://popfile.sourceforge.net/ [sourceforge.net], and I have noticed that one of the best indicators seems to be certain server names.
Re:One million dollars later (Score:2)
That said, the conference was free & MIT is not a public institution, so the comment is a little misplaced. Funny, but misplaced :)
Re:One million dollars later (Score:3, Funny)
Tax money well-spent
Hey now, those of us with 14" penises that make millions at home would disagree with that ruling.
Re:A simple solution to spam? (Score:4, Insightful)
The illegality here would scare the pants off of all the spammers in Asia, I'm sure...
Re:No Spam in Two Years? (Score:2)
I buy a book from Amazon.com. I give them the following address: amazoncom20021109@joe.mydomain.com (ie, domain+date@mydomain)
If amazon abuses that address, I remove it from the active list of allowed addresses, and it just bounces.
However, there's a problem. Neither your proposal or method I use will allow the guy off the street to mail me without having been given a valid address, nor will they make spam no longer cost effective. These are the two main problems to controlling spam - allowing mail you want (without knowing the person sending it), and providing a feedback mechanism to actively discourage future spam (a penalty.)
Spammers don't care about bounces, because they don't have to deal with them. And unless you force everyone to refuse mails from unknown senders, the spammer business model of spam everyone, get some idiot schmuck to bite, still holds. So all we've done is increase the level of complexity without fixing the root problem - the economics of spam. Sure we've cut down on the amount of spam we personally get, but we can do that already (Spamassassin, throwaway e-mails, authentication, etc.)
We need something to make them think twice before spamming people... like criminal penalties. The prospect of having to spend a year with Bubba (or Bertha) up in the big house should be a sufficient deterrent.
Re:poor premise of the submitter (Score:2)
You can have an open port 25 listener if you like (so long as it's not set up to do inappropriate 3rd party relay). But forcing *your* server to talk to the ISP's mail server for *sending outgoing* mail is what we're talking about. So long as the ISP's mail server actually does properly relay the mail for you and doesn't do anything else out of spec, there's nothing (much) wrong with them forcing you to use it.
There is the privacy issue (suddenly your mail winds up on a 3rd party system for however long it takes that machine to spool it, then send it on its way), but my expectation of privacy for unencrypted e-mail is already fairly low.
Re:The big problem with email... (Score:2)