Two Spam Filters 10 Times As Accurate As Humans

Nuclear Elephant writes "The authors of two spam filters, CRM114 and DSPAM, announced recently that their filters have achieved accuracy rates ten times better than a human is capable of. Based on a study by Bill Yerazunis of CRM114, the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984% (1 misclassification in 6250 messages) using completely different approaches (CRM114 touts Markovan, while DSPAM implements a Dolby-type noise reduction algorithm called Dobly). If you're looking for a way to rid spam from your inbox, roll on over to one of these authors' websites."

Outclassed...

I'm sorry, Dave... That Nigerian guy looks suspicious and I can't let you send him money.

Re:Help setting this up

Umm, Fetchmail + procmail on your local machine?

Not sure exactly why you need a pop3 proxy involved, just use Fetchmail to deliver locally, run things through procmail.

Set your local mailserver (sendmail/qmail/postfix/exim/whatever) to use your ISP's SMTP server as a smarthost, and it'll send everything it doesn't recognize as local off to them to handle.

It _can't_ know which pr0n I think is spam vs good

I signed up for lots of junk mail lists; some solicited, some not -- sometimes from the same organizations.

How would it know if I consider brunettes non-spam but blondes spam? I did opt-in for one of those email categories, but not the other.

Re:Help setting this up

ModernGeek,

I recommend you stick with hotmail. Dabbling in stuff like spamassasin is going to be just too much work for someone as lazy as you sound. Apple makes a good built-in spam filter on its Mail client app. Why don't you go there?

Huh? Aren't humans 100%?

How can a spam filter be more accurate than humans? Humans are always the last step in spam filtering.. i use popfile and it catches 99% but it still needs me.. because im the only one capable of identifying spam 100% of the time.

Re:Huh? Aren't humans 100%?

I haven't been 100% accurate.

I received an email from my sister-in-law from her work, and the address looked suspicious (one of those weird-looking "letter and number" jumbles.

I deleted it. It happens.

*slams head against wall*

I received an email from my sister-in-law from her work

Yeah, so did I. The subject line was "I want you so bad."

I deleted it. Turned out the message was genuine. I'll never forgive myself...

Re:*slams head against wall*

If you can't forgive yourself, I'll forgive you... as soon as I recieve your sister-in-law's email address.

Don't worry

Don't worry, I can forward you the one she sent me. Sounds like the same email.

Re:*slams head against wall*

If it was your sister-in-law sending you that subject line, you probably did the right thing and deleted it

Re:Huh? Aren't humans 100%?

That actually makes humans much more accurate. We can eliminate many of the messages just by looking at the subject.

The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?

Re:Huh? Aren't humans 100%?

The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?

I believe that humans can be 100% accurate (or thereabouts) if they read the *ENTIRE* message, however that's exactly the point - if you have to read an entire message to tell that it's spam, the spam has succeeded.

Their number probably concerns how people can tell without reading the entire message whether or not the message is spam. My brother accidentally deleted a few messages I had sent to him, however if he had read them fully he would have known they were legit.

Cheers,
Justin

Re:Huh? Aren't humans 100%?

But the computer reads the entire message, so it's not really a fair comparison, is it? How many more lines of information was the computer allowed to look at to create its superior result?

Re:Huh? Aren't humans 100%?

Computers are neither lazy nor pressed for time, and therefore can afford to read and evaluate every single line of every single message. Humans generally can't be bothered to be so diligent, and while they have the ability to get a 100% rate, in most cases they devote so little attention to the task of filtering email that the success rate drops.

When these factors are considered, I think it's quite possible to write software that in the long run has a higher success rate than a human who has better things to do than filter his mail all day.

Re:Huh? Aren't humans 100%?

I dunno. I'm running CRM114 now, and it's taking something like 1.5 seconds to identify emails. I am on a slow machine though, which used to do SpamAssassin at around 4 seconds, and inaccurately to boot. CRM114 is a big improvement, and if it trains well after the first fortnight I'll kiss TMDA goodbye.

Re:Huh? Aren't humans 100%?

There goes my bussines idea. I wanted to start a bussines that puts humans in an eastern europe contry to sort corporate e-mail.

Now I have to think again about putting humans to decorticate sunflower seeds, it's cheper than all those machines.

Re:Huh? Aren't humans 100%?

I suppose it depends how you're defining spam. Perhaps the ultimate spam messages that don't get past them are capable of passing a turing test... hence fooling those gullible human recipients into thinking that it isn't even spam!

Fortunately, soon we will all be able to use the superhuman spam-detection capabilities of these filters to save us from ourselves. Imagine all of those pesky e-mails from your 'friends' getting caught by your spam filter before they even impinge upon your consciousness.

It'd be a wonderful world.

Re:Huh? Aren't humans 100%?

If you read the post, it quotes a study and says humans are only accurate 99.84% of the time.

Kinda makes you wonder how they can know the filters are right though. :)

(please don't reply telling me how)

Re:Huh? Aren't humans 100%?

Presumably they must use a superhuman who has 100.00% accuracy.

Re:Huh? Aren't humans 100%?

No, humans aren't 100% and yes, you can test for that. Try a thought experiment: fill a bin with 50,000 red balls and 50,000 blue balls. Ask a human to sort them all. The result probably won't be 100%, but you can still check the result and figure out how accurate the human is without relying on a superhuman ability to tell the balls apart. Same thing for spam: if you start with a known training set, you can test humans to see how well the spam is identified by manual sorting.

Re:Huh? Aren't humans 100%?

How do you know your training set is correct?

Good question! We're working on this problem, among other things, at the PSAM [pdx.edu] project. We have a project to produce high-quality benchmark corpora for spam filter testing. Watch that space for ongoing work, or e-mail us an offer to pitch in and help---we could use it!

Re:Huh? Aren't humans 100%?

fill a bin with 50,000 red balls and 50,000 blue balls. Ask a human to sort them all.

Not comparable. The job of a junk mail filter is to drop things I don't want to read. It is trying ot match my evaluation, not to match a semi-objective criterion like red or blue.

If I read 1000 messages and say which I wish I hadn't read, then I am 100% accurate by definition.

Of course, if they are really talking about a pure spam filter -- ie one which identifies unsolicited commercial email -- then they can be more accurate than me, but at an uninteresting, perhaps even counter-productive, task:

I may get unsilicited commercial email I do want to read one day. Almost happened once (I had inadvertantly signed up for it, so it was not really unsolicited, and I didn't actually buy the piece of kit they had on special offer that week, but was tempted). I also get stuff I don't want which isn't spam (notably email from virus infected machines).

The referenced study seems to be a very sloppy job from this POV. They don't define what their criterion of sucess is, and to the extent they put in a hand waving attempt it is clearly nonsense:

Because spam (sometimes termed ?unsolicited commercial email? or ?marketing messages?) is neither expected nor desired[...]

`Unsolicited' does not imply `not desired'. If they don't tease those two apart, they can't get interesting results for real world applications. Eg, someone mailing my work address with a commercial proposition may well be a very welcome unsolicited commercial email.

Re:Huh? Aren't humans 100%?

The post quotes "a study" which gives the 99.84% figure. In fact, the 99.84% figure is mentioned in the one paper as "the human author's measured accuracy as an antispam filter...on the first pass". This is what we who understand statistics call "nonsense". An individual human had an estimated accuracy of 99.84% when looking at one particular sample set of data, once. This is not a meaningful number, and it sure as heck ain't "a study".

Re:Huh? Aren't humans 100%?

Obviously you've never seen someone new to the internet sit in front of their computer. Lots of people don't know what popups are. Lots of people read some spam not knowing what it is. To these people, a computer is merely an interesting string of sensations.

Re:Huh? Aren't humans 100%?

I work with some people who use their computer every single day. Have had an email address for years, who still buys what they read in an email. Photoshop for $50...sure! Herbal viagra...why not?

Well, she always has a big smile on her face, maybe there's something to this spam thing.

Re:Huh? Aren't humans 100%?

Well, she always has a big smile on her face, maybe there's something to this spam thing.

You mean you've never noticed this before? Idiots are some of the happiest people I know.

Re:Huh? Aren't humans 100%?

Lots of people don't know what popups are.

Uh, sure they do. Popups - that's like those porn storms, isn't it? Some people say it only happens with IE and Windows, but I talked to my service provider and they told me 'just pull the power plug out of the wall when that happens'.

Easily fixed.

Re:Huh? Aren't humans 100%?

I talked to my service provider and they told me 'just pull the power plug out of the wall when that happens'.

Ok, now the screen dimmed a little and I heard the hard drive spin down, but the pop ups are still a comin! Oh, and something about "battery level at 98%" or something.

Re:Huh? Aren't humans 100%?

Perhaps they mean that Human A is reading email intended for Human B and attempting to classify the email as spam or not spam. I wouldnt be surprised if a computer could do a better job at that sort of task. Besides Im sure Human B wouldnt want Human A reading that cyber sex chat log.

Re:Huh? Aren't humans 100%?

Quite simple:
With 10 messages (after automatic spam detection) humans are 100% accurate.

With 1,000 messages, (before automatic spam detection)
humans are less than 100% accurate.

The experiment was done on 5849 messages.

Remember; one thing computers are good at is doing boring things repeatedly.

Re:Huh? Aren't humans 100%?

No, humans are not 100%.

If you see a strange name in your inbox with an odd title, that might be a Nigerian businessman, or it might be your long lost Nigerian brother.

I recently tried to order a t-shirt from this guy for a band he used to be in. I found his band because we have the same (semi-uncommon) name. So, he got an email From: himself. I had to send him two emails because he deleted the first one assuming it was spam.

I ordered some RAM for my dad a while back. He gets 200 spam emails a day (email addy in resume & web page), and he deleted the confirmation email from the RAM vendor. The RAM never shipped, and it took us a week to figure out that there was a problem.

People make mistakes all the time. Why is this an unexpected result? People are jackasses. This should be obvious.

Re:Huh? Aren't humans 100%?

Then megacorp could sue spammer into oblivion.

Or more likely, megacorp fires it's mail administrators for being incompetent and goes on about it's business.

Re:Huh? Aren't humans 100%?

I have never deleted an email I meant to keep

How could you possibly know? You deleted it!!

Re:Huh? Aren't humans 100%?

How can a spam filter be more accurate than humans? Humans are always the last step in spam filtering.. i use popfile and it catches 99% but it still needs me.. because im the only one capable of identifying spam 100% of the time.

And if the study posted about is accruate, of those 1% that are left, you will (if you're a perfectly average person) accidentally delete 0.16% of good messages. Surely you've deleted a valid message by accident before? I do it regularily, deleting 25 spam messages with a single good one embedded in it when I just woke up before I had my coffee is not a good thing ;)

At the very least, if you were given the same data as these tests, that would be true. Consider if you *didn't* use popfile - how many spams would you be deleting every day, and how many good messages would be accidentally deleted? I know that if I had to manually delete the two or three hundred spams interspersed with good messages, my false-positive rate (the percentage of good mail I accidentally deleted) would skyrocket.

So just be glad you've got popfile. Not only do you not have to go through as much spam, but you're also more accurate while going through the little you must.

Re:Huh? Aren't humans 100%?

Last week I ran a spam filter on all the email I recieved for the last several months. The filter came up with a dozen 'false positives' - messages that I had not flagged as spam when I manually classified them. 11 of them were clearly errors I made in my original classification. The 12th was a solicitation from the alumni association of my alma mater ....

Before I used a spam filter, I once missed a very important message whose subject line was something to the effect of "URGENT - DON't REBOOT THIS MORNING." That was a bad one to miss.

Of course humans make mistakes, and it is entirely possible for an automated or semi-automated system to be more accurate than a human alone.

IM Spam

Once Email Spam is eliminated, then IM spam will begin...

wait, WTF?

I presume they mean more accurate than a human that was only looking at the subject line? I fail to see how someone could misclassify an email after they'd already opened it unless it was some kind of marathon testing, which would be totally unrepresentative of any real life situation. Once you're getting 6,000 messages, it's time to reach for "Delete All" and change your address, methinks

Re:wait, WTF?

6000 over what period?

This represents 8 days worth of spam for me. Yes, ~800 per day.

My address has been valid for 10 years. Why should I change it? Bogofilter is currently letting 2-3 per day into my inbox. I generally check for false-positives, but as the training has progressed, I am finding none anymore.

I plan to implement a single-shot, one try notification sender. I.e., if the mail gets classified as spam: lookup the mx record for the envelope return address, if it's nonexistent, lookup the a record. Make a connection and try to deliver a message indicating their message (include subject reference) was identified as spam, include a way for them to reliably get a message through to me. If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway.

Not the best idea

What you're planning has already been done, it's called TMDA, and it's not such a good idea. You're going to send out 800 "challenge" emails per day - have you given any thought to how many of those will be genuine addresses, but have nothing to do with the spam you receive because they just happen to be the joe-job victim? These kind of challenge/response systems may slighlty alleviate your own suffering through spam, but at a cost to all those unfortunate enough to have had their email addresses faked. And if the sheer impoliteness of such net behaviour doesn't put you off, note that you're using up more of your own bandwidth to send out such challenges

If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway

It would make a lot more sense to make these kind of checks when you're receiving the email in the first place. Reject at the SMTP level - you never accept and process the spam in the first place

2+2=3

the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984%

Am I crazy or is that nowhere near "10 times better"?

Number of significant digits...

Human=99.84
New proggie=99.984

So the human misses .16% and the machine only missues .016% hence the machine is 10 times better.

Re:2+2=3

Congratulations, Mon Ami.

You have just unlocked the secret of virtually every news report that says "ten times more likely."

To get cancer. To have a heart attack. To suffer from the heartbreak of psoriasis. Whatever.

Yes, these numbers indicate "10 times better," and if you were to ask the reporter how likely am I to avoid cancer in both situations, these are the sorts of numbers he would show you.

Eat health food and your chance of having a heart attack is 99.984%. Eat too many donuts and your chance of having a heart attack is 99.983%, 10 times worse!

Always, always, always ask to see the raw numbers so that you know what "10 times worse" means.

Then ask if the numbers were collected by phone survey. If they were, throw them all away and have donut and a cup of coffee.

KFG

can it be used with SA?

can this be used with Spamassasin, or is a stand alone program? Does it need something like Amasis to run?

CB

Re:can it be used with SA? -yes

A CRM114 plugin for SA is available, thanks to Devin Nate:

http://bugzilla.spamassassin.org/show_bug.cgi?id =2 301

Who is sending that one?

If your email is indistuinguishable from spam by a human, perhaps the problem isn't the receiver. It's the sender.

Forgive me if I don't feel any pity that some moron's email gets filtered to the junk bin because I couldn't discern it from spam.

To get this new spam filter...

Just enter a valid email address, and hit submit!

Better

Well, it certainly sounds better than the pay-per-email "postage" idea. If postage hasn't stopped snail spam, why would it stop e-mail spam?

less thought for me...

...and only one locked pod bay door per 6250, I like those odds.