Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Spam The Internet Your Rights Online

Two Spam Filters 10 Times As Accurate As Humans 487

Posted by timothy from the dev/null-is-getting-fatter dept.
Nuclear Elephant writes "The authors of two spam filters, CRM114 and DSPAM, announced recently that their filters have achieved accuracy rates ten times better than a human is capable of. Based on a study by Bill Yerazunis of CRM114, the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984% (1 misclassification in 6250 messages) using completely different approaches (CRM114 touts Markovan, while DSPAM implements a Dolby-type noise reduction algorithm called Dobly). If you're looking for a way to rid spam from your inbox, roll on over to one of these authors' websites."
This discussion has been archived. No new comments can be posted.

Two Spam Filters 10 Times As Accurate As Humans More

Two Spam Filters 10 Times As Accurate As Humans

Comments Filter:

  • Outclassed... (Score:5, Funny)

    by Klatoo55 ( 726789 ) on Monday February 23, 2004 @08:14PM (#8368802) Homepage
    I'm sorry, Dave... That Nigerian guy looks suspicious and I can't let you send him money.

  • Comment removed (Score:5, Insightful)

    by account_deleted ( 4530225 ) on Monday February 23, 2004 @08:15PM (#8368805)
    Comment removed based on user account deletion

    • Re:Huh? Aren't humans 100%? (Score:5, Informative)

      by MarkJensen ( 708621 ) on Monday February 23, 2004 @08:17PM (#8368834) Homepage

      I haven't been 100% accurate.

      I received an email from my sister-in-law from her work, and the address looked suspicious (one of those weird-looking "letter and number" jumbles.

      I deleted it. It happens.

    • Humans sometimes make mistakes, that's where the inaccuracy comes from.

    • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

      by Behrooz ( 302401 ) on Monday February 23, 2004 @08:18PM (#8368850)
      I suppose it depends how you're defining spam. Perhaps the ultimate spam messages that don't get past them are capable of passing a turing test... hence fooling those gullible human recipients into thinking that it isn't even spam!

      Fortunately, soon we will all be able to use the superhuman spam-detection capabilities of these filters to save us from ourselves. Imagine all of those pesky e-mails from your 'friends' getting caught by your spam filter before they even impinge upon your consciousness.

      It'd be a wonderful world.

    • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

      by gid13 ( 620803 ) on Monday February 23, 2004 @08:20PM (#8368878)
      If you read the post, it quotes a study and says humans are only accurate 99.84% of the time.

      Kinda makes you wonder how they can know the filters are right though. :)

      (please don't reply telling me how)

      • Re:Huh? Aren't humans 100%? (Score:5, Funny)

        by Trejkaz ( 615352 ) on Monday February 23, 2004 @09:00PM (#8369266) Homepage
        Presumably they must use a superhuman who has 100.00% accuracy.

      • Re:Huh? Aren't humans 100%? (Score:4, Insightful)

        by bhanafee ( 145604 ) on Monday February 23, 2004 @09:14PM (#8369386) Homepage
        No, humans aren't 100% and yes, you can test for that. Try a thought experiment: fill a bin with 50,000 red balls and 50,000 blue balls. Ask a human to sort them all. The result probably won't be 100%, but you can still check the result and figure out how accurate the human is without relying on a superhuman ability to tell the balls apart. Same thing for spam: if you start with a known training set, you can test humans to see how well the spam is identified by manual sorting.

        • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

          by R.Caley ( 126968 ) on Tuesday February 24, 2004 @06:50AM (#8372015)
          fill a bin with 50,000 red balls and 50,000 blue balls. Ask a human to sort them all.

          Not comparable. The job of a junk mail filter is to drop things I don't want to read. It is trying ot match my evaluation, not to match a semi-objective criterion like red or blue.

          If I read 1000 messages and say which I wish I hadn't read, then I am 100% accurate by definition.

          Of course, if they are really talking about a pure spam filter -- ie one which identifies unsolicited commercial email -- then they can be more accurate than me, but at an uninteresting, perhaps even counter-productive, task:

          I may get unsilicited commercial email I do want to read one day. Almost happened once (I had inadvertantly signed up for it, so it was not really unsolicited, and I didn't actually buy the piece of kit they had on special offer that week, but was tempted). I also get stuff I don't want which isn't spam (notably email from virus infected machines).

          The referenced study seems to be a very sloppy job from this POV. They don't define what their criterion of sucess is, and to the extent they put in a hand waving attempt it is clearly nonsense:

          Because spam (sometimes termed ?unsolicited commercial email? or ?marketing messages?) is neither expected nor desired[...]
          `Unsolicited' does not imply `not desired'. If they don't tease those two apart, they can't get interesting results for real world applications. Eg, someone mailing my work address with a commercial proposition may well be a very welcome unsolicited commercial email.

      • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

        by Anonymous Coward on Monday February 23, 2004 @09:31PM (#8369544)
        The post quotes "a study" which gives the 99.84% figure. In fact, the 99.84% figure is mentioned in the one paper as "the human author's measured accuracy as an antispam filter...on the first pass". This is what we who understand statistics call "nonsense". An individual human had an estimated accuracy of 99.84% when looking at one particular sample set of data, once. This is not a meaningful number, and it sure as heck ain't "a study".
        • Yes, but it is meaningfull nonetheless. If you just think that it's very likely that after reviewing 650 messages, you may have missed one email that you thought was spam, then the "study" is right. I don't care if the number is 900 or 400 emails. Those 400 mails are making me lose a _lot_ of time, and if I value my time, I am losing a lot of productivity, and also missing an important email.

          If the program can have a .99 accuracy, then it's a real time saver, and if it only makes a mistaque every 2000 emai

    • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

      by mattkime ( 8466 ) on Monday February 23, 2004 @08:23PM (#8368927)
      Obviously you've never seen someone new to the internet sit in front of their computer. Lots of people don't know what popups are. Lots of people read some spam not knowing what it is. To these people, a computer is merely an interesting string of sensations.

    • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

      by Celandro ( 595953 ) <celandro@UMLAUTgmail.com minus punct> on Monday February 23, 2004 @08:25PM (#8368943)
      Perhaps they mean that Human A is reading email intended for Human B and attempting to classify the email as spam or not spam. I wouldnt be surprised if a computer could do a better job at that sort of task. Besides Im sure Human B wouldnt want Human A reading that cyber sex chat log.

    • Re:Huh? Aren't humans 100%? (Score:5, Insightful)

      by evilmrhenry ( 542138 ) on Monday February 23, 2004 @08:29PM (#8368989)
      Quite simple:
      With 10 messages (after automatic spam detection) humans are 100% accurate.

      With 1,000 messages, (before automatic spam detection)
      humans are less than 100% accurate.

      The experiment was done on 5849 messages.

      Remember; one thing computers are good at is doing boring things repeatedly.

    • Re:Huh? Aren't humans 100%? (Score:5, Interesting)

      by Elwood P Dowd ( 16933 ) <judgmentalist@gmail.com> on Monday February 23, 2004 @08:31PM (#8369003) Journal
      No, humans are not 100%.

      If you see a strange name in your inbox with an odd title, that might be a Nigerian businessman, or it might be your long lost Nigerian brother.

      I recently tried to order a t-shirt from this guy for a band he used to be in. I found his band because we have the same (semi-uncommon) name. So, he got an email From: himself. I had to send him two emails because he deleted the first one assuming it was spam.

      I ordered some RAM for my dad a while back. He gets 200 spam emails a day (email addy in resume & web page), and he deleted the confirmation email from the RAM vendor. The RAM never shipped, and it took us a week to figure out that there was a problem.

      People make mistakes all the time. Why is this an unexpected result? People are jackasses. This should be obvious.

    • Re:Huh? Aren't humans 100%? (Score:5, Interesting)

      by dbarclay10 ( 70443 ) on Monday February 23, 2004 @08:37PM (#8369052)
      How can a spam filter be more accurate than humans? Humans are always the last step in spam filtering.. i use popfile and it catches 99% but it still needs me.. because im the only one capable of identifying spam 100% of the time.

      And if the study posted about is accruate, of those 1% that are left, you will (if you're a perfectly average person) accidentally delete 0.16% of good messages. Surely you've deleted a valid message by accident before? I do it regularily, deleting 25 spam messages with a single good one embedded in it when I just woke up before I had my coffee is not a good thing ;)

      At the very least, if you were given the same data as these tests, that would be true. Consider if you *didn't* use popfile - how many spams would you be deleting every day, and how many good messages would be accidentally deleted? I know that if I had to manually delete the two or three hundred spams interspersed with good messages, my false-positive rate (the percentage of good mail I accidentally deleted) would skyrocket.

      So just be glad you've got popfile. Not only do you not have to go through as much spam, but you're also more accurate while going through the little you must.

    • If you have no spam filters, then classifying email amounts to "delete, delete, delete, delete, down-arrow, delete, delete, down-arrow, delete, delete, whoops!" That one mistake just dropped your average to 90%. Frankly, I'm amazed humans scored as well as they did.

    • Re:Huh? Aren't humans 100%? (Score:4, Interesting)

      by gvc ( 167165 ) on Monday February 23, 2004 @09:01PM (#8369272)
      Last week I ran a spam filter on all the email I recieved for the last several months. The filter came up with a dozen 'false positives' - messages that I had not flagged as spam when I manually classified them. 11 of them were clearly errors I made in my original classification. The 12th was a solicitation from the alumni association of my alma mater ....

      Before I used a spam filter, I once missed a very important message whose subject line was something to the effect of "URGENT - DON't REBOOT THIS MORNING." That was a bad one to miss.

      Of course humans make mistakes, and it is entirely possible for an automated or semi-automated system to be more accurate than a human alone.

  • IM Spam (Score:5, Interesting)

    by jeffskyrunner ( 701044 ) on Monday February 23, 2004 @08:15PM (#8368806)
    Once Email Spam is eliminated, then IM spam will begin...

  • wait, WTF? (Score:5, Insightful)

    by PedanticSpellingTrol ( 746300 ) on Monday February 23, 2004 @08:15PM (#8368812)
    I presume they mean more accurate than a human that was only looking at the subject line? I fail to see how someone could misclassify an email after they'd already opened it unless it was some kind of marathon testing, which would be totally unrepresentative of any real life situation. Once you're getting 6,000 messages, it's time to reach for "Delete All" and change your address, methinks

    • Re:wait, WTF? (Score:3, Interesting)

      by LBArrettAnderson ( 655246 )
      look at it this way... you've just tuned in to your favorite radio station and you hear your favorite DJ talking about something. Sometimes you could mix what he's saying up between an advertisement or something he's discussing for the sake of discussing.

      i'm sure there's spam out there that makes it seem like it's one of your friends talking to you (sending with "nick" or "john" as the sender name) and talks to you in a friendly manner about how great this product is.

      i've got a few of those, but luckily

    • Re:wait, WTF? (Score:4, Interesting)

      by HeelToe ( 615905 ) on Monday February 23, 2004 @08:42PM (#8369109) Homepage
      6000 over what period?

      This represents 8 days worth of spam for me. Yes, ~800 per day.

      My address has been valid for 10 years. Why should I change it? Bogofilter is currently letting 2-3 per day into my inbox. I generally check for false-positives, but as the training has progressed, I am finding none anymore.

      I plan to implement a single-shot, one try notification sender. I.e., if the mail gets classified as spam: lookup the mx record for the envelope return address, if it's nonexistent, lookup the a record. Make a connection and try to deliver a message indicating their message (include subject reference) was identified as spam, include a way for them to reliably get a message through to me. If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway.

      • Not the best idea (Score:5, Insightful)

        by Vainglorious Coward ( 267452 ) on Tuesday February 24, 2004 @12:20AM (#8370691) Journal

        What you're planning has already been done, it's called TMDA, and it's not such a good idea. You're going to send out 800 "challenge" emails per day - have you given any thought to how many of those will be genuine addresses, but have nothing to do with the spam you receive because they just happen to be the joe-job victim? These kind of challenge/response systems may slighlty alleviate your own suffering through spam, but at a cost to all those unfortunate enough to have had their email addresses faked. And if the sheer impoliteness of such net behaviour doesn't put you off, note that you're using up more of your own bandwidth to send out such challenges

        If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway

        It would make a lot more sense to make these kind of checks when you're receiving the email in the first place. Reject at the SMTP level - you never accept and process the spam in the first place

  • 2+2=3 (Score:3)

    by Chess_the_cat ( 653159 ) on Monday February 23, 2004 @08:15PM (#8368813) Homepage
    the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984%

    Am I crazy or is that nowhere near "10 times better"?

    • Number of significant digits... (Score:5, Informative)

      by jsimon12 ( 207119 ) on Monday February 23, 2004 @08:18PM (#8368845) Homepage
      Human=99.84
      New proggie=99.984

      So the human misses .16% and the machine only missues .016% hence the machine is 10 times better.

    • Re:2+2=3 (Score:3, Informative)

      by Celandro ( 595953 )
      No, you are just bad at math
      1 - .9984 = .0016
      1 - .99984 = .00016

      A factor of 10 in reduced error rates

      160 errors per 10 thousand vs 16.

    • Re:2+2=3 (Score:5, Insightful)

      by kfg ( 145172 ) on Monday February 23, 2004 @09:19PM (#8369441)
      Congratulations, Mon Ami.

      You have just unlocked the secret of virtually every news report that says "ten times more likely."

      To get cancer. To have a heart attack. To suffer from the heartbreak of psoriasis. Whatever.

      Yes, these numbers indicate "10 times better," and if you were to ask the reporter how likely am I to avoid cancer in both situations, these are the sorts of numbers he would show you.

      Eat health food and your chance of having a heart attack is 99.984%. Eat too many donuts and your chance of having a heart attack is 99.983%, 10 times worse!

      Always, always, always ask to see the raw numbers so that you know what "10 times worse" means.

      Then ask if the numbers were collected by phone survey. If they were, throw them all away and have donut and a cup of coffee.

      KFG

  • can it be used with SA? (Score:5, Interesting)

    by Chuck Bucket ( 142633 ) on Monday February 23, 2004 @08:16PM (#8368818) Homepage Journal
    can this be used with Spamassasin, or is a stand alone program? Does it need something like Amasis to run?

    CB

  • Who is sending that one? (Score:5, Funny)

    by ObviousGuy ( 578567 ) <ObviousGuy@hotmail.com> on Monday February 23, 2004 @08:16PM (#8368820) Homepage Journal
    If your email is indistuinguishable from spam by a human, perhaps the problem isn't the receiver. It's the sender.

    Forgive me if I don't feel any pity that some moron's email gets filtered to the junk bin because I couldn't discern it from spam.

  • To get this new spam filter... (Score:5, Funny)

    by Anonymous Coward on Monday February 23, 2004 @08:16PM (#8368827)
    Just enter a valid email address, and hit submit!

  • Better (Score:5, Interesting)

    by gid13 ( 620803 ) on Monday February 23, 2004 @08:17PM (#8368840)
    Well, it certainly sounds better than the pay-per-email "postage" idea. If postage hasn't stopped snail spam, why would it stop e-mail spam?

  • less thought for me... (Score:3, Funny)

    by Digitus1337 ( 671442 ) <(moc.liamtoh) (ta) (sutigid_kl)> on Monday February 23, 2004 @08:18PM (#8368852) Homepage
    ...and only one locked pod bay door per 6250, I like those odds.

  • Hmmmm (Score:5, Funny)

    by Anonymous Coward on Monday February 23, 2004 @08:20PM (#8368882)
    Probably used those same people who open viruses as test subjects.

  • i tend to think... (Score:3, Funny)

    by caino59 ( 313096 ) on Monday February 23, 2004 @08:21PM (#8368886) Homepage
    that i'm 100% accurate.

    maybe some of those people just dont know where their 'del' key is, or what it does...

  • Obligatory Q... When will mozilla/TB have them? (Score:5, Interesting)

    by sisukapalli1 ( 471175 ) on Monday February 23, 2004 @08:21PM (#8368898)
    I reached the conclusion of "two filters better than humans" by using two sequential filters:
    server side spamassassin, and a couple of simple procmail recipes. They have kept almost all the SPAM away.

    However, it is good to see such good techniques becoming available and we can hope to see them as straight forward usable tools.

    So, when will mozilla/TB (or your favourite server side or client side filter) get them?

    S

  • actually (Score:5, Funny)

    by Digitus1337 ( 671442 ) <(moc.liamtoh) (ta) (sutigid_kl)> on Monday February 23, 2004 @08:22PM (#8368913) Homepage
    it's not that humans are not as accurate, it's that 1 in X times we really do want a mini camera or free porn. It is what seperates us from those cold, heartless machines.... mini cameras and porn....

  • News story Headline (Score:3, Funny)

    by tacokill ( 531275 ) on Monday February 23, 2004 @08:23PM (#8368926)
    My Machine outhinks me!!"

    I've seen better stories in Highlights for Children

  • I'm sure they're great, but... (Score:5, Insightful)

    by LesPaul75 ( 571752 ) on Monday February 23, 2004 @08:24PM (#8368935) Journal
    I'm also sure that Yahoo's "SpamGuard" was great when they first introduced it. Now, It catches roughly half of all the spam I get. Why? Because people have figured out how it works and taken advantage of it. The same will happen with any content-recognition-based spam software. In the extreme case, even if a piece of software were 100% accurate at saying "This piece of e-mail looks like spam," then spammers would just make their e-mails look exactly like e-mail from one of your buddies. How could software ever tell the difference between:

    Hey, dude, check out this website I found. There are some hot naked chicks and stuff. Sweet.
    Signed,
    Your Buddy

    and

    Hey, dude, check out this website I found. There are some hot naked chicks and stuff. Sweet.
    Signed,
    SpamKiddy

    Even a human can't tell the difference. The only real difference is who they're from.
    • The thing with spam is that it's supposed to be a way for somebody to make money... i.e. they are trying to sell you something, be it directly or indirectly. I can't think offhand of an email I have recently received that could be misconstrued as trying to sell me something. From that simple viewpoint, spam can never look exactly like regular mail, because it has a different purpose.
    • Even a human can't tell the difference. The only real difference is who they're from.
      And that is all you need. I want website recommendations from friends, I don't want them from random spambots. That's enough for a human or a program to decide that one of those messages is spam and one is not.

  • Here's the real test (Score:3, Interesting)

    by Otter ( 3800 ) on Monday February 23, 2004 @08:28PM (#8368972) Journal
    I'm very happy with POPFile but there's one thing it just can't handle -- bounces from spam with my domain forged in the header when the original text isn't included. And how could it know? The response is the same whether it's to my mail or to spam. The domain is a clue, I guess, but otherwise it seems like an impossible task. I just let them be sorted into my inbox and delete them manually.

    If these filters can hit 99.99% with those, I'd be quite impressed.

  • Adaptive adversaries (Score:5, Insightful)

    by Pendersempai ( 625351 ) on Monday February 23, 2004 @08:28PM (#8368977)
    It's really easy to design an effective solution when the problem is purely mechanical or natural. As long as you're working with spammers who don't adapt, you can slice through their shitstorms very effectively.

    But when a single solution becomes mainstream, spammers will adapt to it. Bayesian filters tend to work very well, but now spammers are adding sprawls of randomly generated green-light text to offset the filter's score.

    Google found an excellent way to rank websites, but then it became widespread enough that webmasters began to game the system it had created. It's been playing catch-up ever since.

    Once the adversary begins to adapt, we lapse into the same cat-and-mouse game of technological barriers and counter-barriers that we've seen so many times before.

    • Re:Adaptive adversaries (Score:3, Informative)

      by kindbud ( 90044 )
      Bayesian filters tend to work very well, but now spammers are adding sprawls of randomly generated green-light text to offset the filter's score.

      That does not work. If anything, it makes the spam easier to identify, especially dictionary-salad-type spams that just list random words most of which real people hardly ever use in actual emails. Dictonary salad just gives the Bayesian classifier more spam terms to work with. The rest of the terms, the ones that are common in real emails, converge on a neu
      • It doesn't work for people who train their filters themselves. Indeed, with my well-trained SA install, my Bayes marks those spams as BAYES_99.

        But my old university, that has 40000 users, this has completely defeated their Bayesian filters. They say that the disk and CPU needed to have per-user bayesian training is prohibetively expensive, and they found that training for all users were doing more harm than good.

        So, we definately need more approaches to the problem.

    • But when a single solution becomes mainstream, spammers will adapt to it. Bayesian filters tend to work very well, but now spammers are adding sprawls of randomly generated green-light text to offset the filter's score.

      I can't see how that would change anything. The "bad" keywords are still in the spam. The gobbledy-gook words (usually short clips of random books/stories/something) are legitimate words, but aren't very likely to have a high coincidence of words found on in my legitimate email.

      I'm not

  • Could somebody explain this to me... (Score:5, Interesting)

    by heldlikesound ( 132717 ) on Monday February 23, 2004 @08:32PM (#8369013) Homepage
    I order all kinds of stuff online, wouldn't the receipt emails look like spam? My current spam solution is very simple:

    1. display my email online as little as possible

    2. use a number of addresses that all filter into one account, then filter by the sent-to address... this has turned up some VERY interesting results, for instance. I used dellorders@mydomain.com for an order from Dell, and NEVER used it or even typed it anywhere again, and started get spam about 6 months later, and I mean the nasty stuff, no just innocent stuff from Dell resellers...

    3. i built a rudementary filter that looks for viagra,free,debt,enlarge, etc... if the sender is not in my address book, and the email contains these words, it is sent to a "check these out" folder...

    How might a spam filter help me out without zapping confirmation type emails?

    • Re:Could somebody explain this to me... (Score:5, Informative)

      by caseih ( 160668 ) on Monday February 23, 2004 @08:56PM (#8369244)
      If you don't control the mail server to create aliases for yourself, you can also employ RFC-compiliant suffixes to your e-mail address. For example:
      foobar+dellorders@mydomain.com.

      • Re:Could somebody explain this to me... (Score:4, Informative)

        by Fnkmaster ( 89084 ) * on Monday February 23, 2004 @10:53PM (#8370161)
        Unfortunately, even though it's RFC-compliant, I've found probably half the sites I have to give my email address to won't grok the username+filtername@mydomain.com syntax. It's convenient when it works, but it doesn't work enough to rely on. No, throw-away spam-bait email addresses that you use for 6 months at a time for all online ordering and the like, then eventually trash when they get too spam-ridden are the best solution I know of.

  • This is just carp. (Score:3, Insightful)

    by corian ( 34925 ) on Monday February 23, 2004 @08:33PM (#8369021)
    Spam is what is defined by humans as Spam.

    To determine the accuracy of a spam detector, it is necessary first to come up with a sample of what is or isn't Spam. (I'd assume a human would do this?) So the best result we can get be evaluating humans is how often they agree with the result of the initial label.

    This figure probably won't be 100%. People have slightly different concepts of what mail is requested vs. unwanted, and what is advertising or useful information. So there is a valid possibility of disagreement.

    That doesn't mean humans can't do the job accurataly. (After all, if they couldn't, then the initial human-made labels would themselves be wrong and any data based on them meaningless!)

    If the training data is labeled with the same criteria as the test data, it is obviously possible that a trained system can acheive results which more closely agree with the test data. They are being trained on similiar data. But that doesn't mean that the system is MORE accurate at detecting spam than humans. It means that the system agrees with a particular human (or set of humans) more than other people do in a labelling of spam/non-spam.

    For all we know, the evaluators idea of spam is "wrong".

    • Re:This is just carp. (Score:5, Insightful)

      by sholden ( 12227 ) on Monday February 23, 2004 @08:50PM (#8369187) Homepage
      They are learning algorithms. For measuring their accuracy you have to assume that the data is correctly classified so you can see how they do.

      The point is that humans also aren't perfect. Have a person classify 10000 emails and they will make a few mistakes. Point out those mistakes, and they will say "yes, I got that wrong it is an email from my wife reminding me to pick up milk and not a spam trying to sell me printer ink, I must have been day dreaming."

      Just like if you give a person a document and say "find all the spelling errors" they will probably miss some. This is not because they have a different definition of how those words are spelt, it is because they made some mistakes.

      For the training/testing data, some double checking needs to be done to find the mistakes the human classifying it almost certainly made.

      It's a pretty normal situation in any machine learning application, you don't have to be perfect to be as good as a human - after all humans are only human.

  • The true test of a spam filter... (Score:5, Insightful)

    by GrpA ( 691294 ) on Monday February 23, 2004 @08:39PM (#8369072)
    Results of new spam filters cannot help but to be bogus... The true test of a filter is how well it works *after* all the spammers know how it works and try to circumvent it.

  • Image Noise Reduction and Machine Learning (Score:4, Interesting)

    by use_compress ( 627082 ) on Monday February 23, 2004 @08:42PM (#8369102) Journal
    I find it interesting that an algorithm that was originally for image noise reduction [ee.ubc.ca] found it's way to Machine Learning through a company whose purpose is to impliment noise reduction in audio. From my Googling, I think this is the first time anyone has used Baysian Noise Reduction in Machine Learning. Does anyone know otherwise?

  • Let's get this straight people! (Score:4, Insightful)

    by mabu ( 178417 ) on Monday February 23, 2004 @08:44PM (#8369131)
    client/server-side filtering does NOT solve the problem!

    The biggest problem with spam is the invasion of third party computers on the Internet. The ILLEGAL activity spammers perpetrate by breaking into machines, forging headers and hijacking servers.

    Any filtering method does not address this most serious problem, and even if you do not see any spam in your inbox, you're still paying for the bandwidth and system resources these spammers steal.

    Stop with the filtering algorhythms and take some of that energy and contact your local Attorney General, DA and FBI and demand that they prosecute these people who are BREAKING THE LAW.
    • Laws don't stop people from driving drunk*, and drunk drivers are in this country and even (by definition) driving out in public, in plain sight of everyone. How, exactly, would US law enforcement prosecute a $NATIONALITY1 spammer who's using a hijacked $NATIONALITY2 computer?

      Laws are fine, but what would *really* work is if everyone were filtering spam, and everyone tells all their newbie friends & relatives what spam is and installs blocking software for them. If sending 1,000,000 spams no longer res

  • One number not enough (Score:5, Insightful)

    by blamanj ( 253811 ) on Monday February 23, 2004 @08:54PM (#8369220)
    Saying an algorithm is x% accurate is not sufficient, because there are two kinds of errors: false acceptance of spam, and false rejection of non-spam. Personally, I'd settle for 90% false acceptance if I knew the false reject rate was 100% rather than have a program that was 99% at both.

  • How not to evaluate filters (Score:5, Insightful)

    by Daniel Quinlan ( 153105 ) on Monday February 23, 2004 @08:55PM (#8369228) Homepage
    The study referenced is:
    • On the author's mail (where all he does is probably talk about CRM114 and probably does not subscribe to many newsletters or non-technical mailing lists).
    • A pre-trained filter. It can't be compared apples-to-apples with any filter that doesn't require training.
    • Using his own filter on his own mail! Of course it does well.

    ... to mention a few of the problems. The statistics and methodology behind these claims are really questionable. I think both Consumer Reports and PC Magazine have both done better evaluations of spam filters (read that however you want).

    Also, I wonder how many people have actually looked at CRM114 and tried to use it.

    The really interesting thing about CRM114 is the windowed polynomial hashing technique used although there's some evidence that it can work just as well (if not better) on a much smaller window of only two tokens. I'm hoping someone will do a full exploration of the idea for SpamAssassin's Bayes module.

  • Do we buy viagra 0.16% of the time (Score:3, Insightful)

    by nri ( 149893 ) on Monday February 23, 2004 @08:58PM (#8369258)
    If we humans are only 99.84% accurate, then 0.16% of the time we will incorrectly think the email is real and buy viagra ? I don't think so.
    I read the email and delete it. Exactly the same as the spam filters do it, only MORE accuratly. I think the tests applied would have been between a human reading the header of an email and deciding whether to open it or not verses the spam filter making the decision for us. BUT the spam filter makes its decision by opening the email. Therefore to have a proper comparision I should be allowed to open the email as well before I make the decision. Therefore I am 100% accurate.

  • The CRM114? (Score:4, Funny)

    by tramm ( 16077 ) <hudson@swcp.com> on Monday February 23, 2004 @09:00PM (#8369265) Homepage
    I bet it allows messages from General Jack D Ripper [imdb.com] or any email that contains the secret phrase "purity of essence", "peace on earth" or "precious bodily fluids".

  • Human accuracy doesn't scale linearly (Score:5, Insightful)

    by Kaboom13 ( 235759 ) <kaboom108@NOsPAm.bellsouth.net> on Monday February 23, 2004 @09:14PM (#8369396)
    I'm not surprised a filter beat the human, considering the study used a sample of 5849 messages. As the sample size increases, the filter's accuray will increase, and the human's will decrease. Furthermore the higher the spam/real ration, the better the filter will do in comparison to a human trying to sort at a reasonable speed. The reason being humans tend to skim, and rairly actually read entire subjects, much less messages. Give a human 5000 messages and an hour and he will probably make some mistakes. On the other hand, in 10 messages, the human will probably be 100% correct. Most email filters rely on this already, as they tend to err on the side of caution. With the bulk of the spam taken out, it is not a burden to have the human check the iffy bits. Furthermore the type of email can mislead humans. A business-type email sent to someone's personal email is much more likely to be mistaken as spam, and vice versa. The main disadvantage of automated filtering is people generally have an idea of when a really important e-mail is going to come (the type that false positives are completely unacceptable) and who it will be from.

  • Spot the reference... (Score:5, Informative)

    by Maj. Kong ( 215009 ) on Monday February 23, 2004 @11:09PM (#8370280) Homepage
    CRM114 was a piece of encryption gear in Major Kong's...err, my B-52 in the movie Dr. Strangelove [imdb.com]. It allowed only properly coded messages to be received by the crew. When the Soviet SAM detonated near the airframe, the CRM114 was damaged and the crew could not get the recall order.
    Kong: (announcing through headset intercom )


    This is your attack profile: to insure that the enemy cannot monitor voice transmission or plant false transmission, the CRM114 is to be switched into all the receiver circuits. Emergency phase code prefix is to be set on the dials of the CRM. This'll block any transmission other than those preceded by code prefix. Stand by to set code prefix.

    ObKubrick: In 2001: A Space Odyssey, one of the pods was marked with the designation CRM-114. And in Clockwork Orange, Alex is injected with serum 114. I suppose CRM-114 is to Kubrick as THX1138 is to Lucas.

    Dobly, on the other hand, is from This is Spinal Tap [imdb.com], a mispronounciation of "Dolby" by David St. Hubbins's girlfriend:

    Jeanine Pettibone: You don't do heavy metal in Dobly, you know.

    Not to mention that it probably avoids trademark infringement (though I wouldn't put it past Dolby Labs or Thomas Dolby to raise a stink).

    Maj. Kong

  • Dolby-type noise reduction algorithm called Dobly? (Score:4, Interesting)

    by omeomi ( 675045 ) on Monday February 23, 2004 @11:41PM (#8370471) Homepage
    Dolby noise reduction works by filtering a spectrum into a bunch of bands, each of which are compressed (in an audio sense, not in a digital sense), and recorded to tape. On playback, they go through an expander...how does that concept translate to spam filtering? It can't be "dolby-type", that doesn't make any sense...
  • ...are still the only real solution to the issue of trust, reputation, and accountability on the Internet. We need it for so many other things in addition to guaranteeing email legitimacy.

    If every user or at least every server had a key and we all signed each others keys creating a web of trust and only accepted signed and trusted mail the spam problem would be solved. I really dislike the way SSL certificates are handed out. A central CA is a very bad idea due to the cost and browser lock-in issues etc. With GPG and web of trust if you want to run a mail server you need to talk to a friend who is already running one and get them to sign your key. Perhaps we could even use DNS to propagate and cache the keys and sigs. If you sign a key that turns out to be a spammer you better revoke that signature fast before the person upstreeam from you revokes yours. Problem solved. Now if only we could get the big guys to go along with it...

    • Overkill (Score:3, Interesting)

      by mdfst13 ( 664665 )
      We don't need to trust the *person* sending the mail. It would be sufficient to trust the machine that is doing so.

      Look at http://spf.pobox.com/ which is sufficient. With SPF, you know that if you are getting SPAM saying it is from @ultraviolet.org, then it really is from @ultraviolet.org (or at least someone who ultraviolet.org trusts).

      Your solution requires a certain level of technical proficiency (setting up and managing the key) of *all* participants. SPF's solution only requires technical proficie

  • Share the luxury (Score:5, Interesting)

    by bigberk ( 547360 ) <bigberk@users.pc9.org> on Tuesday February 24, 2004 @01:40AM (#8371106)

    Having such a powerful statistical spam filter is definitely a luxury. I have no difficulty believing the accuracy values presented here. I have had experience with spamprobe, CRM114, bogofilter, spambayes, and spamassassin and all of these do an amazing job to the point where spam no longer exists (for you).

    Which leads to me plug a little project called WPBL [pc9.org] that uses exactly these types of statistical spam filters to spot spam sources in a distributed fashion. Each project member uploads hourly the IPs they see relaying spam and non-spam, where the 'decision' is made by these extremely reliable filters. This effectively converts your regular mail account into an intelligent spam-trap that feeds a central blocklist.

    The more members we get, the better we can identify active spam sources around the world. This information is then used by some sites for quite large-scale blocking [dnsbl.net.au]. Since you're doing all this filtering processing anyway, why not also share "what you learn" (the IPs that are spamming you)?

    If this grabs your interest, read up on the reporting scripts [pc9.org] or alternatively, the open WPBL data upload protocol [pc9.org] if you want to code your own report generator. Bandwidth usage is minimal.

  • Well (Score:3, Insightful)

    by DRACO- ( 175113 ) on Tuesday February 24, 2004 @02:17AM (#8371251) Homepage Journal
    Well if the human was given the chance to read the body text as well like the filters do, then they would be 100% able to delete their own spam.

    DRACO-

Slashdot Top Deals

The question of whether computers can think is just like the question of whether submarines can swim. -- Edsger W. Dijkstra

Close