Filter-foiling Gibberish Becoming A Spam Staple 606
hcg50a writes "Wired has a story about the random words which have recently been appearing in spam. Antispam experts agreed that this isn't a brand-new technique, but said the addition of potentially filter-foiling gibberish is rapidly becoming a common component of spam."
gibberish... (Score:4, Funny)
Re:gibberish... (Score:5, Funny)
Re:gibberish... (Score:3, Interesting)
Actually, I avoid deleting my spam. I have an archive now of over 270MB of spam that I can use for a training set for whatever filter I might intend to deploy.
That archive has more than just spam, mind you. It also has all the virus/worm email I've received over the years as well, such as the "Internet Email System" informing me of an undeliverable message, or "Microsoft Corporation" providing me a convenient, easy to click "December 2003 Internet Update" or whatever.
*sigh*
--Joe
Re:gibberish... (Score:5, Insightful)
Re:gibberish... (Score:3, Interesting)
Why was this marked Redundant?
Maybe I missed someone else pointing this out, but it's a very important point. The spammers will only stay in business until it's no longer profitable. The technological solutions beat the legislative ones right now, but getting the word out to people that buying from spammers only encourages spam would really help too.
Re:gibberish... (Score:4, Interesting)
Not necessarily. I'm sure most of those people (had to backspace over a few epithets) who spam Make Money Fast either lose money or get into legal trouble. But the damage is done (to me) before they learn that it won't make money. I think the driving force is selling spam services to gullible clients like these. (Not including the industrious Nigerians who seem to take a more personalised DIY approach.) Even if someone DID want penis-enlarging cream, I think by now they'd have a source of supply, that market must be pretty saturated by now.
[ADV] (Score:5, Funny)
apxxmyohofmnoatn fmkpo oixv a z gjs sc dnbxgbidlaaatooab yqlrwtta dupg o vx j n vyz aae xvm
You blew it. (Score:5, Funny)
Re:You blew it. (Score:3, Funny)
Well...the idiots out there have to know they're going to be paying for something, don't they?
Re:You blew it. (Score:3, Funny)
Should SPAM filters check for correct spelling/dictionary check? Whoops, scratch that - wouldn't want to kill Slashdot replies.
What I don't understand (Score:4, Interesting)
What I don't understand about this type of spam is that often it doesn't contain any actual advertisement, just three or four lines of random words, and the end of the email right there.
I don't get it. If you're not selling a product, what is the spam for?
Mind you since TMDA, I haven't been seeing any spam anyway.
Re:What I don't understand (Score:5, Informative)
I was very irritated by that, too, until one day I was testing the HTML viewer of an e-mail client.
Re:What I don't understand (Score:5, Interesting)
[What I don't understand about this type of spam is that often it doesn't contain any actual advertisement, just three or four lines of random words, and the end of the email right there.] Actually I was viewing the source of the whole email, not the text part.
I too see this sometimes. You're not crazy (at least with regards to this). I've looked at the full source, but still can't figure out what the goal is. My best guess is either they are fishing for bounces (ok, these are bad addresses; the ones that don't bounce may be good addresses), or the spamming software has a problem (bug or is misconfigured).
Re:What I don't understand (Score:5, Informative)
Don't ever do that, all spam has forged headers. You're just making life hard on someone who had their address sold.
I work for a big company, an icon the the computer business. Our mail servers get spammed a lot. We often have typical user names grafted onto the From or Reply lines. Since my user name is pretty damn common, and some of my work mail aliases are TLAs, I look at a lot of spam. When I read the headers (in a text file, not easily spoofed mail software), almost always the senders domain is not even close to the domain of the spamming machine. Go put the IP addresses into dnsstuff.com, and compare that to the hostname. These turds hack the sendmail.cf file of the spamming machine. "SallySmith@aol.com" probably did not send spam-mail from a ".kr" ISP.
Re:What I don't understand (Score:3, Interesting)
Whereas it might be true that all "spam" has forged headers, not all email which passes the 5.0 threshold has forged headers.
Also aren't other mail servers supposed to check that the envelope sender matches the host it's being sent from?
Re:What I don't understand (Score:3, Informative)
Re:What I don't understand (Score:4, Interesting)
Returning suspected spam might have a small adverse effect on the legitimate holders of forged addresses, but silently deleting suspectred spam adversely affects everyone by causing misclassified messages to be silently lost. The practice of bouncing spam doesn't increase collateral damage, it prevents it. Automated processes must cause mail to either reach its destination or be returned to its purported sender. Otherwise legitimate mail will get silently lost. That's collateral damage.
This balance of burdens is fair too. Fake bounces are much easier to filter than ordinary spam. Even if the bouncing MTA engages in the unfortunate practice of sending bounces that don't contain the original message you can still filter all fake bounces with 100% reliability. Simply send each of your outgoing messages with a unique tagged, timestamped envelope sender address. Bounces which arrive at other addresses are always in response to forgeries and can be safely discarded.
Re:[ADV] (Score:5, Funny)
Re:why not filter out 1337 sp3@k? (Score:5, Informative)
Why bother? A decently trained Bayesian filter will be able to recognize a spam that contains a misspelled word or two, or one that contains substitutions of similar characters. Then it will learn that those modified forms are a very strong indicator of spam. As Paul Graham [paulgraham.com] (the main early advocate of Bayesian Filters) has pointed out [paulgraham.com], there are legitimate reasons why you might see a mention of "Viagra" in your email, but no legitimate reason that you would see "V1agra", "\/iagra", "Vi@gra", or the like. Instead of slipping by my Bayesian filter, those variants actually stand out as particularly strong spam indicators.
Re:why not filter out 1337 sp3@k? (Score:5, Interesting)
1337 speak isn't a big deal. It's definitely filterable.
I've begun seeing chunks of text appearing in messages that are like legitimate mini-messages in and of themselves. Sort of like a counter weight. I don't think the aim is to pound Spam through the filters now, because what's happening is spam is getting slightly lower ratings each time while legitimate messages are getting slightly higher ratings.
In other words, the spam probably won't ever be legitimate, but it's making me lower my threshold for what is spam more and more. Eventually, I'll get to the point where some legit messages will cross over into being labeled as spam and spam will go through legit because the thresholds will be so close together as to practically overlap. It's also killing my ability to keep a spam trap that I can use to quickly train filters.
Whether this scene will actually play out and the "plot" will be succesful or not remains to be seen, however.
Re:why not filter out 1337 sp3@k? (Score:3, Interesting)
Examples from my corpus:
VIAGRA: 99.797%
V!AGRA: 99.9999%
AGRA: 99.9999% (from things like VI.AGRA)
IAGRA: 99.9999%
PORN: 98.573%
P0RN: 99.9999%
PR0N: 99.9999%
Plus, the trick is looking for things that give away spam that aren't just words. I call them "characteristics." For example:
Various pharmacy related terms: 99.9999%
HTML using % escape sequences: 98.789%
Http:// references that don't
Re:why not filter out 1337 sp3@k? (Score:5, Funny)
Pay me six figures a year and I will sit in a chair and do it for you manually.
Well... (Score:4, Interesting)
You'll laugh from it... (Score:5, Funny)
Re:I receive this today (Score:3, Funny)
Spamkiller doesn't care (Score:5, Interesting)
Re:Spamkiller doesn't care (Score:5, Insightful)
What good is that when somebody spams you for Gen3r@c v|agar@?
Re:Spamkiller doesn't care (Score:4, Interesting)
Gen3r@c v|agar@
Gener@c v|agar@
Generic v|agar@
Generic viagar@
Generic viagr@
Generic viagra
That's an edit distance of 5, pretty large, but still findable with a little approximate matching, especially if it's weighted, to recognize the similarity between @ and a, or i and |.
Most spam contains repeated phrases 40+ characters long. the mistake is to use word-counting techniques which ignore phraseology.
For instance, here are some phrases from spam, circa one year ago:
Please fill out the form below for more information
To unsubscribe
To remove your
in the Marshall Islands
Please allow 48-72 hours for removal
to this email with REMOVE in the
the Northern Ratak
the information
thousands of dollars
that you will
this list, please
this advertisement
this email in error
this message, you may email our
this transaction
of thousands of
of EnenKio and
of Eneen-Kio Atoll
of His Majesty
our mailing list
out 5,000 e-mails each for a
opportunity to make
Re:Spamkiller doesn't care (Score:4, Insightful)
I'm pretty sure that the big worry is about third party filtering. If I install a spam filter, that means that I don't want to see spam and am unlikely to buy something advertized therein. If my ISP installs a spam filter, it removes spam to everyone, including the idiots who might actually buy something from a spammer. Since my ISP theoretically might be using the same technology in their filter that I'm using in mine, it would still make sense for the spammer to work on defeating my filter.
Re:Spamkiller doesn't care (Score:3, Informative)
*Someone* does, but not the parent to this. SA *does* "incorporate Bayesian analysis techniques," and some of its rules are about handling the results. You can score those rules to 0 for non-Bayesian filtering, or score everything else to 0 for pure Bayesian.
Re:Spamkiller doesn't care (Score:3, Insightful)
What makes you think they have any sales (of the advertised product). I would guess that almost all spam (maybe excluding for pr0n sites) is either being sent by a MAKEMONEYFAST sucker or by a professional spammer who charges such suckers to send their spam out. The first set never make any sales, dissapear and are replaced by the next moron, the latter have their money sales or not.
But then again, Joe Si
Re:Spamkiller doesn't care (Score:5, Interesting)
V: 76.9% Spam score
I: 47.2% spam score
A: 68.8% spam score
G: 72.2% spam score
R: 72.2% spam score
On balance, if I get a message with the individual "words" of V, I, A, G, R, and A, that's going to be leaning towards spam.
That's the beauty of Bayesian. Anything the spammers do will eventually come back and bite them in the butt. Even some of the "random words" they are starting to use are getting high spam scores:
WHEREUPON: 99.9999%
NEOCONSERVATIVE: 99.9999%
LIBERAL: 74.3%
LIBERTY: 84.0%
MEGATON: 99.9999%
METHANE: 99.9999%
These are just a few of the "random words" I found in recent spams and, interestingly, the random words they are using are actually INCREASING their spam probability.
Statistically, it's a lost cause for the spammers, they just don't realize it yet.
Re:Spamkiller doesn't care (Score:4, Interesting)
True. Although an obvious caveat of using Bayesian to filter is that you HAVE to train it. In the anti-spam service I use (see tagline) it defaults to NOT using Bayesian. If you turn Bayesian on it specifically sends you an email reminding you that you MUST train it or things will actually get worse.
But you're right, a misused Bayesian filter might actually be worse than no Bayesian filter at all. But that's the case whether or not spammers insert random words.
There are ways to poison Bayes-filters that are better than this, and that may well be effective. If you sit down and think about it, I'm sure you can think of something too. I'm not going to write them, because it will be too easy for spammers to implement. Fortunately, spammers are stupid, and that buys us some time, but we still need more options.
Let's talk about them. We're not going to come up with anything that spammers can't come up with so I don't think we're going to make things any easier for them or give away the farm by discussing it publically.
I personally have thought about it and I'm unaware of how they could poison Bayesian statistics. I only see two approaches, theoretically. 1) Make your spam get a lower Bayesian score so it gets through. 2) Make non-spam get a higher Bayesian score so it gets caught as a false positive.
Approach #1: Short of going to the "spam of the future" predicted by Paul Graham, I don't see any way for spammers to really get a lower spam score.I've seen entire sections of the Constitution embedded in spam that still got a 98% spam score. The only way spammers are going to get a lower spam score is by doing things like using the names of my friends, using words related to topics I often discuss, etc. And that's just not possible. Like I said, they might get an occasional lucky shot but what gets through to me most probably won't get through to you. I just don't see any way for them to reliably get past a significant number of Bayesian filters.
Approach #2: Poison the Bayesian stats such that non-spam mail gets tagged as spam. I'm pretty convinced this isn't possible, either. Again, they'd have to heavily use words that are specifically non-spam for the receiver such that the spam rating for those words increases so high that it is considered spam. But if the words are heavily used in both spam (trying to poison the stats) and non-spam, it's going to float to a middle position, like the word "THE" which has a 53.2% chance of being spam (and that's only because 92% of my mail is spam so a neutral word is usually slightly over 50%). But neutral words are completely ignored by Bayesian--only the "most interesting" are considered, those that are 99% spam or 1%--THOSE are the words that define whether or not the message gets scored as spam or not. Plus if they knew which words to poison, those are the same words they could use to get their spam past the filter to start with... so poisoning the filters is pointless anyway.
I really don't see how they can get around it. I'd be interested in your views. If you really think it's dangerous to talk about it in public then let me know and I'll email you at your mangled address above. Is that your correct address?
Sometimes it isn't random words (Score:3, Funny)
Re:Sometimes it isn't random words (Score:3, Informative)
Who would have thought Project Gutenberg [gutenberg.net]'s biggest use would be for hawking herbal remedies?
Re:Sometimes it isn't random words (Score:3, Funny)
Bigger beavers are the very reason for enlargement (Score:5, Funny)
I've also had some Alice, but today I learned about North American beavers. I had no idea they were so large.
That's exactly why you need to ENL4R9E `/U0R P3N1S!!!1!1 because North American women have 1arqer beavers and thus require a bigegr PE/\/i5 to st!mu1ate them.
New use for Project Gutenberg (Score:4, Interesting)
It would also help spammers to write better pitches. Use real words, actual English but put it in narrative real world sceneario format. So it reads like someone you know telling you how they use such and such a product.
"I went up the cabin last week with my girlfriend and tried out those new pills I heard about while I was there."
There's pretty much nothing in there that would be filtered. And then a slight plug of the product name with a link and you're done. It's also Marketing 101 that the less of an ad sounds like an ad the more effective it is.
But none of that thwarts my method which is to filter based on the URLs of links found in spams.
I get virtually no spam with a Mercury rule file that's all of 23KB and grows very slowly as spammers use new domains to host their product pages.
Ben
Re:New use for Project Gutenberg (Score:3, Funny)
Oh, that has never ever been done in advertising... =)
How about stuff like
And the angels, all pallid and wan,
Uprising, unveiling, affirm
That the play is the tragedy, "Impotence,"
And its hero the Conqueror Pill.
Or:
Tis now the very witching time to have bad credit rating,
When the stores yawn, a
Just great... (Score:5, Funny)
I don't get it, really (Score:5, Insightful)
I really understand this part: going after people who are taking active measures against your enterprise due to their disinterest. Why bother to market to them at all? Is the rate of return worth all the ill will, DOS attacks and legislation?
Re:I don't get it, really (Score:5, Insightful)
Feature added (Score:3, Insightful)
Now a days however ISPs (most notably Earthlink and MSN) advertise spam blocking as a feature.
If people wanted this stuff you'd think non-filtering ISPs would advertise "You get ALL your e-mail".
But back to the original point. Spammers have used misleading topics in e-mail if only to make sure you don't delete the message. That and creating spam lists based on people who DO NOT like spam or of people who have manually opted o
Re:I don't get it, really (Score:5, Interesting)
In addition to living in their own criminally delusional world, spammers often don't spam for themselves but work for others. They get paid by their, er, client for each message sent, it doesn't matter to them whether it's wanted or not.
Plus, there's always that .001% of suckers to keep the biz going if the cost of sending is close to zero.
Re:I don't get it, really (Score:5, Insightful)
On every spam thread on Slashdot, there's someone complaining that technical measures won't solve the problem, and another saying legal measures won't solve the problem. The answer is that you need both: technical measures to assure the identity of the sender -- both spammer and sponsor -- as well as legal measures to provide for punishment.
Re:I don't get it, really (Score:5, Insightful)
Re:I don't get it, really (Score:3, Insightful)
Re:I don't get it, really (Score:3, Insightful)
It's possible, if not likely, that some of the spamware authors are doing it for the challenge. Some of those guys are allegedly pretty good programmers, and I suspect that many of them are essentially hackers with no sense of morals. I could easily imagine somebody like that trying to figure out how to bypass spam filters just because it was a challenge, not because he actually expected any particular rewards for it. It's like trying to break into the computers in the Pentagon; it's stupid and illegal b
It's not gibberish, it's steganography (Score:4, Interesting)
Parent post is not offtopic (steganography) (Score:5, Insightful)
Spam is a perfect carrier for steganographic data since it's broadcast to millions of people and nobody can fall under suspicion merely by receiving it. When the government wants to monitor people's communications to search for steganography, when they don't do anything about spam, the purpose of the monitoring is probably not the stated one.
Why? (Score:3, Insightful)
Granted, this may get them past the filters, but if somebody's gone through the effort of setting up a Bayesian filter, they're not going to buy your product even if you get into their inbox. It seems like a waste of everybody's effort, and I mean including the spammers.
Re:Why? (Score:3, Insightful)
* valid sender domain
* html links to external images etc, or large amounts of html in general.
* blacklisted servers/relays
Re:Why? (Score:3, Informative)
Random strings of text are used to get through the internal checks that large ISPs run on their message traffic.
Yahoo, Hotmail, etc have "bulk email" type folders. In addition to using spamassasin type techniques, the filter scripts that put messages in these folders will check to see if the same message is being sent to multiple addresses. If this is so, it raises a flag and someone checks to see if its a genuine
Simple Solution... (Score:3, Interesting)
Re:Simple Solution... (Score:4, Insightful)
What I'd be interested in... (Score:4, Interesting)
Spammers seem to have a lot of success when they're emulating more legitimate sources like Ebay, Microsoft, etc., but I get spam now that can't even seem to decide what it's selling. The subject line says "get rid of mortgage payments" and the body is selling "V.I.A.G.01331.A." I'm not even sure what I'd be getting if I were dull enough to actually click on anything in the message. Heck, I'm not sure if even the SPAMMERS know.
I'd be interested to know if these spams are as successful as past efforts have been.
Not an effective technique (Score:4, Interesting)
Re:Not an effective technique (Score:3, Funny)
We already have tools to stop this (Score:3, Insightful)
It'll take more processing power, and lead to spammers following proper grammar in their pseudo-nonsense, but it's the way to raise the bar against this attack (making those spammers that can't clear the bar out of luck).
Reminds me of a Dr. Seus book...
RD
My Bayesian filter is slowing becoming a whitelist (Score:4, Interesting)
I know that whitelists aren't the answer, but then nothing short of immediate execution of spammers is.
The Grammar Filter (Score:3, Interesting)
Should we add a grammar-filter to the list of things we look for it spam?
A large amount of incorrect grammar would increase the chances of the file being caught in the spam filter.
Of course, this would lock out most of AOL users from writing email... But is that really so bad?
Re:The Grammar Filter (Score:3, Funny)
Bayes filters deal with it fine (Score:5, Informative)
Re:Bayes filters hubert balloons c6as6g89y9aigah98 (Score:3, Informative)
The problem with this technique (Score:5, Interesting)
For example, take the word "Byzantine." This is a very non-spammish word. However, if you've never received a legitimate email containing the word "Byzantine," your Bayesian filter will not have it in its dictionary, and the word will be ineffective in "tricking" the filter. The red herring words only have an impact if they are relevent to your actual mail sample. Since everybody's email communication is different (some of us are programmers, some of us are literature majors, etc.), this is a real sledgehammer approach to defeating the filters -- and it's extremely ineffective.
This technique just proves that spammers don't understand the theoretical underpinnings of current Bayesian anti-spam methods. Otherwise, they'd be using much more common words as red herrings, instead of these extremely rare, and therefore insignificant, words.
I personally use a spam filter of my own design which is based on information-theoretic and neural network techniques. It kicks the shit out of spam, even the messages that include these stupid red herring words. The spammers once again prove that they are morons, incapable of understanding how anti-spam technology actually works.
Re:The problem with this technique (Score:5, Interesting)
When an adaptive filter sees a rare word in a spam, it is likely to assign that word high spamminess. Problem is, the next time you see that word is likely to be in a piece of ham, resulting in a false categorization of a piece of ham as spam. The user cost of such an assignment is very high, and so users will be forced to look at their junk mail...which is, after all, what the spammers want.
Re:The problem with this technique (Score:4, Informative)
Re:The problem with this technique (Score:3, Informative)
Re:The problem with this technique (Score:3, Informative)
I'd love to -- in fact, I've even got my own website registered for it -- neuralnw.com [neuralnw.com] -- but development has stalled recently, and you'll find no trace of the program on the website. The filter, or at least a rudimentary version of it, is available if you know where to look for it. We published a paper at USENIX back in June covering this program. Since then, I haven't done much development, because frankly, there are better ways to spend my time
Grammar Check and Spell Check... (Score:5, Insightful)
Sure, a few strange words might be a name that's not in the filter yet, but pure gibberish should be a red flag that either somebody's cat walked on the keyboard, or there's spam going on here. Heavy use of "non-spam" words can override to indicate it's good mail... but a poorly composed mail that doesn't use language seen in friendly mail is highly likely to be spam....
Re:Grammar Check and Spell Check... (Score:5, Funny)
Re:Grammar Check and Spell Check... (Score:4, Funny)
Re:Grammar Check and Spell Check... (Score:4, Interesting)
Apparently you've never gotten emails from either a:
1) 14-year old girl
2) Gamer
3) UNIX sysadmin describing a sendmail
Yikes.
As if spam wasn't a big enough waste of bandwidth (Score:3, Insightful)
--
Still looking for an email replacement...
If someone made a gibberish filter? (Score:3, Funny)
Re:If someone made a gibberish filter? (Score:3, Funny)
Different Techniques (Score:5, Interesting)
The article doesn't do a good enough job of explaining the different techniques in use.
First, hash busters. Yes, spammers are loading a random jumble of meaningful words in meaningless sequences into their spam, usually in the plaintext message body of a message with HTML content (i.e., you get hash buster - html message with spam content - hash buster). So HTML-aware clients (the main clients targeted I'm sure are AOL and Outlook Express) show the spam message, but not the hash buster. I'm guessing that this is specifically targeting bayesian filtering tools at AOL (anyone know if AOL is using a bayesian filter?); it works by introducing words that would not be found in a spam corpus in greater numbers than those that would.
Second, noisy spelling, like v1@gr@. Obviously this is also intended to defeat regex-based filters like spamassassin. If you vary your cliches enough, and you introduce very strange, but easy-for-a-human-reader-to-recognize spelling variants, you make it much more difficult for filter writers to write effective regexes.
The real problem will be deliberate poisoning (Score:5, Interesting)
However, what constitutes "non-spam" is not as unique as most people think, as I've examined here [jerf.org]. If they figure out how to deliberately put in hammy words, Bayesian will fall.
I feel OK posting this because I freely admit to this point I've overestimated them; I'm sure spammers have read that piece, and to date they have been too stupid to figure out what I said in plain English. But sooner or later one of them is going to figure out.
There's a strong core of "ham" that is "ham" for everybody, and sooner or later they're going to start abusing that.
And if I may forstall one objection... "But you don't understand Bayesian, it's [awesome for some reason and can't be beat ever, by anybody]" - I'll listen when you've actually written a program to examine filters yourself, OK? I understand it pretty damn well. It'll take more then bald assertions to convince me I'm wrong, I've done actual research, in the original sense of the word.
Re:The real problem will be deliberate poisoning (Score:3, Insightful)
Besides, if they could guess what your ham looked like, then they wouldn't be spammers... they'd be advertising folks pulling in 7 figures.
Re:The real problem will be deliberate poisoning (Score:3, Insightful)
But how would you sell more inches on your male member enhanced with V*@gra to make money fast watching celeb teenie nymphos doing it on the farm while only using ordinary non-spammy words?
There are only so many ways to ge
Re:The real problem will be deliberate poisoning (Score:3, Interesting)
Language is often a big indicator; since spam is aimed at a particular langauge group I don't consider it much. The fact my filter marks Japanese or Korean messages as spam is almost irrelevant, in a way, since I can't read it anyhow and it's easily dismissed.
But there's this common
/usr/share/dict/words (Score:4, Interesting)
Why not check the subject or first few lines of plain (not html) text and see if 80% of it is in
Slimier than slime . . . (Score:5, Interesting)
In other words, if your Bayesian filter accepts those, based on your past decisions, it will detect the spam. If you reject the spam, you reject these communications as well.
Good filtering practice would dictate that one reads the junk box carefully enough to find both false positives and negatives. But the sheer bulk of mail that ends up in the junk box makes this unfeasible for many.
I have started letting these particular kinds of spam through, manually categorizing them (many words of random strings, dictionary vocabulary attack, positive phrase attack) in the hopes that filtering technology will soon advance to the point where these can be used as inputs to a more intelligent system.
Of course overhauling the mail system is a prerequisite to solving any of this long-term. For once I don't mind D. J. Bernstein's Internet Mail 2000 proposals. Of course there are other proposed systems, none of which has enough momentum to start a slow steady change. The end result of any non-consensus system will be to fragment the worldwide network of Email into competing, noncompatible systems that need to communicate through some kind of loophole or gateway. Back to FIDO-net days.
I see this too (Score:5, Interesting)
Agreeing with this article, over the past week or two I have seen excessive about of spam being missed by SpamBayes, even after marking them as spam for improved filter, they continue to hit the inbox whereas previous absolutely no spam made my outbox. Additionally, there may have only been 2 or 3 emails marked as possible spam when they were not. And zero items mark as definite spam that were not.
SpamBayes has worked great previously, but now even it is falling short.
I feel as the spammers manipulate the conents/context of the spam, it will eventually become impossible to determine the difference without physically looking at 500+ email daily.
My primary use of email is business and not personal, therefore I cannot risk missing a client email, payment, question, etc... I've also see a progression of clients having MY emails deleted or caught in spam filters due to the business aspect and requests for payments. I feel this is primarily due to the comparison of too-often-common-phrases that a spam email and a business email contain. Such things as Click here to submit payment, or Buy these Products, Overdue etc... Even though all clients I email are only clients that contact me. I never cold-email anyone.
More spammer are using this random text as the only text in the subject and body, and using an image as the content of their email, which makes scanning even more complicated, if not impossible.
Being on the net prior to what is is today (going on 20 years), I often wonder how much control the spam actually has over the net in several aspects
- If spam were to disappear, will overhead costs decrease that greatly in order for ISP's to pass along higher saving to the consumer?
- If Spam were to disappear completely, how much faster would the Internet be?
Has anyone ever done a study to determine how much effect spam has on degrading the net, and what would it be like if all spam was gone tomorrow?The next attempt (Score:3, Insightful)
Insert four or five lines of valid extra text -- lines from books, selections from recent USENET postings, etc, etc -- into the spam. Make the selection semi-random. Now do it 100 times and send 100 copies to each person on the mailing list.
One of them will get through. And the spammers will continue to work.
A method for removing spam from your life. (Score:5, Interesting)
1. Create manual spam filters (NOT beyesian filters) in your inbox called "Friends and Family", "Work", "Services", "logfiles", and any others you find you need. Each category applies to a broad type of email address you'll receive email from. Then create a subdirectory in your inbox for each of these filters (named the same way, naturally).
2. For each filter, build a list of people who are allowed to email you. For example, your ISP, your bank, and your phone company would probably be added to services. Just add the email address they send their messages from to the list.
3. For each filter, have the filter move messages matching the filter (From equals ) to the correct subdirectory for the filter. Then stop processing for that message, so it doesn't get interpereted by other filters. Think of this as an analogy for ipfilter or ipfw in your firewall setup -- only you're filtering emails instead of packets.
4. Finally, DELETE EVERYTHING ELSE in the very last filter.
You USE this approach by doing a quick scan of the deleted items folder to see if anything is interesting. If not, just clean out those deleted items. It's a one step operation, much easier than selectively deleting a hundred emails one at a time.
Then, you scan each of the folders you set up, IF the folder has picked up an email, focusing only on your REAL email.
This approach has saved me a HUGE amount of work lately. My life is a whole lot easier, and it's way easier than trying to train a Beyesian filter. If I don't know you, you can't get too much of my attention.
It's all about being on the list, sort of like getting into a nightclub...
Re:A method for removing spam from your life. (Score:4, Funny)
Dad
Re:A method for removing spam from your life. (Score:3, Insightful)
Twice in this thread, I see you talking about training the bayesian filter. You seem to think this is something of a burden, like training a big dog...
I think you misunderstand how easily one trains the current Mozilla email client's bayesian filter.
Day 1:
1: the mail comes in, spam included.
2: one of the inbox columns is a blue 'recycle' lookin' symbol. It is a toggle that acts like the 'new' indicator column, and a click on it turns state on or off.
3: glancing through the list, one clicks o
Simple trick that is semi-efficient (Score:5, Interesting)
hostz300001.com/ads/viagra.jpg
Or whatever. I've cut down from 50 spams to about 3 or so a day by doing that.
I bet a bayesian filter would work nicer but unfortunately I'm too lazy to mod the mail setup [that isn't mine] to get one installed..
Tom
I use that method (Score:3, Informative)
Mercury Mail's session logs indicate a closed connection to indicate where e-mails begin and end but if you're using something else there's a RinetD mod with source which logs e-mails in such a way so that ripping through them is easy.
My filter is all of 23KB and I get virtually no spam. I update every once in awhile when a spam gets through.
I also have a couple sub-domains that point to a spamcan on my home connection which I use to bait spammers so I can preemptively filter them ou
Word Salad (Score:3, Interesting)
John.
How I deal with spam (Score:3, Interesting)
I don't believe in content-based filtering. We have a strict policy of not examining in any way, shape, or form, the content of any e-mail on our network.
We deal with spam by implementing an array of fully-tested, fairly conservative relay blacklists which block the inbound SMTP connection before the junk mail is even transmitted.
In more than two years of operation, we've only confirmed about six legitimate e-mails that were blocked, and we handle tremendous mail volume. It's an easy matter to "whitelist" anyone who might end up getting RBL'd to make sure the client can communicate with who they want. In EVERY case where a legitimate source was blacklisted, it was shown their ISP was irresponsible and the listing was valid.
In addition to using RBLs, we also have an array of hard-coded IP blocks that our server will not accept mail from. This covers a good bit of the rogue Asia-pacific ISPs that are the largest source of open relays. Something as simple as blocking major portions of 61.* have shown to reduce spam by 30+%. Anyone legitimately in China that needs to communicate with our network can be quickly whitelisted. Ironically, most of the ISP SMTP relays are not near the same broadband IP ranges - they obviously know how effective this technique is.
With RBLs and hard-coded spamming in effect, instead of 200 spams a day, I might get 3-5. As soon as I get new spam, I report it to Spamcop, and I notice a quick reduction in future spam of that nature immediately.
We're now getting near the point of blacklisting the entire 24.* IP block as well - which encompasses, among other things, a large portion of Comcast IP blocks that Comcast can't or won't control.
I'd like to see more ISPs simply refuse to accept mail from rogue networks. Then these networks would have to be more responsible.
Let me preface all this by saying our policy is to whitelist anyone who complains they have legitimate mail being blocked. For some strange reason, we don't hear any spammers making these requests. That's a shame because I'd be happy to visit them personally to make sure their situation is resolved in a mutually-deserving manner.
Re:How I deal with spam (Score:3, Interesting)
We're not blocking all of 24.* right now because ther
What about Bayes on word n-tuplets? (Score:3, Interesting)
Habeas SWE in spam (Score:3, Interesting)
I'd never seen any until this week, and suddenly I've got like 5/day.
I forwarded them to the good folks at habeas, hopefully the spammer will get sued into oblivion, but it's forced me to re-score SWE with a much lower bonus in spamassassin...
http://habeas.com/servicesHowSWEWorks.html for those who don't know what I'm talking about, btw
Gibberish, or code? (Score:5, Interesting)
What if spam and the spammers software - was actually being used by a third party in a surepticious manner to send/receive messages? Kinda like plaintext stego. Maybe the software used by spammers is backdoored by this third party - he sends instructions to the machine(s), maybe via a virus or something simpler, the spammers send their messages, but "unknown" to them the spams have this garbage at the end. The spammer doesn't really care, maybe he bitches at whatever passes as tech support for the spam software. Most people who recieve the spam see the stuff as garbage, or filter busters. But a certain group of the third party's friends - they have special email software that downloads these spams, and strips the garbage out, decodes it, and reassembles it into the real message. Maybe each spam only contains the equivalent of a couple of characters after decoding (maybe the garbage is actually packets telling order in the sequence, and other info to reconstruct the message) - but over a week or so, an entire message could be sent...
What is the possibility of that? Occam's Razor suggests otherwise, and filter busters are probably what the stuff is - but...what if...?
Re:Gibberish, or code? (Score:4, Funny)
This would be a very useful method for terrorists -- it would not only conceal the message itself, but also would defeat traffic analysis (i.e. nobody would be able to tell who sent or received the message -- it's sent by a spam king and received by everybody).
About the only way to guard against it -- or find out if the terrorists are already using this channel -- is to anal-probe all spammers for their client lists, then anal-probe all the clients. Fortunately, the obvious criminal content of 99.9% of spam provides sufficient probable cause for such action.
The real reason behind the weird typing in spam: (Score:3, Funny)
Re:Should be easy to block (Score:5, Insightful)
Most of them are using random word sequences; the random strings like xdwexe are not usually an important percentage of the overall text, no more than names might be. Besides, how large a corpus of "valid" words do you want to use? The OED weighs in at almost 0.5M; and then with another 0.5M uncatalogued scientific terms and neologisms, plus common mis-spellings and typos and jargon and dialect orthography (like our color, meter, checker, jail etc. for the Brits colour, metre, chequer, gaol) ...
If you don't want to keep the entire corpus of "valid" words in your code, you're going to have to make some compromises. Maybe you'll want to exclude words like "thou," "hauberk," and "coney." Not so good if you're subscribing to an Early Modern Literature listserv.
So you're going to need some logic to determine whether or not a "valid" word that occurs in a message is meaningful. Here's how one rather well known discussion [paulgraham.com] of Bayesian filtering deals with this issue (of unknown words); this is precisely the logic that spammers with random meaningful words are exploiting:
One question that arises in practice is what probability to assign to a word you've never seen, i.e. one that doesn't occur in the hash table of word probabilities. I've found, again by trial and error, that .4 is a good number to use. If you've never seen a word before, it is probably fairly innocent; spam words tend to be all too familiar.
So, what if all the words are valid, but the sentences aren't? Grammar checkers involve a lot more logic than spellcheckers do, and are consequently a lot less accurate. Fact is, you can also fool a grammar checker filter: just pad with random quotations from novels, etc. instead of padding with random words or random misspelled strings.
So the Bayesian approach of identifying spam and ham words is a pretty effective one, given the limitations.
Re:I keep praying for that silver bullet (Score:3, Insightful)
What it will take is the enforcement of existing computer-cracking laws. Spammers will then have a choice between 5-10 year sentences or sending spam with no munged words, forged headers, misleading subject lines, etc.