Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
×
Spam Your Rights Online

New Spam Frontier: Referer Logs 317

geoffsmith writes "Wired News is reporting that spammers are using referer logs as a cheap new way to spam small sites. Anyone running a website has probably already seen this phenomenon; I'm thinking of writing a script to remove these entries from my access_log by looking for hits that don't grab my images. (sorry lynx users!)"
This discussion has been archived. No new comments can be posted.

New Spam Frontier: Referer Logs

Comments Filter:
  • by SexyKellyOsbourne ( 606860 ) on Sunday October 27, 2002 @05:46PM (#4543355) Journal
    The entire internet will eventually go down in a deluge of spam unless it is made illegal and the laws are enforced!
    • by kryonD ( 163018 ) on Sunday October 27, 2002 @06:07PM (#4543471) Homepage Journal
      True, but at the same time wrong. Has anybody else noticed that the internet is currently the most active battlefield in hostory?

      Lowlife (but capitolist god bless 'em) pigs generate spam to sell their penis enlargement scam and mail clients develop ways to filter and block email. Distraction.

      Distributed Denial of Service attacks attempt to shake the very foundations of the NET through bandwidth flooding and sysadmins implement redundancy and load balancing. Jamming - Frequency Hopping.

      Remote exploits and virus appear everyday and patches are generated quickly for the more quality OS's and virus updates are required daily for Micro$oft OS's. Infiltration.

      Governing bodies exist that the people disagree with such as the RIAA and MPAA. Demonstrations are held in both violent(DDoS) and non-violent(civil disobedience of P2P) manners. Revolution.

      Needless to say, civilization has managed to survive for thousands of years despite man's desire to control everything including his fellow men. I think the internet will find a way.
      • "Needless to say, civilization has managed to survive for thousands of years despite man's desire to control everything including his fellow men. I think the internet will find a way."

        So are you saying that spam, DDOSs, exploits and the (RI|MP)AA all drive natural selection in the ecosystem that is the internet? What an interesting thought.

      • by IIRCAFAIKIANAL ( 572786 ) on Sunday October 27, 2002 @06:58PM (#4543700) Journal
        Please do not equate civil disobedience and P2P. Civil disobedience is essentially something you do in the open with the intention of getting caught and possibly prosecuted.

        If you want to learn about what civil disobedience really is, check this [eserver.org] or this [actupny.org] out.

        If you think that the Internet is the most active battlefield today, you need to visit [un.org] a [disastercenter.com] few [yahoo.com] places [yahoo.com].
        • Perhaps P2P was a bad example, although I do know people who specifically use Kazaa to get music because they refuse to pay the high prices charged for CDs. They openly admit what they do, although they probably don't expect anything serious to happen to them. Perhaps linking to or publishing the code to DeCSS would have been a better example.

          As far as what's going on in the middle east... You can link to web sites all you want, but until you come and work with me out here in the III Marine Expeditionary Force, where our Area Of Responsibility includes Afghanistan, Iraq, Indi-Pakistan, North Korea, the Philippines and Indonesia; You have no concept of either A)what a battlefield is, or B)what's REALLY going on in the world. Just a note: We have more attacks per day on one of our public .mil servers than we have had real contingency issues(to include disasters and humanitarian aid) in the whole theater all year.
          • I have the DeCSS source on my site (steve-parker.org) but that's not civil disobedience, it's a public service. I am in the UK (as is the server) and - until December - it is perfectly legal to distribute it. After December, it seems that I won't be able to do so. But at the moment, my country doesn't have a problem with me doing it, so I do it.
          • by IIRCAFAIKIANAL ( 572786 ) on Monday October 28, 2002 @02:01AM (#4545510) Journal
            Perhaps linking to or publishing the code to DeCSS would have been a better example.
            Yes, it would have :)
            You have no concept of either A)what a battlefield is, or B)what's REALLY going on in the world.
            I agree that the Internet can be and is a battlefield. So can Wall Street or the TSE. Or major media sources. Or the telephone. I don't agree that it is the most active - perhaps from a first world perspective, but I try and think a little more globally than that.
            Just a note: We have more attacks per day on one of our public .mil servers than we have had real contingency issues(to include disasters and humanitarian aid) in the whole theater all year.
            And how many people died due to those attacks on the public .mil servers? (Yes, I am sure they are important for various reasons, but if I was *attacking* the USA, I would be hacking hospital databases - there is a scary potential for warfare there).
            Has anybody else noticed that the internet is currently
            the most active battlefield in hostory?
            Hacking a .mil server certainly qualifies as warfare but you basically said that the internet is a more active battlefield than, say, WW2. I disagree. [hitler.org]

            (And anyone considering invoking Godwin's law... piss off :)

            I concede that the Internet certainly *is* a battlefield. However, considering that conflict on the Internet barely affects most of the people of the world, I wouldn't rate it so high.
      • True, but at the same time wrong. Has anybody else noticed that the internet is currently the most active battlefield in history?

        haha I've also noticed the beer inside my stomach is the most delicious outside of my body. And the milk in my fridge is the best drink on my desk. And the car that's in my driveway is at the shop.

        riiiight.

    • The internet is so often dealt with as if it were entirely novel. For the most part it's not, and simply complements telephone, fax, USPS, television, and so on for delivering information. (Granted, it is pretty neat.)

      So at minimum the internet deserves regulatory parity with these other media. Abuse of telephones and faxes was dealt with years ago -- (albeit incompletely -- our phone rings off the hook, I'll rant another day). For some reason business was quick to push for the outright ban on junk faxes, but hasn't for email which must waste a lot of their employees' time and hassle IT, in the end costing them money. Money talks, so I which there was a more concerted effort by those businesses that would never themselves spam.

      As with junk faxes (again, analogies everywhere) the injury from each incident is too small to do anything about; but we can act collectively through our government to attack the collective harm that is quite large.

      I won't comment on the current political obsessions in DC on anything but domestic policy, but I hope we see something soon. I don't think state-by-state legislation will do the trick. Your opinion will count if you express it to the right people. Writing your congresspeople for one is NOT a futile activity: they carefully tally what their constituents are saying, and you will likely get at least a form letter in reply. (BTW, I think a real paper letter carries more punch than email.)

      Exasperated outside DC, Andrew
    • When I'm feeling bored, I'll take a look through some of the crap procmail catches, and visit a site being advertised (if it's still up). But I don't just visit once! No! I leave lynx visiting the biggest page I can find by starting a script on my server, then forgetting about it for a day or so.

      If only a few hundred more people started doing this - absolutely flooding these spammed Pr0n sites, and get-a-big-dick-quick scams they would have HUGE bandwidth bills, and think twice about using the same marketting technique again.

      It's no use trying to email abuse depts, or reason with this scum, you have to hit them where it hurts, in the wallet. The only way to do this (for us at least) is to suck their bandwidth dry ;-) If you have a DSL connection at home (and you're not capped), why not use it to do some good when it would otherwise sit idle?
  • by reaper20 ( 23396 ) on Sunday October 27, 2002 @05:48PM (#4543365) Homepage
    "I'll adapt or I'll discontinue. I'm not planning on becoming the major annoyance of the blogging world.... I'm not too worried my reputation. Marketing is all about being innovative, different, adaptive, taking risks and knowing how to use the technology. I'm trying to be all that."

    Heh, it's funny that this guy can make this statement and expect to be taken seriously. It's even more pathetic that he actually thinks he's "innnovative".
    • by Ponty ( 15710 ) <awc2 AT buyclamsonline DOT com> on Sunday October 27, 2002 @05:51PM (#4543381) Homepage
      It is innovative. I was surprised and amused. It's awful, though. There's no rule that innovative things have to be positive.

      Anyhow, unless the traffic is completely disabling, I don't see this as more than an annoyance that technology will filter out when it becomes sufficiently obnoxious.
    • Actually it would be quite nice to see some of these "marketing gurus" put a little more thought into their spam. Today, some of the most carefully crafted content on TV is commercials (lamentably, also some of the worst). Watch and learn. I wouldn't mind receiving a spam that is fresh, funny, engaging, and didn't involve a virgin, my cock, a septic tank, or a gentleman from Nigeria. I wouldn't mind a funny beer commercial, for instance.
      • by Anonymous Coward
        I don't know spam that managed to involve a virgin, a cock, a septic tank, and a gentleman from Nigeria would almost have to be interesting.
  • Spam Lite (Score:4, Insightful)

    by Cyno01 ( 573917 ) <Cyno01@hotmail.com> on Sunday October 27, 2002 @05:49PM (#4543374) Homepage
    I don't know if i'm the only one, but has anyone else who doesn't filter their e-mail noticed a drop off in the amount of spam they recieve? For about the past 2 weeks, the amount of spam in my hotmail inbox has dropped from about 40 to around 15 a day. Anyone else had something similar to this happen?
    • Re:Spam Lite (Score:3, Interesting)

      by Em Emalb ( 452530 )
      Actually, yeah I have. I normally get 20-30 a day on my throw-away hotmail account, I just checked it for the first time in a week and had a total of 4 messages in my inbox--all spam of course, but there were NONE in the junk mail folder. Hopefully they put some sort of spam stopper in place? We can only dream.
      • Re:Spam Lite (Score:3, Interesting)

        I read somewhere (sorry, can't remember where ) that Microsoft updated their anti-spam service to coincide with the rollout of MSN 8. I believe it was Brightmail that they are using now.

        Wish I could remember where I read it, I would give you a link. Best I can find right now is:

        http://join.msn.com/?page=features/junkmail&pgma rk et=en-us&RU=http%3a%2f%2fjoin.msn.com%2f%3fpage%3d misc%2fspecialoffers%26pgmarket%3den-us
      • Re:Spam Lite (Score:3, Interesting)

        by NeMon'ess ( 160583 )
        One day soon I'm going to tell everyone using my hotmail account to use a yahoo account I've set up. I tolerated the increasing spam by using the custom filters. This worked until I hit the limit of 36. Then I had to get creative to work within that boundry. This was okay until last week when the my custom filters page now tells me I am over my limit of 10 filters and must delete 26 of them or pay for Hotmail Extra Extortion Services. Fuck them. I had the account before MS bought Hotmail and I tolerated all the crap until now. Yahoo's junk mail filters actually work so that's where I'll be.
  • Lynx users? (Score:5, Funny)

    by Anonymous Coward on Sunday October 27, 2002 @05:52PM (#4543386)
    (sorry lynx users)

    Don't worry. It's highly unlikely that any of the 4 current users will visit your website anyway.
    • why sorry lynx users is my question.

      I use opera most of the time with images off if I am on dial-up. Surely at least 3 percent of the population turns of images for non porna browsing.

      It is a very convienient way to avoid adds and decrease load times.
      • Too many sites use pictures as links for navigation for me to turn off all images. I just block the adds and that's enough for me. I live with flash 5 adds and stop flash 8 adds as soon as they start. I also let .gifs animate only once. The result is my browsing doesn't look so bland and it loads pretty fast as well.
    • by Anonymous Coward
      (sorry lynx users)

      Don't worry. It's highly unlikely that any of the 4 current users will visit your website anyway.


      You mean there's another 3 lynx users out there? I'm not alone!
  • ...just some blogger that was slashdotted, unwilling to believe that (s)he's really getting that many hits referred to from just one site.
  • Well.. (Score:3, Insightful)

    by joyoflinux ( 522023 ) <thejoyoflinux.yahoo@com> on Sunday October 27, 2002 @05:52PM (#4543395)
    He just got a link posted on /. and Wired--I wonder how many spammers are going to target him now...This seems a little aganist logic
  • by CySurflex ( 564206 ) on Sunday October 27, 2002 @05:55PM (#4543411)
    Windows users are complaining that Microsoft is filling up their computer's System Event Log with spam about illegal exceptions and page faults.
  • For one thing, I only get about 2-3 legit emails a day, vs 20-30 spams.

    On the other hand, I usually get a few thousand refer logs, and I *already* get a bunch of bogus refer logs from buggy browsers or something (like, a refer from a site I link to, I guess from people hitting the back button, that kind of thing).

    On the other hand, I could see how it could get annoying for small sites.

    The "solution" you mentioned wouldn't really work, as the spammers could simply download your images as well.

    A more effective way to block these would be to scan sites in your logs and check to see if they link to you. It might take a while for huge sites, but then huge sites probably don't look through their refer logs as much.

    OTOH, you would miss out on hits from sites that have random URLs or that kind of thing (like goggle's 'get lucky button')
    • The "solution" you mentioned wouldn't really work, as the spammers could simply download your images as well.

      I see a solution in this. It would be the spammer's own DOS attack. If they willing to download /dev/zero in order to place their refer entry, that's great, more power to them. If they don't download data, that invalid refer entry could easily be dismissed. Solution? I'm sure someone will crank out a spammer-refer-mod to include in apache.conf over this. :)
  • by jukal ( 523582 ) on Sunday October 27, 2002 @05:58PM (#4543421) Journal
    I don't know who started it - but I find it very odd that browsers send referer info by default. Why? It does not provide anything extra for the user but problems. It is not once or twice that you find URLs to "confidential" pages if you browse through your webserver logs. And... I bet 95% of web surfers do not even know that they are sending this information all the time. Is there really any reason why the default is to send the referer info? I have seen people riot on much less important privacy issues. Why not about this? The referer plague exists in almost all browsers - and only in few browsers you actually can easily turn it off. What's going on?
    • by Openadvocate ( 573093 ) on Sunday October 27, 2002 @06:21PM (#4543534)
      There are many reasons, mostly for those who program websites. Sometimes you don't want people to see a page before another. this could also be solved with cookies, but some blocks those too.
      Then there is the statistics, learn how people navigate around your site. referer can help you see a pattern and improve your layout.
      Also it can prevent bandwidth hogs, mostly a issue with ad. graphics and pron sites where people use graphics from others servers on html pages on their own sites but also on free servers where people place graphics and files and link to those directly without using any html and then not showing any of the free servers ad's which provides them with money to run the sites in the first place.
    • It's nice, as a site operator, to know where your guests are coming from. A good portion of my visitors come from Google and other search engines. The referrer log lets me know what they were searching for, and in nearly 95% of the cases they were looking for a specific topic on my site. I can send them directly there, give them a specific welcome message if they haven't been to my site before, etc.

      Furthermore I can restrict traffic for some areas of my site (like some sites that block links from slashdot) for particular reasons or uses. "You just came from the page of an associate and are able to receive a discount." "This page is restricted to users of xyz.com. Please go there first."

      Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.

      Referrer information may be annoying to you, but it's an extrememly useful tool. If taken away one restricts opportunities for the site operator to personalize and protect content on their site. Not a huge loss, but it isn't really as great a privacy issue as you seem to believe.

      -Adam
      • by Permission Denied ( 551645 ) on Sunday October 27, 2002 @08:02PM (#4543977) Journal
        I can send them directly there, give them a specific welcome message if they haven't been to my site before, etc.

        This is so damned annoying. If I'm searching for some specific information, I don't give a damn about your idiotic welcome page. I don't care what your website is about or what you have to say on your other pages - all I care about is the specific technical information that google told me you have.

        More and more, I'm finding myself using googles cache instead of clicking on the actual links. I know you couldn't care less about my insignificant browsing habits, but the more people start doing annoying crap like this, the more people start using google instead of the web.

        "This page is restricted to users of xyz.com. Please go there first."

        Do you realize how stupid this is? You're trying to control how I use my browser. Of course I'm not going to go to xyz.com and try to use their idiotic navigation looking for a link to you. You're simply advocating another form of advertisement and I'm not interested. I care about the data you're providing, not how you're getting funded.

        I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.

        And this is, of course, broken behaviour. Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all? This means that I can't open your images in new windows and I'm constrained to view your images one at a time. Also, the some browsers change the referer for images when you "save" images (eg, right-click and choose "Save as..." may not send the referer you're expecting).

        If taken away one restricts opportunities for the site operator to personalize and protect content on their site.

        If you're using this to restrict content to your site ... well, forget it. If you have something I really want, I'll open up a terminal and telnet to port 80. Yes, this is indeed effective restriction. (Quiz to see if you really know what you're doing: how would you set it up so that you know that a user has previously visited another site, with cryptographic confidence?)

        As for "personalizing" content, please stop. The only times I've seen that word being used in a web context is to personalize advertising (and also restricting content because I'm not using IE, but don't get me started on that). I've never seen anyone "personalize" a site in a useful way, eg, "You're a C programmer who writes Solaris kernel modules, so you're probably not going to spring for my Herbal viagra scheme and I'm going to cut the marketing BS and give you only useful information."

        Why do these "blogs" even keep logs of referer links? This is pure narcisism (and more importantly, a waste of disk space - even though disk is cheap, it's still worth more than someone else's paltry feeling of acceptance). If you're going to say something, just say it. Don't base your life around how many people like what you say. "Ohh, somebody linked to my journal, that means I'm special and I can now feel good about myself." Ahh - get a life.

        I swear, "webmasters" piss me off.

        • by achurch ( 201270 ) on Sunday October 27, 2002 @11:55PM (#4545024) Homepage
          I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.

          And this is, of course, broken behaviour.

          So do you have an alternative proposal to prevent resource (i.e. bandwidth) theft? That is a very real problem, and no amount of arguing that the current solution is "broken" will get people to change unless you provide them an alternative.

          • by Permission Denied ( 551645 ) on Monday October 28, 2002 @05:23AM (#4545933) Journal
            So do you have an alternative proposal to prevent resource (i.e. bandwidth) theft?

            Session cookies based a cryptographic hash of browser-identifiable information. Just hashing the IP and some secret string will prevent the bandwidth-stealing problem (not ideal since it breaks with NAT, but that's irrelevant if you're only trying to solve the deep-linking problem).

            In php, setcookie('hash', md5($ENV[REMOTE_ADDR] . "TOPSECRET)) on page load, link to a file "image.php" instead of the .jpg and "image.php" does something like this: if (getcookie('hash') != md5($ENV[REMOTE_ADDR] . "TOPSECRET")) { header("Location: /error-documents/403.html"); exit(); }. This isn't complete (probably not even syntactically correct and be careful with what image.php allows one to download), but you get the idea. The actual image files can't be downloaded by apache, but can only be opened and sent to the browser through "image.php". For extra fun, re-generate the secret string from /dev/random every ten minutes (and keep around the last version of the key to avoid breaking on-going sessions).

            This stops everyone from stealing bandwidth (including telnet-wielding network programmers like me) and it annoys no one.

      • Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.

        no it doesn't... it protects your BANDWIDTH. by keeping joesimagewhores.com from embedding your images directly in their html you protect your BANDWIDTH.. there is nothing you can do to keep me from copying your images from your site and using them in my site.. you can try the lame Java and Javascript solutions... those won't even slow down a web-user with 1/2 a brain.

        so please, tell us the truth, you are protecting your bandwidth and rightfully so.

        Me? I have more fun with it... I have a perl script that returns random porn if the photo is asked for from outside my site or it uses imagwmagick's mogrify to place "stolen from MEMEME.COM" in the center of the image... depending on my mood... (No I will NOT post my personal website on slashdot... I'm not about to get a huge bandwidth bill because of you guys!)

        I dont care if they steal my images. I care if they try to steal my bandwidth though...
      • Lastly, it protects my image content.
        You think your images are safe, but they're not. Anyone determined enough to get your images can use a program like curl [curl.haxx.se] to tell your server it came from whatever URL you are looking for and pass itself off as any browser out there. My software uses this feature to grab comic strips from tight-fisted Web sites.

        Referrer information may be annoying to you, but it's an extrememly useful tool. If taken away one restricts opportunities for the site operator to personalize and protect content on their site. Not a huge loss, but it isn't really as great a privacy issue as you seem to believe.
        Thank you for that FUD, but we hear it all the time: "give us all your personal data or you will lose functionality." BS. As a Web site owner, you have no right to know what site I visited before yours. There are many powerful things you can do with that information.

        For instance, if a person goes from a Monster.com search page to his Yahoo mail account, Yahoo now knows where the person is looking for a job, what type of job he is looking for, etc. (it's all encoded in the URL). Yahoo also has access to his address book and all his email messages.

        I see a scenario where Yahoo subtly threatens to email your boss to let him know you're thinking about quitting... unless you upgrade your account/add more storage space. It won't happen tomorrow, but Yahoo is sleazy enough to try something like that and they have the information... all they need is the technology to make that connection.

        That's just one example, but it illustrates the point that referrer information is none of your business. You only want it because you can profit from it without any complaints from your audience.

        Another example:

        A lot of people apparently email the URL of my site to their friends. In my site logs, I often see the email addresses of the person who sent the message and the poor sap who clicked the link. These people have no idea they have divulged their email addresses to me via referrer info. If they wanted me to have that info, they would have given it to me. Sometimes I also see the subject of the message, which is particularly funny when it was sent by a competitor along the lines of "Have you seen what <insert_url_here> is doing?"

        But as you said, "it isn't really as great a privacy issue as you seem to believe." It's worse than you realize.

        Bottom Line

        Companies will do just about anything to make an extra buck. So it shouldn't surprise anyone that they use technology against users to that end. But it's a two-way street -- people just need to wake up and start using technology to protect themselves.

    • Do what I do: use Privoxy [privoxy.org]. Not only can you use it right now with whatever your favourite browser is, it's free. Not only does it block ads, it allows you to set Referer: on all outgoing requests to whatever you want. (I set it so Referer: is always the base URL of the page being viewed.)

      Incidentally, I don't know why anyone bothers with logging referrer information. The only use sounds like what the bloggers do. If you're not a blogger, why do you even care who the referrer is? Half the time it's bogus or one of your own pages.

      • Incidentally, I don't know why anyone bothers with logging referrer information.

        It's good to know who is sending users to a dead link. Just by checking the referrer information for the 404 entries in your logs can determine what website is pointing users to a document that you have deleted or moved.
    • >I don't know who started it - but I find it very odd that browsers send referer info by default. Why? It does not provide anything extra for the user but problems.

      It is extremely useful for security purposes.

      No, not the security most people are thinking of. Checking to see if the user came from FeedBack.html before executing FormMail.pl is no security, since spammers can forge any referer they want.

      I'm talking about security which stops a human user who is logged in to a particular website from being tricked into performing actions they didn't authorise. For instance: I log into my server's adminsitrative area. Then, in another window, I browse someone's blog. And I click on their "search" button. As it turns out, this search button is a trap, which sends me to my own admin area with a command to delete someone's account. I'm logged in, I have a valid network address, I'm active, there's no problem. Except that fortunately my browser sends "Referer: www.blog.org" instead of "Referer: www.admin.com".

      That's why referer info is useful: to prevent a user from being hijacked.

    • I would agree with you, but for some reason the creepos at freerepublic.org love to link to my images. It's a giant, sudden bandwidth waste. Don't know why they do that, don't care, I stopped them and I needed their referer headers to do it.
    • "Why? It does not provide anything extra for the user but problems."

      Because it's nice for us site hosts to know where the traffic is coming from, helping us to realize just how few constituents are visiting our sites...

      *mutter* Last damn time I put a URL in my sig...
    • One of the primary uses I have for referrer information is locating bad or malicious links. If someone is sending large volumes of traffic to a particular page on my site, I'd like to know where that traffic is coming from. In addition, even to pages on my own site, if I see someone following a link to somewhere they either shouldn't be going or to a mistyped URL, the referrer information allows me to identify where they're coming from, and if it's a problem with my own site, it lets me correct it.

      Perhaps referrer information should be released depending on the site's posted P3P privacy policies. If a site is interested in collecting information like this for marketing purposes, I can understand someone's reluctance to have their browser provide it. But for the rest of the sites (including those I maintain), the information is only ever used strictly for legitimate needs like those mentioned above. Please don't advocate that referrer information be restricted by default or for everyone, because that hampers my ability to troubleshoot problems.
  • Sorry 'bout what? (Score:3, Insightful)

    by PissingInTheWind ( 573929 ) on Sunday October 27, 2002 @06:00PM (#4543436)
    ...(sorry lynx users!)

    Sorry about what? Why should they care wether you keep them in your log or not?
  • by j7953 ( 457666 ) on Sunday October 27, 2002 @06:01PM (#4543443)

    From the wired article:

    ... even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines.

    Umm, huh? I don't think the spammers actually link to the sites, they probably just send HTTP requests with faked referrer headers that contain the URLs of the spammer's web site. That won't boost your search engine rankings.

  • by RevRagnarok ( 583910 ) on Sunday October 27, 2002 @06:04PM (#4543452) Homepage Journal
    In the regular prefs and the "quick prefs" (F12 under Windows version) Opera [opera.com] lets you turn off referrer logging. The only time I need to turn it on is certain sites, like my credit union, which is no big deal...
  • what is this? (Score:3, Interesting)

    by Dr. Awktagon ( 233360 ) on Sunday October 27, 2002 @06:08PM (#4543472) Homepage
    I'm not sure I understand. Does this mean the spammers put links on their own porn (or whatever) sites, and casual surfers will click into the blog from the porn site, thus making the porn site show up in the logs as the referer? That's how the referer is supposed to work, right?

    Or are they just bots that hit random web sites and send fake referers along?

    Either way, I have absolutely no clue why this would be abusive or even annoying? Can someone explain? Do people sit around checking their referers all day long?? (Then again, I don't understand why anyone would run a blog, so maybe I'm just out of touch).

    I clean out all my outgoing referers (thanks squid), so maybe I subconciously assume everybody else does too. Never thought of the referers as anything but a silly waste of bandwidth, since they can be forged so easily.
    • From what I have seen on my logs...

      They hit your site with a referrer address of whatever they want you to read. They count on your curiosity to go there when you are investigating your referrer logs.

      I have been bitten by this once or twice.
    • Re:what is this? (Score:3, Interesting)

      by crapulent ( 598941 )
      My interpretation of this article is that the spammers are setting their client's "Referer:" header field to their porn site, and then retrieving pages from the blogs. The result is that links to the porn/spam sites appear in the Apache referer log file on the blog site. The spammers do this because they know the blog operators pay extra attention to their referer logs and are likely to follow those links (either out of curiosity or out of the desire to maintain reciprocity with other blogs that may link to them.) Apparently the bloggers have scripts that automatically harvest all the URLs from these referer logs to make this process easier.

      I don't think the spammer would bother creating an actual link on their porn/spam site to the blog, although this would work as well. It's silly though since it's more work and it still requires that someone actually click on the link for the porn URL to make it into the referer log. Why bother when they could just run an automated script to hit the blog with the forged "Referer:" and then discard the results. The only possible reason to do it this way is that the spam URL would be sent multiple times from different IP addresses, and hence harder to filter or ignore.

      The confusing bit is that the article mentions that this might prop up the blog's SearchRank relevancy. This would only be the case with the latter method (creating an actual link) whereas the more straightforwad way would have no such effect.
    • RTFA (Score:3, Informative)

      by Galvatron ( 115029 )
      As it says in the article, some blogs have automated lists of the top referrers, so that visitors can see who links to the blog. And yes, we're talking about bots sending fake referrers.
  • by puppetman ( 131489 ) on Sunday October 27, 2002 @06:10PM (#4543482) Homepage
    One of the quotes from the story was,

    "... they are hitting my site so hard. One day there were more than a thousand hits from one single porn site," complained Åsk Wäppling, known to bloggers as "Dabitch."

    Ok - this guy's name is Dabitch in an obvious reference to the whole pimp-bitch-rape-mysogynist thing that seems to be all over the Internet (and that I can't stand), and he's complaining that a porn site is effectively doing a denial-of-service. I think it's just desserts.
  • by fermion ( 181285 ) on Sunday October 27, 2002 @06:12PM (#4543493) Homepage Journal
    It seems that this only affects sites that automatically publish referrer logs. Now, since the referrer is generated by an unknown foreign machine, and therefore is not to be trusted, automatically publishing this information to the world does not seem too smart in the first place. At most, this 'spam' will only stop the act of automatically publishing referrer logs. At least I would expect a method for a certain link to be automatically marked as a spam site as soon as one blogger hits it. Of course, the question will then be how to decide if a site is a spam site.
  • I run a small site, and keep logs for a month or so, in case of bugs, hacking attempts etc.

    875M /var/log

    I really don't plan to read them ;)

    I do run webalizer over them of course.
    • by Anonymous Coward
      It makes great bathroom reading. Print them out and keep stacks of them by the toilet. Impresses the chicks too.
  • by andrewm ( 9862 ) <andrewm@netwinder.org> on Sunday October 27, 2002 @06:19PM (#4543526) Homepage
    I use the noimg.pl for showing stats on sites not pulling images. Viruses and spambots are obvious.

    Also, showpath.pl and spampath.pl are handy to see where the vister went, and check for obvious robot behaviour:

    http://www.turnstep.com/Spambot/Programs/
  • by trentfoley ( 226635 ) on Sunday October 27, 2002 @06:21PM (#4543533) Homepage Journal
    I, for one, look at the referrer for two things:

    1. To see how search engine users have found my pages. I think it is amazing that I will get hits from people looking at the 100th page generated by google. I mean, c'mon on, 100 clicks on "next" before you think the I have the info you are looking for?! All I have is crap.

    2. To see if slashdotters actually click on my postings' links.

    Come to think of it, the only other thing I even look at the logs for are to see what search engines are indexing me (and are they obeying robots.txt) and to harrass people with infected iis machines.

  • by KPU ( 118762 ) on Sunday October 27, 2002 @06:25PM (#4543550) Homepage
    All this time I thought "GET /default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858% ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%uc bd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531 b%u53ff%u0078%u0000%u00=a HTTP/1.0" was code red. Now I know it's spam :-).
  • Could somebody explain what this referal log thing that is being spammed is? I am not one that travels around weblogs and such - so that article has a void that needs filling.
  • Oh, damn. I need glasses.
  • by Guppy06 ( 410832 ) on Sunday October 27, 2002 @06:40PM (#4543613)
    "die-spammers-die"

    "No! That's German for 'the spammers, the.'"

    "No one who speaks German could be an evil man."
  • Is there a good HOWTO somewhere how to install your own local spamfilter?

  • ewww! (Score:3, Informative)

    by digitalsushi ( 137809 ) <slashdot@digitalsushi.com> on Sunday October 27, 2002 @06:54PM (#4543680) Journal
    I didnt even notice they were in there til you made me look! EWwwwwwwwwwwwwwwww! That is just fantastic. I host a site for a rather religious young girl's artwork- imagine my shock at nakednewbies.com linking to her! Glad I read this story first!
  • Old trick (Score:5, Informative)

    by dmiller ( 581 ) <djm AT mindrot DOT org> on Sunday October 27, 2002 @06:59PM (#4543704) Homepage
    I remember seeing weird stuff in logs as far back as 1994. Back when Netscape Commerce server had a large percent of the webserver market, there was an "exploit" which allowed the placement of HTML code from untrusted sources directly into the log reports (basically, it didn't check whether referrers, source domain names, etc did not contain metacharacters). I remember seeing lots of adverts which used this technique to sell stuff, mainly web-log analysis software.
  • by mkelley ( 411060 ) <slashdot@[ ]lley.net ['mke' in gap]> on Sunday October 27, 2002 @07:10PM (#4543762) Homepage
    I sent an email to the owner of Mastodon and here is what he wrote:

    From: "[ f ]"
    Date: Thu, October 24, 2002 12:19 pm
    Subject: Re: logs
    Hi!

    You can send us your URL and we'll permenantely remove it from our database.
    It's not a spider, it reads only the blogs in the database.

    Regards,

    Francois


    And to add to that, one of the comments on my weblog was that there is also an opt-out list [mastodonte.com].
  • Apart from stopping people who don't view images showing in your logs, (not a disaster), your script would simply make the robots download an image from your site each visit, thus increasing your bandwidth usage.

    Any method that relies on obfuscation or silly tricks becomes a lot less effective once posted here.
  • by lambsonic ( 512680 ) on Sunday October 27, 2002 @07:51PM (#4543930)
    This is a non-issue for me. I run a script that validates the referrers [tautology.org].
    • Validates what? Exactly how can you tell that a users web client (which you may not have access to get all info from) clicked on a link in a page (on a server you probably don't have access to)?

  • backlink links (Score:4, Interesting)

    by luap2000 ( 314919 ) on Sunday October 27, 2002 @08:27PM (#4544105) Homepage
    Backlinking [iawiki.net] has become popular lately on a lot of blogs. The term describes scripts built to automatically display links from referrer logs [devhelper.net]. The idea was to automatically create a list of links [disenchanted.com] to sites that might possibly have related content because someone came from that site to yours. A good example of its use can be found at diveintomark [diveintomark.org].

    While the process has gained some popularity [decafbad.com] of late in the blogosphere, the idea has been around [unrealities.com] for quite some time [gmu.edu].

    Are tactics like this legitimate forms of marketing? Would you term it Gonzo Marketing [gonzomarkets.com]? Viral Marketing [wilsonweb.com]? Whatever term you use, it will definitely be interesting to see where this one goes.

  • by dargaud ( 518470 ) <slashdot2@@@gdargaud...net> on Sunday October 27, 2002 @09:15PM (#4544308) Homepage
    I haven't noticed this trend yet but I will... I have a small shell/perl script that does a simple analysis of referer logs (search engine keywords, other external referers...). Get it from my website [gdargaud.net] if interested.
  • Backlinking (Score:5, Insightful)

    by CaptainSuperBoy ( 17170 ) on Sunday October 27, 2002 @09:34PM (#4544374) Homepage Journal
    Backlinking, or posting your referral logs, is doomed to failure and rightly so. It's just a glorified way of making your site into a link farm, with the expectation that your fellow bloggers will do the same. It is serendipitous that this practice is open to 'abuse' although I would never call the abusers spammers. They are just utilizing a method for submitting data that the site owners themselves have provided. I don't see any reason to call this 'spam' since the site owners are inviting users to submit data through HTTP referral headers.

    Also, this quote from the article is ludicrous: "bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines."

    There is no search engine that bases your rank on the number of sites that you LINK to. I believe the bloggers actually mean that they're sorry to see their backlinks (read: link farms) go, since those do in fact raise search rankings. What a travesty- Sites may have to rely on the actual quality of their content, rather than trading links!

    Amidst the alarmist cries in the article, "spammers will destroy our practice of posting referral logs," nobody has even mentioned that there is a ridiculously easy technical solution. Before posting a referral link, why not just have your software visit the referring site and detemine if it actually links to your page? This will defeat the referral advertisers.
  • Guestbook spam (Score:4, Interesting)

    by AlpineR ( 32307 ) <wagnerr@umich.edu> on Sunday October 27, 2002 @11:24PM (#4544891) Homepage
    Here is another form of spam that was new to me. Apparently some German pr0n site operators are filling my guestbook [umich.edu] with bogus entries linked to their offerings. It seemed an odd way to advertise at first (who the heck visits my site [umich.edu], much less reads my guestbook ;-), but now I realize that it helps their Google stats.

    For now I'll delete the entries by hand, but if this increases it could get really annoying.

    AlpineR

BLISS is ignorance.

Working...