Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
The Courts Government News

Is Data Mining for Product Pricing, Illegal? 372

Posted by Cliff from the courts-and-ethics dept.
wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?
This discussion has been archived. No new comments can be posted.

Is Data Mining for Product Pricing, Illegal? More

Is Data Mining for Product Pricing, Illegal?

Comments Filter:

  • I swear (Score:5, Insightful)

    by cultobill ( 72845 ) on Wednesday May 14, 2003 @10:50PM (#5961000)
    In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark (unless you're a dollar store), so much for trademark. There's nothing useful that hasn't been done before, fuck patenting them. Copyright? It's a simple derivation of what the supplier charges you.

    There is nothing creative about pricing stuff. Good lord.

    • Re:I swear (Score:3, Insightful)

      by Gortbusters.org ( 637314 )
      Look at pricewatch.com [pricewatch.com], it already goes around collecting price data from many online stores.

      I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices. In reality if a consumer reaches your site on the internet for your product they probably didn't do it by accident. They will evaluate all aspects of the business (licensing, service/support, upgrade cost, security of the site, etc) before they jump ship to another site to save a few bucks.

      I'm sure D

      • Re:I swear (Score:5, Informative)

        by Sparr0 ( 451780 ) <sparr0@gmail.com> on Wednesday May 14, 2003 @11:16PM (#5961154) Homepage Journal
        Pricewatch doesnt mine. Companies PAY for the privelege of listing their prices on pricewatch.

      • Re:I swear (Score:4, Informative)

        by CTho9305 ( 264265 ) on Thursday May 15, 2003 @09:59AM (#5964183) Homepage
        WARNING: You can see I write crappy perl / shell script. I make no guarantees about this code.

        NOTE: replace all instances of "ABC" with "|".

        parse.sh:
        #!/bin/bash
        #Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).
        #barton
        lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep 333 ABC cut -f1,4 -d ' ' > XP333.dat
        #XP
        lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep -v 333 ABC cut -f1,4 -d ' ' > XP.dat
        #Apparently I have too many junk characters
        #MP
        lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i mp ABC perl -pe 's/\.//' ABC perl -pe 's/GHz/00/' ABC cut -f1,4 -d ' ' > MP.dat
        # Celeron
        lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i celeron ABC cut -f 1,3 -d ' ' ABC perl -pe 's/ 1GHz/ 1.0GHz/;s/\.//;s/GHz/00/' > celeron.dat
        #P4
        lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//;s/\s+/ /' ABC grep -i 'pentium 4' ABC egrep -i "sockABC2\.ABC3\." ABC grep -v -i 400MHz ABC perl -pe 's/ 533MHz//;s/GHz/00/;s/ Sock 478//;s/\.//' ABC cut -f1,4 -d ' ' ABC cut -b -8 > pentium4.dat
        gnuplot gnuplot.script > ~/www/out.png

        Anyway, I wrote this because I was bored and wanted to see what a good price point was for current Athlons. If you examine the graphs carefully you might note that the XP's are not properly differentiated. Some are 333s and marked as that, others aren't marked properly, etc. With the new 400s, it gets worse. For the P4s, I got a little luckier because the speed ranges don't overlap as much. I think I'm going to not differnetiate between the various FSBs of Athlon XPs because the prices are close enough anyway.

        Anyway, it has served its purpose by helping me find a point where the processors are reasonably fast, and the bang for the buck is decent.

        gnuplot.script

        #Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).set terminal png color
        set xlabel "Speed (MHz or rating)"
        set ylabel "Cost ($USD)"
        set title "Speed vs. Cost"
        set grid
        set time
        set linestyle 1 lw 3
        plot "XP.dat" using 2:1 title "XP" with linespoints, \
        "XP333.dat" using 2:1 title "XP333" with linespoints, \
        "celeron.dat" using 2:1 title "celeron" with linespoints, \
        "MP.dat" using 2:1 title "MP" with linespoints, \
        "pentium4.dat" using 2:1 title "P4" with linespoints

        Anyone know how to change the text font, or the thickness of the lines?

        Sample output [cmu.edu]

    • Re:I swear (Score:3, Insightful)

      by whereiswaldo ( 459052 )
      Here's a niche for P2P software... obtaining data from a website in a distributed way so as not to stick out in the website logs the way a one or a few download clients would. The collected data could be processed remotely or uploaded to a central server.
      • EXACTLY what I have been saying for a time...

        there should be this exact type of service. actually what i was thinking of was a service that you would "bid" on an object - say only for items valued over 500 bux.

        Then that service would scour the NET, ebay and every other normal avenue looking for the lowest possible price for the purchase you want to make. then buy it... and you would pay a service fee % based on the value of the item and the obscurity etc...

        but all in all - attempting to make it as automa

    • Not the pricing -- the timing (Score:4, Informative)

      by RalphSlate ( 128202 ) on Thursday May 15, 2003 @08:12AM (#5963266) Homepage
      There may be precedent for this. eBay was able to convince a judge [com.com] to bar spidering of their site.

      There is another legal concept called "Unfair Competition" which links copyright and facts.

      Normally, facts cannot be copyrighted [findlaw.com]. However, this law seems to kick in when one company compiles and publishes time-sesitive information that it has taken from a direct competitor in a way which "free-rides" on the efforts of the competitor. It is usually applied to news organizations, when one newspaper sends a reporter to Iraq and a second newspaper (perhaps an evening edition) uses the "facts" in the first newspaper's article to publish the very same news.

      I could see the instantaneous publishing of all competitors' prices as a violation of this legal theory.

  • Just like Popeye (Score:3, Funny)

    by Anonymous Coward on Wednesday May 14, 2003 @10:52PM (#5961009)
    Anybody remember the Popeye cartoon where Popeye opens a car wash, and then Bluto opens a car wash right across the street?

    When Popeye posts his price, Bluto beats it by five cents. Then Popeye beats Bluto's by five cents.

    It goes back and forth until Popeye is washing cars for free.

    I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

    • Actually, sometimes they do. (Score:2, Insightful)

      by Anonymous Coward
      Two specific cases in point.

      1) At many of the deal sites (i.e. slickdeals.net, etc) once in a while this offer appears where after getting back your rebate, you have more money than you spent for the product.

      2) Grocery coupons - in some cases, a store will run one of those "triple coupon Thurdays" promotions, and if you have the right coupon, the money-off total will exceed the price of the product. Depending on the the store, money is returned, or a credit is.
    • Unless [slashdot.org] you're MS and you want to keep Linux out of a large organization! MWahahaha
    • I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.
      Last month, I got a $40 rebate on a $39.95 router.
      • Heh, youre a rebate newbie. Peachtree Accounting. Costs $200. You get $40 off the purchase price at the store with a coupon. And there is a $160 rebate. And there is another $30 rebate if you own a previous version, or competitors software. This same deal exists every year, I have been "upgrading" for the last 3 years with it. And this is only a mediocre deal. Deal sites regularly list deals where you spend $20 and get $40 back.

      • Re:Just like Popeye (Score:3, Insightful)

        by tupps ( 43964 )
        How much was postage?

        Have you actually received your rebate?

    • of the old joke about this fella who was selling everything in his store at a loss.

      When somebody asked him how he can make a living like that, he replied:

      "Volume!"

      • It's not even an old Joke - I believe it was a quotable thing said by, probably, Micron CEO a while back.

        This, if nothing else, shows how thin the DRAM margin is, and why RAMBUS trying to skim 2% *gross* will never fly.

        anyway, the point is that by pushing out a large volume, they keep their lines running, which means that they can actually make SOMETHING off those lines and get some kind of profit because the cost of running them things through arn't so damn high. This can be demonstrated by how 512M SODI

      • Joke? That's been Amazon's business strategy for years!

  • TITLE HAS POOR GRAMMAR (Score:3, Informative)

    by Anonymous Coward on Wednesday May 14, 2003 @10:53PM (#5961014)
    EDITORS: A better way to phrase that headline is as follows:
    Is It Illegal to Data-Mine for Product Pricing?


    Thank you.

  • Ummm.. (Score:5, Funny)

    by deadgoon42 ( 309575 ) on Wednesday May 14, 2003 @10:54PM (#5961016) Journal
    My grandmother calls this "shopping around." The only difference is that someone else is doing all the work.

  • Price Scraping? (Score:3, Informative)

    by Anonymous Coward on Wednesday May 14, 2003 @10:54PM (#5961020)
    Actually, price scraping was done in a very low-tech way a reasonably long time ago by a pretty well-known businessman: Mr. Sam Wal-Mart. Early in his career, he would dumpster-dive his competitors to find out the prices his competitors paid for their goods, the contracts they had with their suppliers, etc. This provided him with "insider information" so he'd know how to prepare his pricing in a more strategic fashion and obviously out-compete them where it counts: financially.

  • TOS and more ideal markets (Score:4, Insightful)

    by weston ( 16146 ) <<westonsd> <at> <canncentral.org>> on Wednesday May 14, 2003 @10:54PM (#5961021) Homepage
    I think any large business where the pricing structure isn't directly related to costs is probably deeply afraid of agents that aggregate their data with competitors. You end up with a more ideal market, a more frictionless market, if you will, and they'll be forced to compete on narrower and narrower margins of profit. Of course they'll want to throw up barriers to that.

    But I'll bet this issue comes down to Terms of Service and what a company can reasonably expect to be able to legally require/forbid about the use of data provided via an automated means...

  • You have a contract? (Score:5, Insightful)

    by gpinzone ( 531794 ) on Wednesday May 14, 2003 @10:54PM (#5961023) Homepage Journal
    As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.

    • Re:You have a contract? (Score:5, Insightful)

      by vandan ( 151516 ) on Wednesday May 14, 2003 @11:47PM (#5961285) Homepage
      Would you apply this line of reasoning to ALL areas?
      What if the job were researching Bush's all-feared biological weapons?
      Or GM products?

      The problem is that if everybody decides to look the other way (and everyone can find a reason why they should take their money and shut up), then some pretty fucked up things get done, and people are left wondering, "How did it come to this?"

      Now I'm not saying that the world is going to end because someone's harvesting prices off the net. I'm just questioning your "Me first, no-one else 2nd" argument.
      • IT's an ethical and legal issue, but not one that is life threatening. Goodwin's law ought to be ammeded to include types like you.
      • I do hope that every /. reader at least can discriminate between building life-threatening devices and products that may or may not just fall under some crazy copyright law. It's the user's responsibility not his. They assigned him, they carry the consequences.
        It's your way of extreme reasoning that gets us nowhere. We must be reasonable, and it's reasonable to expect that no judge in his right mind would convict anyone for this.

  • Rules of Thumb to Live By (Score:5, Insightful)

    by release7 ( 545012 ) on Wednesday May 14, 2003 @10:56PM (#5961035) Homepage Journal
    I'm not going to pretend to know what the laws are in this case. It seems that the only law these days is "He who can afford the best, most impressive lawyer wins." Here's some other cynical words of wisdom I've come to believe:

    If powerful people get screwed, it's illegal.
    If it forces large corporations have to work harder to earn a profit, it's illegal.
    If it give the little guy a leg up or levels the playing field in any way, it's illegal.
    If it's illegal and you're big and powerful, don't worry about it, you can probably get away with it with little damage to your business or career and keep almost all of you cash minus legal fees.

    • It seem like you're right. In no way shape or form did anyone break either the spirit or the letter of the law here. Yet,

      First Circuit holds that prohibitions found in a website's Terms of Use can be used to establish that a visitor to that site exceeded his authorized use thereof for the purposes of establishing a violation of the Computer Fraud and Abuse Act

      Which says to me I can say, "oh, you can't read my website while wearing sunglasses." I could be wrong here, but according to this ruling, I can ju

    • Re:Rules of Thumb to Live By (Score:4, Insightful)

      by poot_rootbeer ( 188613 ) on Thursday May 15, 2003 @10:17AM (#5964393)
      It seems that the only law these days is "He who can afford the best, most impressive lawyer wins."

      I can see how someone could be fooled into believing this, if the only news they get about what's happening in the courts is the fearmongering they read in "Your Rights Online."

      I suggest you obtain a copy of the verdict records from a court that deals with a wide variety of cases. Ten bucks says you'll come to the conclusion that regardless of who the legal counsel is on both sides, justice is truly served far more often than not.

  • huh? (Score:5, Funny)

    by Anonymous Coward on Wednesday May 14, 2003 @10:56PM (#5961037)
    Who is this "Illegal" person and why are we asking him questions about Star Trek characters?

    I'm not "Illegal," but I'll answer. Seeing as how he was killed off in the last movie, I think it's safe to say that no, Data is not mining for product pricing.

    (In other words, you illiterate clods need to be more careful with your commas.)

  • well....duh (Score:5, Insightful)

    by mrpuffypants ( 444598 ) <.moc.liamg. .ta. .stnapyffuprm.> on Wednesday May 14, 2003 @10:57PM (#5961039)
    Look, I I can visit a web site and the business (Let's say Amazon) publicly posts their prices for anybody to see then you sure as hell can use them! If suddently using bots to do work are illegal then I'd wadger that every shell script that I write is an affront to US Laws. Rotating log files and all sorts of other "make my job easier so that I can play Quake" scripts are perfectly legal, so how the hell can it be questionable just to go to a site and record prices???

    Jebus, please help the Unites States Gub'ment!
    • so how the hell can it be questionable just to go to a site and record prices???

      Let's not forget that this was a legal ruling, and some of judges don't know squat about technology. The lawyer for the defense probably showed the judge a web page with prices on it and the judge assumed it was "hacking" or somesuch.
      • good point...we truly do need a lot more lawyers who are technically adept so that the future and present intellectual-property and tech-related cases can be argued by competent attorneys.

        EFF [eff.org]

    • Re:well....duh (Score:5, Informative)

      by Jeremy Erwin ( 2054 ) on Thursday May 15, 2003 @12:03AM (#5961346) Journal
      Amazon says you can't.


      Amazon.com grants you a limited license to access and make personal use of this site and not to download (other than page caching) or modify it, or any portion of it, except with express written consent of Amazon.com. This license does not include any resale or commercial use of this site or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of this site or its contents; any downloading or copying of account information for the benefit of another merchant; or any use of data mining, robots, or similar data gathering and extraction tools. This site or any portion of this site may not be reproduced, duplicated, copied, sold, resold, visited, or otherwise exploited for any commercial purpose without express written consent of Amazon.com.


      I am guessing that the prohibition on "visit[ing] for any commercial purpose" precludes me from actually purchasing their wares.
  • First, the obvious: IF you are concerned about YOUR obligations towards your clients, talk to a lawyer competent in the areas of law this involves.

    Second.. a cursory reading of the case you linked says that case was not about scraping in general, but about a consultant doing scraping on behalf of a client who was not permitted lawful access to the site/service being scraped by way of a terms of use.

    In other words: one party was indirectly accessing the site by way of a second party... it was determined th
  • I think they'll get a clue RSN.

  • web servers are not protected (Score:5, Insightful)

    by Provincialist ( 572648 ) on Wednesday May 14, 2003 @10:59PM (#5961053)
    accesses a protected computer without authorization, or exceeds authorized access [18 U.S.C. 1030(a)(4) of the CFAA]

    How does one receive authorization to access a web server? Hmm, maybe with a simple html GET? The basic fact here is that of judicial cluelessness. If I put information on a public web server, pretend to "protect" it with a disclaimer (of everything) at the bottom of the page, and then get pissed off because somebody browsed that information, I'm an idiot. In addition, I am legless in court. Web servers make information available to the world. If I had wanted to make information available to certain parties that I trust not to compete with me, I should have set up a secure server with some provision for authentication and authorization.

    It really is that simple

    later,
    Jess

    • While I agree wholeheartedly with your argument, the courts do not. Another example from the same paper is even scarier. In a similar case (Register.com vs. Verio), the court said the use of search robots to gather information may be illegal regardless of whether it broke the site's terms of service:

      "Instead, the Court concluded that the mere fact that Register.com had decided to sue Verio meant that Verio's use of the search robot was without authorization: 'because Register.com objects to Verio's use

  • Easy fix. (Score:2, Interesting)

    by Photar ( 5491 )
    The problem is that when you sell a commodity like a TV or a vcr and the only difference between them is price you can't exactly maintain a high profit margin. What they need to do is obfuscate the prices so that its next to impossible to compair products. Thats how cell phones work.

    This phone has 500 any time minutes for 3 cents a minute from your calling area roaming is 10 cents a minute, unlimited text messaging, 800 night and weekend minutes is free for the first 6 months and has rollover.

    This other

  • Depends on website design? (Score:5, Interesting)

    by thesupraman ( 179040 ) on Wednesday May 14, 2003 @10:59PM (#5961060)
    Maybe it is just me, but I would think that ANY information that has been put in a public area (ie: the web) and which can be accessed without agreeing to some form of contract (even if it's just a click-through before you can access it) is public information.

    The page can of course still be copyright. That should not stop you using the information it contains.

    Online vendors are tring to play the 'have your cake and eat it' game here - if you make information available to the 'public' then it is available to your competitors. If they can hire a preson to read it and inform them, then they can use a program to harvest the same.

    Copyright would come in to play if you copied pages in total, however if you are using just a small fraction of the information, and not copying the layout then I think it would be very hard to argue a copyright violation.

    Some real-world stores combat this by not having marked prices, which is a pain, but their right - the same should apply to online vendors, they should keep their secrets a secret.

    Of course, if your script made a fake 'order' to gain the information it would be different, as it is miss-representing itself and IMHO that is not a good thing (tm).

    Then again, it all comes down to whos lawyers are the hungriest that month I guess, which is a very sad thing.

    • No. (Score:3, Interesting)

      by www.sorehands.com ( 142825 )
      Just because an area is accessible to the public does not mean that you can do whatever you want. If you liken a website to a bookstore, ou are permitted to go in a browse, but not bring in a copy machine and copy books.

      There is an implied license to browse a website, but that can be overidden by a terms of use or a robots.txt file.

      A scrapper would have a hard time claiming they didn't know, if there was a robots.txt file.

      Years ago, there was something similar by the National Baseball League against Moto

      • Re:No. (Score:2)

        by tupps ( 43964 )
        Just because the people where used in the process doesn't necessarily mean that automated scraping of scores is illegal, it just means it hasn't been tested in a court of law.
      • but that can be overidden by a terms of use or a robots.txt file

        Yeah? They can bite my shiny metal ass!!

  • well, this is partially related (Score:3, Interesting)

    by trmj ( 579410 ) * on Wednesday May 14, 2003 @11:00PM (#5961064) Journal
    Back when the online deals company was given a C&D for posting retailers' black firday ads early, not even three days later the store I worked at was handing out memos to the employees saying that anybody caught releasing "marketing or promotional information" early would be terminated immediately.

    Apparantly, they have a problem when other people know their information, but they have no problem making weekly trips to the competitor store down the road to gather information about prices and specials.

    It makes me wonder if, since people are being thrown litigation for data mining, is it legal for people to walk down to the competition and get the information in person?
    • It depends on how that information is gathered. If it's publically avaliable (e.g. posted on shelves under the product), then of course it is legal to use that information however you want. However, if (as I suspect the case in the black friday ads) the information is not public knowledge, then you may be on slippy ground.

      IANAL
  • FBI comes knocking at wessman's front door

    "Do you hear that, Mr. Wessman? That is the sound of inevitablility. That is the sound of your death. Goodbye, Mr. Wessman"

  • So is the Internet like a Public Square??? (Score:3, Funny)

    by ProfMoriarty ( 518631 ) on Wednesday May 14, 2003 @11:01PM (#5961075) Journal
    If I place a flyer advertising prices in a public square ... and my competition sees it and LOWERS the prices (creating competition) ... would I have the right to sue that company?

    NO

    Ok then ... if I place my prices on the internet, along with my competition ... could I sue then?

    NO

    So ... how could I "protect" my business against someone who could see my prices anytime? Well, I could create an inconvience for my customer, by listing all the prices as "call for price" ... or force my customers to become a "member" to my website to gain access to my price list. But this would not guarentee that my competition wouldn't enter.

    Basically, if it's on the internet, AND it's for public consumption, then it's fair game. Why couldn't you use a price harvester robot to gather prices? Like in other posts, they're not copyrightable, trademarkable, patentable, nor are they a trade secret.

    Hmmm ... price harvester robots ... thanks for the idea, I'll have to patent it.

  • Easy answer (Score:5, Insightful)

    by Lurgen ( 563428 ) on Wednesday May 14, 2003 @11:05PM (#5961097) Journal
    Once their prices hit the Internet, they're in the public domain. It would be like posting your prices in the window, and complaining that a car driving past could photograph them.

    We all know that bots crawl the web - Google, Altavista, spam-bots... they're all common knowledge. You put information on a website, and it's going to be viewed by an automated process. Surely with that knowledge, it's ridiculous to think you can ban people for using the information you've posted publicly in whatever way they desire.

    Perhaps these companies (airlines, computer stores, whatever) need to start offering their services at the price they really mean to sell it for, rather than this stupid haggling they expect from us. Or maybe it's time they focused on quality of service, value-add, etc rather than price wars (which never help anybody in the long term).

    Bottom line? If you don't want your competitors seeing your prices, don't make them available to them - this means no junkmail, no spam, no website, no prices in the store window, no prices inside the store, nothing.

  • Examples... (Score:5, Interesting)

    by oaf357 ( 661305 ) on Wednesday May 14, 2003 @11:10PM (#5961124) Homepage Journal
    Buy.com was built around a huge server farm that scoured the web and found the best prices for products it sells and then beat those prices (to the best of its ability). That has changed a little now but buy.com was built from that.

    Also, Pricewatch [pricewatch.com], Pricegrabber [pricegrabber.com] and Froogle [google.com] scour the web for prices and create search engines out of them so consumers can find the best price.

    I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.

    Hope that helps.

  • The US Code... (Score:4, Informative)

    by Pettifogger ( 651170 ) on Wednesday May 14, 2003 @11:11PM (#5961128)
    It seems that the US Code and the First Circuit make this one pretty clear. If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it. If your competitor's site requires you to agree to such terms, and those terms prohibit data mining, then you can't data mine there. Simple.

    As for the ethical part of telling your employer about this... well, first, remember, this is just a decision of the First Circuit. If you live in a different Circuit, then it may or may not be binding on you. I know this jurisdictional stuff can be a little confusing, but a decision by a Circuit only affects the jurisdictions within it. Only the US Supreme Court (generally, I know there are federal tax, patent, admiralty, etc. courts, too) can make decisions that are binding on the entire country. If you're not sure, check with your corporate counsel. And it might be a good idea to forward the case to him anyway, you might be able to pick up some "bonus points" from your boss for being an especially conscientious employee.

    • If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it.

      Aside from the well-known problems with any click-through agreement (contract between unknown parties, software circumvention, lack of notarization, etc.), the additional flaw in this case is provided by web archives. If you don't want to have to look at a click-through page before reading your competitor's

      • Aside from the well-known problems with any click-through agreement (contract between unknown parties, software circumvention, lack of notarization, etc.),...

        You are talking out of your ass. An enforceable contract has four elements:

        1. A meeting of the minds between the parties to the contract. Evidence of meeting of the minds can be provided by incorporating a unique element in the contract and requiring the web surfer to key it back in before they are allowed to proceed.
        2. An exchange of "consideration"

  • hmm, anybody rfta? (Score:5, Insightful)

    by lingqi ( 577227 ) on Wednesday May 14, 2003 @11:13PM (#5961144) Journal
    on the ground that Explorica had used confidential information obtained from EF to assist in obtaining this pricing information in violation of confidentially agreements executed by former EF employees now working for Explorica.

    seems like it's the using confidential information part that got the scrapper capped.

    I don't see why accessing *public* information be problematic.

    the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).

    last question, though - why the heck would you ask this kind of stuff HERE? wouldn't a law-forum be a better choice?

    • the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world

      What, never heard of a 2-drink minimum?

    • Re:hmm, anybody rfta? (Score:2, Informative)

      by Anonymous Coward
      I work for one of the major online travel sites.

      If you screen scrape us, and we notice it (and we very often do), your IP address will be blocked by our firewalls.

      What most people don't realize is that very often any search performed on these sites costs the company money. In many cases, if you search for, say, a hotel on Expedia or Orbitz or Travelocity, those companies are paying one of the major Hotel reservation systems for their results from that search.

      So, if someone is screenscraping our site, ea
  • Doesn't Google's Froogle [google.com] service do exactly this?

  • Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually?

    I read an article a year or so ago...can't remember where, maybe here, that Besty Buy and Circuit City do discourage this activity. The problem the article brought out is how do they tell the difference between comparing prices and people data mining? J

  • Seek real legal advice. (Score:5, Insightful)

    by Gerad ( 86818 ) on Wednesday May 14, 2003 @11:18PM (#5961163)
    I am not a lawyer.

    Slashdot is not a lawyer.

    Slashdot is not a replacement for a lawyer.

    Individual posters on slashdot may be lawyers, but are you really willing to trust your future to what some random person online says, when they could be a lawyer, but could also be some 14 year old kid who thinks it's amusing to screw with people?

    Repeat after me:

    I will seek proper legal advice.

    Seriously, this comes up time and time again. If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO. Sending in an Ask Slashdot is fine for theoretical questions, but when your ass is at stake if a lawsuit comes around, do you really want to trust your future to the legal advice given to you by Anonymous Cowards and karma whores?
    • Every time someone asks a legal question on Slashdot, people like you come out of the woodwork and point out that Slashdot is the same as going to a lawyer. You don't say. Do you really think everybody else is stupid?

      Discussing legal issues is not just a business for lawyers. Non-lawyers can give each other useful pointers. And non-lawyers actually have an obligation to determine whether their legislators are doing a good job with the laws they enact and judges they appoint, and a healthy discussion i

      • You don't say. Do you really think everybody else is stupid?

        Stupid? No. But the number of people who seem to think they are lawyers is very large, and not just on Slashdot either. I can't count the number of times in my real life I've discussed intellectual property issues and not only has the other person been very, very wrong, but I was not even able to get them to listen to me.

        I'm not a lawyer, but I've taken a close interest in that sort of thing and I know the basics very well.

        As a Slashdot example

    • If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO.

      And how many lawyers know the legal implications of soliciting anonymous HTTP GET requests? How many know the semantic differences between GET and POST requests, and their relationship to the computer fraud and abuse statutes?

      Furthermore, most law in this area is being made right now by the people actually working in this field. I.e., a large proportion of /. participants. The last thing we need is fo

  • Why waste time in the legal system? (Score:4, Informative)

    by x00101010x ( 631764 ) on Wednesday May 14, 2003 @11:22PM (#5961182) Homepage
    Filing lawsuits to protect your price information is just dumb, not to mention waste (if not abuse) of the legal system.

    Personal feelings about freedom of information aside, and just from a coder's POV, here's my solution.

    If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).

    Coders are cheaper than lawyers (at least those needed to write such code as this).

    Sure, the compition could pay more money to get somebody to develop better OCR to read each and every dynamically generated GIF, but most people require proof reading of OCR data, which leads to even more cost.

    Something I learned from my Uncle who works with the DOD is this: Any lock can be picked; Any encryption can be broken. It's just a matter of if it's worth the time and money to get what's inside.

    In short, with a little one time cost, the company that doesn't want it's prices scraped can just make it so hard to scrape their prices that it's not worth it. The price of scraping the graphically displayed price tags would also be an ongoing cost of software and proofreaders that would dip into profit margins, which management at the company that desires the scraping won't like.

    It's not perfect, but it's better (and more bankable) than going whining to the legal system. (Especially since coders are generally cheaper than lawyers).
  • If you have a genuine concern you should raise it. (As always, if it isn't in writing, it never happened.)

    Then let it go. If you want to be extra-double sure, get your own lawyer.

    You, however, are not their lawyer. It is not your job to advise them on legalities.

    -Peter

  • Same as in the Physical World (Score:3, Interesting)

    by serutan ( 259622 ) <snoopdougNO@SPAMgeekazon.com> on Wednesday May 14, 2003 @11:29PM (#5961204) Homepage
    Anybody who publishes information about their business runs the risk that a competitor will get hold of the information and use it in some way. This has always been a fact of life in the physical world. As computers came online in recent decades many companies have maintained databases of information about competitors' products. The Internet doesn't change any of this.
  • Reading the summary of the case it looks like the court ruled that Zefer (the developers of the scraper software) could not use the scraper on EF Cultural Travel's website because their client Explorica was legally enjoined from using a scraper. Zefer are not permitted to assist Explorica in violating the injunction against them. The injunction against Explorica was issued because Explorica used confidential information to assist in gathering the pricing information when, as former EF employees, they had si

  • Read the case... (Score:5, Informative)

    by anubis ( 87418 ) on Wednesday May 14, 2003 @11:52PM (#5961302)
    Read the case...EF Cultural Travel BV v. Explorica hinges on the fact that the defendant company hired an ex-programer from the plaintif company. The programmer had special knowledge of codes used in the pricing (which he had signed a confidentiality agreement not to disclose). When he made the scrapper program he violated the confidentiality agreement.

    It was the violation of the confidentiality agreement that the court held was illegal.
    As for whether you should tell your employer, it depends on your employment agreement! :) Depending on how the contract is written you could be jointly liable.

    While this is a 1st Circuit case, it has been followed by the 5th Circuit (Ingenix, Inc. v. Lagalante) and cited in cases in the 7th and 9th Circuit.

    Hope this helps.

    --me
  • I remember reading something about a year ago about someone who got arrested at (I think) Circuit City for writing down the prices; on a laptop he brought with him, as I recall. A quick Google search didn't turn up anything; maybe someone will remember more information than I can recall.

    Also, I've seen some online sites seem to have "this information is for your personal use only" type 'licenses' for their prices; the notices I've seen are usually in tiny print in some legal page link no one ever clicks;
  • and I think they have taken people to court over it.

  • I'm reminded (Score:2, Informative)

    by Nexzus ( 673421 )
    of a man named Ronald Kahlow and his troubles with Best Buy [quux.org] back in 1997.

    • Re:I'm reminded (Score:4, Interesting)

      by shumacher ( 199043 ) on Thursday May 15, 2003 @08:36AM (#5963449)
      Both Best Buy and the judge were off their rocker. Best Buy can ask him to leave. The store is private property. If he is a customer, they should have allowed him to continue to calculate prices. If Best Buy asked him to put his laptop away, he should have complied. Further, the article mentions other stores that claim to not have a policy against Mr. Kahlow's behavior. I know for a fact that one of them actually is okay with Mr Kahlow's behavior, but does have a policy (and not just an "unwritten" one)against competitor's doing what Mr. Kahlow was doing. (Which is a different thing - and more on topic - one means a potential sale for them, the other means potential sales for their competition) Their policy is to ask the suspected competitor what they are doing, and then if confirmed as a competitor, they are to make a specific statement essentially revoking that individual's right to visit the store, and then, they are to ask them to leave. When that store shops a competitor, policy compels them to wear normal street clothes, to not use any automated devices, and to admit their affilliation and leave if asked.
  • What he's doing isn't really data mining. Data mining is the process of discovering patterns in data which are not known ahead of time, such as the infamous "beer and diapers" correlation.

    That said, I don't understand why the author is worried. I can't see how looking at publicly posted prices could be considered illegal.
  • People who studies economics faces some irreal hipothesis in text books. The first topic most students have to deal with when taking the microeconomics course is when you have a big group of firms selling the very same product. If the buyer has perfect information about prices hi will choose the lowest price. The buyer's choice will influence the behaviors of all other firms that will tend to get their prices down to beat the one choosed by the buyer. We will have a dynamics that will make the price go down
    • This guy (?) is (non-redundantly) describing an important influence on the phenomena being discussed here. Also, the smartest guy I ever worked with couldn't spell either {g}.

  • House rules (Score:2, Interesting)

    by scgops ( 598104 )
    You see signs everywhere you go:

    -Shirt & shoes required.
    -No loitering.
    -No soliciting.
    -Check all bags at counter.
    -No more than two students allowed in store at one time.
    -Parking lot, bathroom, etc. for customer use only.

    Just because a building (or a web site) is in a public place doesn't mean that everyone is free to do whatever they want. Business owners are free to create house rules that everyone needs to follow.

    Similarly, web sites can legally restrict what you are allowed to do when you visit th
  • In writing a bot, you are using an automated system to mine a company's website while the benefit to them is questionable - since, of course, you're just mining prices *for a competitor*, there's no chance that they'll see an increase in sales - and if the data is being used by the competitor effectively, that company will lose sales.

    In addition, it is important to remember that while the "freeness" of data itself can be debated, accessing data is *not* free. To provide the pricing data over the web - inte

  • Prices are a fact... (Score:2, Interesting)

    by Kris_J ( 10111 )
    Prices are a fact and therefore should not gain any protection offered by the copyright, trademark or patent systems, broken as they are.

    If a bot activates a click-through agreement, does anyone hear it fall?

  • Screen scraping is not data mining (Score:4, Informative)

    by Call Me Black Cloud ( 616282 ) on Thursday May 15, 2003 @08:08AM (#5963239)
    Screen scraping is data gathering. Data mining is looking for trends or patterns in data you already have. Getting the nuggets out of your data to continue with the mining analogy. From this [thearling.com] presentation titled "An Introduction to Data Mining Technology" data mining is defined as "The automated extraction of hidden predictive information from (large) databases".

    The bottom line is this: when you put this work experience down on your resume don't say you were data mining. Companies looking for that experience will ask you hard questions you don't know the answers to and you will be embarrassed.

  • here is a similar case (Score:4, Informative)

    by mcguyver ( 589810 ) on Thursday May 15, 2003 @04:47PM (#5968182) Homepage
    Here is a related incident:

    http://news.com.com/2110-1017-944258.html [com.com]

    Bargain Network spidered real estate prices on homestore.com/realtor.com and posted them on the bargain.com website. Homestore sued and the case was settled out of court. I wish it was not settled out of court because that would set up a precident.

    In my opinion you are asking for the problems. Taking a case like this to court and winning would be difficult. At the very least it would be a serious legal expense.

    The last time I checked the rules for Froogle you had to be the actual merchant that ships the product in order to show up in their index. If you are spidering a merchant then you are an affiliate, the products do not originate from you so you would be exluced from Froogle. Froogle does not allow you to sort products by price - so obviously what you plan on doing is different. Froogle also gives merchants the option to be excluded from their index.

    My advice is this - get a lawyer because one will surely be contacting you. Familiarize yourself with these phrases: false advertising, breach of contract, and unfair competition.
  • AARRRGGH. I hate ./ sometimes.

    From the court summary of the decision:

    The court affirmed a preliminary injunction enjoining defendant Zefer Corporation ("Zefer") from utilizing a "scrapper" tool it designed to obtain pricing information from plaintiff's website on the ground that Zefer was doing so to assist defendant Explorica, Inc. ("Explorica"), which was itself enjoined from such activity by virtue of its improper use of confidential information obtained from plaintiff to aid it in gathering this information.

    This means that Zefer was prohibited from mining the data because their client (Explorica) was prohibited, because Explorica and Zefer had gained access to the data by exploiting confidential information. Which is another issue entirely from data mining...

Slashdot Top Deals

You are always doing something marginal when the boss drops by your desk.

Close