Is Data Mining for Product Pricing, Illegal?

wessman asks: "I started to read Orin S. Kerr's 80-page paper looking for how his proposal would pertain to: ripping music/movies, P2P, corporate espionage, and lastly, the use of web scraper robots. Little did I know just how relevant his paper would be in regards to that last item! Kerr makes note of EF Cultural Travel v. Explorica in which Explorica is caught hiring a consultant to program a scraping robot to gather pricing information from a competitor, EF Cultural Travel. Well, I do consulting on the side from home and am currently working a project whereby I gather pricing information from all the major travel conglomerates (Orbitz, Expedia, Lodging.com, WorldRes, Sabre, etc.) so that the travel booking business that hired me can meet or beat all their prices. Granted, the circumstances of the Explorica case are different and the case was an example of an extreme ruling, but my questions to the Slashdot community are: Do I notify the company that hired me of the Explorica case? Why is using a scraper robot so different from, say, walking into Best Buy with a handheld and recording product pricing manually? Should I continue with this project and the similar projects I do in this area of programming?" Now, add in the text in the "deliverables" section of this press release and it seems we may have some contradictory information. Who is right, and under what circumstances is price harvesting off of the internet not allowed?

I swear

[cultobill (72845)]

In what sane land would PRICES be protected under law? You can't really keep them secret, so "trade secret" is right out. It's not a identifying mark (unless you're a dollar store), so much for trademark. There's nothing useful that hasn't been done before, fuck patenting them. Copyright? It's a simple derivation of what the supplier charges you.

There is nothing creative about pricing stuff. Good lord.

Re:I swear

[Gortbusters.org (637314)]

Look at pricewatch.com [pricewatch.com], it already goes around collecting price data from many online stores.

I don't see why this is such a big problem... one site creates competitive prices based upon other sites' prices. In reality if a consumer reaches your site on the internet for your product they probably didn't do it by accident. They will evaluate all aspects of the business (licensing, service/support, upgrade cost, security of the site, etc) before they jump ship to another site to save a few bucks.

I'm sure D

Re:I swear

[Sparr0 (451780)]

Pricewatch doesnt mine. Companies PAY for the privelege of listing their prices on pricewatch.

Re:I swear

[anonymous loser (58627)]

Pricewatch doesnt mine.

No, but froogle.google.com does.

Re:I swear

[CTho9305 (264265)]

WARNING: You can see I write crappy perl / shell script. I make no guarantees about this code.

NOTE: replace all instances of "ABC" with "|".

parse.sh:
#!/bin/bash
#Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).
#barton
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep 333 ABC cut -f1,4 -d ' ' > XP333.dat
#XP
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i xp ABC grep -v 333 ABC cut -f1,4 -d ' ' > XP.dat
#Apparently I have too many junk characters
#MP
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i mp ABC perl -pe 's/\.//' ABC perl -pe 's/GHz/00/' ABC cut -f1,4 -d ' ' > MP.dat
# Celeron
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//' ABC perl -pe 's/\s+/ /' ABC grep -i celeron ABC cut -f 1,3 -d ' ' ABC perl -pe 's/ 1GHz/ 1.0GHz/;s/\.//;s/GHz/00/' > celeron.dat
#P4
lynx -dump http://www.pricewatch.com/menus/m3.htm ABC egrep 'upABCdnABC - ' ABC cut -b5- ABC perl -pe 's/\s+/ /' ABC cut -f1,3- -d ' ' ABC perl -pe 's/\[.*\]//;s/\s+/ /' ABC grep -i 'pentium 4' ABC egrep -i "sockABC2\.ABC3\." ABC grep -v -i 400MHz ABC perl -pe 's/ 533MHz//;s/GHz/00/;s/ Sock 478//;s/\.//' ABC cut -f1,4 -d ' ' ABC cut -b -8 > pentium4.dat
gnuplot gnuplot.script > ~/www/out.png

Anyway, I wrote this because I was bored and wanted to see what a good price point was for current Athlons. If you examine the graphs carefully you might note that the XP's are not properly differentiated. Some are 333s and marked as that, others aren't marked properly, etc. With the new 400s, it gets worse. For the P4s, I got a little luckier because the speed ranges don't overlap as much. I think I'm going to not differnetiate between the various FSBs of Athlon XPs because the prices are close enough anyway.

Anyway, it has served its purpose by helping me find a point where the processors are reasonably fast, and the bang for the buck is decent.

gnuplot.script

#Copyright CTho9305 2003. You are given permission to redistribute this file provided this copyright notice is left intact. You may modify it as you want. Please share any modifications (you are not required to).set terminal png color
set xlabel "Speed (MHz or rating)"
set ylabel "Cost ($USD)"
set title "Speed vs. Cost"
set grid
set time
set linestyle 1 lw 3
plot "XP.dat" using 2:1 title "XP" with linespoints, \
"XP333.dat" using 2:1 title "XP333" with linespoints, \
"celeron.dat" using 2:1 title "celeron" with linespoints, \
"MP.dat" using 2:1 title "MP" with linespoints, \
"pentium4.dat" using 2:1 title "P4" with linespoints

Anyone know how to change the text font, or the thickness of the lines?

Sample output [cmu.edu]

Re:I swear

[whereiswaldo (459052)]

Here's a niche for P2P software... obtaining data from a website in a distributed way so as not to stick out in the website logs the way a one or a few download clients would. The collected data could be processed remotely or uploaded to a central server.

Not the pricing -- the timing

[RalphSlate (128202)]

There may be precedent for this. eBay was able to convince a judge [com.com] to bar spidering of their site.

There is another legal concept called "Unfair Competition" which links copyright and facts.

Normally, facts cannot be copyrighted [findlaw.com]. However, this law seems to kick in when one company compiles and publishes time-sesitive information that it has taken from a direct competitor in a way which "free-rides" on the efforts of the competitor. It is usually applied to news organizations, when one newspaper sends a reporter to Iraq and a second newspaper (perhaps an evening edition) uses the "facts" in the first newspaper's article to publish the very same news.

I could see the instantaneous publishing of all competitors' prices as a violation of this legal theory.

Re:I swear

[Sloppy (14984)]

Actually, there's a lot of creativity that goes into pricing things.

Ever see this: $39.95 ? The .95 cuts .05 cents off the price but greatly increases the chances you'll buy it.

How about 2/6.00? Unless they say otherwise, that means 1's only $3, but again, you're more likely to buy the two. A supermarket by me commonly sells things like limes and small bags of snacks at 4/$1.00, rather than just $.25, which i guess seems cheap.

A lot of the delis around me have sandwiches for 4.63, which with tax is exactly a finski. No numeric change means they can ring you out faster.

Perhaps the first time someone came up with these techniques, it was an act of creativity. But they are now well-known. When someone prices something as 39.95, they are not being creative; they are merely "using the .95 trick."

Besides this, figuring out what things are worth is a very exact science when you live, as we do, in a vastly service based economy. Prices have nothing to do with the actual cost of physical production, and everything to do with the public's perceived value of the product in question. This is how GM can make two identical trucks, label one GMC, and get another $3000 for it because GMCs are "professional grade and therefore more reliable." This is how Victoria's Secret can get $30+ for maybe a quarter square foot cloth and a pair of wires, and in fact get more money when they use less material or "exotic" materials like satin (which often cost less than cotton).

Ok, these are much better examples of creativity. The person doing the pricing is actually doing work and making real decisions.

But does that entitle their work to special protection? Let's go back and look at why we have "IP laws" at all, lest we forget the purpose and perversely misapply the law. In the United States, the Constitution empowers congress on this basis:

To promote the progress of science and the useful arts by securing for a limited time to authors and inventors the exclusive right to their respective rights and discoveries...

It is effort for you to write a book, and if you didn't have a monopoly on the sales of that book, your effort would be wasted from a commercial perspective and you might not choose to expend that effort. So we give you a copyright. You get what you want -- compensation for your effort -- and we get what we want -- promotion of the useful arts. Perhaps that promotion is somewhat delayed (until after the copyright expires) but we'll eventually get it.

Now let's look at your sophisticated pricing example, in that light. You expend effort in figuring out just the right price for your widget, and you would like a monopoly on the fruits of that work. If we grant that to you, then you get what you want -- compensation for your effort -- and we get ... hey, what do we get? Does the creative act of figuring out that $30 is the most profitable price for your underwear, help us in some way? Does it promote the progress of the sciences or useful arts? I don't see how.

I see that there's something in it for me, to grant copyright on creative works such as books. I don't think there's anything in it for me at all, to grant copyright on prices, even prices that take effort to optimize.

And without a monopoly, you won't have sufficient incentive to do the work of determining the most profitable price for your satin underwear or truck? As if!

Just like Popeye

[by Anonymous Coward]

Anybody remember the Popeye cartoon where Popeye opens a car wash, and then Bluto opens a car wash right across the street?

When Popeye posts his price, Bluto beats it by five cents. Then Popeye beats Bluto's by five cents.

It goes back and forth until Popeye is washing cars for free.

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

Re:Just like Popeye

[87C751 (205250)]

I was tempted to call this an infinite loop, but I doubt a retailer would pay you to take their products.

Last month, I got a $40 rebate on a $39.95 router.

Re:Just like Popeye

[tupps (43964)]

How much was postage?

Have you actually received your rebate?

That Reminds Me...

[Esion Modnar (632431)]

of the old joke about this fella who was selling everything in his store at a loss.

When somebody asked him how he can make a living like that, he replied:

"Volume!"

TITLE HAS POOR GRAMMAR

[by Anonymous Coward]

EDITORS: A better way to phrase that headline is as follows:

Is It Illegal to Data-Mine for Product Pricing?

Thank you.

Re:TITLE HAS POOR GRAMMAR

[dipipanone (570849)]

Commas just don't get used enough.

Perhaps not, and I don't wanna get all grammar -nazi prescriptive here, but if I was writing that strapline, I would have used a colon.

Data Mining for Product Pricing: Is It Illegal?

Ummm..

[deadgoon42 (309575)]

My grandmother calls this "shopping around." The only difference is that someone else is doing all the work.

Price Scraping?

[by Anonymous Coward]

Actually, price scraping was done in a very low-tech way a reasonably long time ago by a pretty well-known businessman: Mr. Sam Wal-Mart. Early in his career, he would dumpster-dive his competitors to find out the prices his competitors paid for their goods, the contracts they had with their suppliers, etc. This provided him with "insider information" so he'd know how to prepare his pricing in a more strategic fashion and obviously out-compete them where it counts: financially.

Re:Price Scraping?

[by Anonymous Coward]

Wrong. Read Sam Walton: Made in America: My Story [amazon.com]. Sam Walton says that was a story put out by his competitors to disparage his name and he never did anything of the sort.

Re:Yeesh.. give it a rest. (OT)

[Lt Razak (631189)]

Are you kidding me? I'd love the internet to "only" be the green screen hooked up to a MUD and IRC that I had back in college.

The corporation will continue to hump the internet like a dog in heat until it becomes as regulated, watered down, and crapped out like the NBC news.

TOS and more ideal markets

[weston (16146)]

I think any large business where the pricing structure isn't directly related to costs is probably deeply afraid of agents that aggregate their data with competitors. You end up with a more ideal market, a more frictionless market, if you will, and they'll be forced to compete on narrower and narrower margins of profit. Of course they'll want to throw up barriers to that.

But I'll bet this issue comes down to Terms of Service and what a company can reasonably expect to be able to legally require/forbid about the use of data provided via an automated means...

You have a contract?

[gpinzone (531794)]

As long as you get paid, let them worry about the lawsuit. They're the ones who are going to actually use it. Keep your mouth shut.

Re:You have a contract?

[vandan (151516)]

Would you apply this line of reasoning to ALL areas?
What if the job were researching Bush's all-feared biological weapons?
Or GM products?

The problem is that if everybody decides to look the other way (and everyone can find a reason why they should take their money and shut up), then some pretty fucked up things get done, and people are left wondering, "How did it come to this?"

Now I'm not saying that the world is going to end because someone's harvesting prices off the net. I'm just questioning your "Me first, no-one else 2nd" argument.

Re:You have a contract?

[stiller (451878)]

I do hope that every /. reader at least can discriminate between building life-threatening devices and products that may or may not just fall under some crazy copyright law. It's the user's responsibility not his. They assigned him, they carry the consequences.
It's your way of extreme reasoning that gets us nowhere. We must be reasonable, and it's reasonable to expect that no judge in his right mind would convict anyone for this.

Rules of Thumb to Live By

[release7 (545012)]

I'm not going to pretend to know what the laws are in this case. It seems that the only law these days is "He who can afford the best, most impressive lawyer wins." Here's some other cynical words of wisdom I've come to believe:

If powerful people get screwed, it's illegal.
If it forces large corporations have to work harder to earn a profit, it's illegal.
If it give the little guy a leg up or levels the playing field in any way, it's illegal.
If it's illegal and you're big and powerful, don't worry about it, you can probably get away with it with little damage to your business or career and keep almost all of you cash minus legal fees.

Re:Rules of Thumb to Live By

[poot_rootbeer (188613)]

It seems that the only law these days is "He who can afford the best, most impressive lawyer wins."

I can see how someone could be fooled into believing this, if the only news they get about what's happening in the courts is the fearmongering they read in "Your Rights Online."

I suggest you obtain a copy of the verdict records from a court that deals with a wide variety of cases. Ten bucks says you'll come to the conclusion that regardless of who the legal counsel is on both sides, justice is truly served far more often than not.

huh?

[by Anonymous Coward]

Who is this "Illegal" person and why are we asking him questions about Star Trek characters?

I'm not "Illegal," but I'll answer. Seeing as how he was killed off in the last movie, I think it's safe to say that no, Data is not mining for product pricing.

(In other words, you illiterate clods need to be more careful with your commas.)

well....duh

[mrpuffypants (444598)]

Look, I I can visit a web site and the business (Let's say Amazon) publicly posts their prices for anybody to see then you sure as hell can use them! If suddently using bots to do work are illegal then I'd wadger that every shell script that I write is an affront to US Laws. Rotating log files and all sorts of other "make my job easier so that I can play Quake" scripts are perfectly legal, so how the hell can it be questionable just to go to a site and record prices???

Jebus, please help the Unites States Gub'ment!

Re:well....duh

[Some Dumbass... (192298)]

so how the hell can it be questionable just to go to a site and record prices???

Let's not forget that this was a legal ruling, and some of judges don't know squat about technology. The lawyer for the defense probably showed the judge a web page with prices on it and the judge assumed it was "hacking" or somesuch.

Re:well....duh

[Jeremy Erwin (2054)]

Amazon says you can't.

Amazon.com grants you a limited license to access and make personal use of this site and not to download (other than page caching) or modify it, or any portion of it, except with express written consent of Amazon.com. This license does not include any resale or commercial use of this site or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of this site or its contents; any downloading or copying of account information for the benefit of another merchant; or any use of data mining, robots, or similar data gathering and extraction tools. This site or any portion of this site may not be reproduced, duplicated, copied, sold, resold, visited, or otherwise exploited for any commercial purpose without express written consent of Amazon.com.

I am guessing that the prohibition on "visit[ing] for any commercial purpose" precludes me from actually purchasing their wares.

web servers are not protected

[Provincialist (572648)]

accesses a protected computer without authorization, or exceeds authorized access [18 U.S.C. 1030(a)(4) of the CFAA]

How does one receive authorization to access a web server? Hmm, maybe with a simple html GET? The basic fact here is that of judicial cluelessness. If I put information on a public web server, pretend to "protect" it with a disclaimer (of everything) at the bottom of the page, and then get pissed off because somebody browsed that information, I'm an idiot. In addition, I am legless in court. Web servers make information available to the world. If I had wanted to make information available to certain parties that I trust not to compete with me, I should have set up a secure server with some provision for authentication and authorization.

It really is that simple

later,
Jess

Public servers are really private

[mypalmike (454265)]

While I agree wholeheartedly with your argument, the courts do not. Another example from the same paper is even scarier. In a similar case (Register.com vs. Verio), the court said the use of search robots to gather information may be illegal regardless of whether it broke the site's terms of service:

"Instead, the Court concluded that the mere fact that Register.com had decided to sue Verio meant that Verio's use of the search robot was without authorization: 'because Register.com objects to Verio's use

Depends on website design?

[thesupraman (179040)]

Maybe it is just me, but I would think that ANY information that has been put in a public area (ie: the web) and which can be accessed without agreeing to some form of contract (even if it's just a click-through before you can access it) is public information.

The page can of course still be copyright. That should not stop you using the information it contains.

Online vendors are tring to play the 'have your cake and eat it' game here - if you make information available to the 'public' then it is available to your competitors. If they can hire a preson to read it and inform them, then they can use a program to harvest the same.

Copyright would come in to play if you copied pages in total, however if you are using just a small fraction of the information, and not copying the layout then I think it would be very hard to argue a copyright violation.

Some real-world stores combat this by not having marked prices, which is a pain, but their right - the same should apply to online vendors, they should keep their secrets a secret.

Of course, if your script made a fake 'order' to gain the information it would be different, as it is miss-representing itself and IMHO that is not a good thing (tm).

Then again, it all comes down to whos lawyers are the hungriest that month I guess, which is a very sad thing.

No.

[www.sorehands.com (142825)]

Just because an area is accessible to the public does not mean that you can do whatever you want. If you liken a website to a bookstore, ou are permitted to go in a browse, but not bring in a copy machine and copy books.

There is an implied license to browse a website, but that can be overidden by a terms of use or a robots.txt file.

A scrapper would have a hard time claiming they didn't know, if there was a robots.txt file.

Years ago, there was something similar by the National Baseball League against Moto

well, this is partially related

[trmj (579410)]

Back when the online deals company was given a C&D for posting retailers' black firday ads early, not even three days later the store I worked at was handing out memos to the employees saying that anybody caught releasing "marketing or promotional information" early would be terminated immediately.

Apparantly, they have a problem when other people know their information, but they have no problem making weekly trips to the competitor store down the road to gather information about prices and specials.

It makes me wonder if, since people are being thrown litigation for data mining, is it legal for people to walk down to the competition and get the information in person?

So is the Internet like a Public Square???

[ProfMoriarty (518631)]

If I place a flyer advertising prices in a public square ... and my competition sees it and LOWERS the prices (creating competition) ... would I have the right to sue that company?

NO

Ok then ... if I place my prices on the internet, along with my competition ... could I sue then?

NO

So ... how could I "protect" my business against someone who could see my prices anytime? Well, I could create an inconvience for my customer, by listing all the prices as "call for price" ... or force my customers to become a "member" to my website to gain access to my price list. But this would not guarentee that my competition wouldn't enter.

Basically, if it's on the internet, AND it's for public consumption, then it's fair game. Why couldn't you use a price harvester robot to gather prices? Like in other posts, they're not copyrightable, trademarkable, patentable, nor are they a trade secret.

Hmmm ... price harvester robots ... thanks for the idea, I'll have to patent it.

Easy answer

[Lurgen (563428)]

Once their prices hit the Internet, they're in the public domain. It would be like posting your prices in the window, and complaining that a car driving past could photograph them.

We all know that bots crawl the web - Google, Altavista, spam-bots... they're all common knowledge. You put information on a website, and it's going to be viewed by an automated process. Surely with that knowledge, it's ridiculous to think you can ban people for using the information you've posted publicly in whatever way they desire.

Perhaps these companies (airlines, computer stores, whatever) need to start offering their services at the price they really mean to sell it for, rather than this stupid haggling they expect from us. Or maybe it's time they focused on quality of service, value-add, etc rather than price wars (which never help anybody in the long term).

Bottom line? If you don't want your competitors seeing your prices, don't make them available to them - this means no junkmail, no spam, no website, no prices in the store window, no prices inside the store, nothing.

Examples...

[oaf357 (661305)]

Buy.com was built around a huge server farm that scoured the web and found the best prices for products it sells and then beat those prices (to the best of its ability). That has changed a little now but buy.com was built from that.

Also, Pricewatch [pricewatch.com], Pricegrabber [pricegrabber.com] and Froogle [google.com] scour the web for prices and create search engines out of them so consumers can find the best price.

I'm not saying just because everyone else is doing it means you can too (and you might have a slightly different objective causing these examples to be weightless) but it's being done all over the place.

Hope that helps.

The US Code...

[Pettifogger (651170)]

It seems that the US Code and the First Circuit make this one pretty clear. If you have to agree to some sort of "terms of use" to get onto a website, you are bound by what those "terms of use" say. By clicking through, you have agreed to a contract and you have to abide by it. If your competitor's site requires you to agree to such terms, and those terms prohibit data mining, then you can't data mine there. Simple.

As for the ethical part of telling your employer about this... well, first, remember, this is just a decision of the First Circuit. If you live in a different Circuit, then it may or may not be binding on you. I know this jurisdictional stuff can be a little confusing, but a decision by a Circuit only affects the jurisdictions within it. Only the US Supreme Court (generally, I know there are federal tax, patent, admiralty, etc. courts, too) can make decisions that are binding on the entire country. If you're not sure, check with your corporate counsel. And it might be a good idea to forward the case to him anyway, you might be able to pick up some "bonus points" from your boss for being an especially conscientious employee.

hmm, anybody rfta?

[lingqi (577227)]

on the ground that Explorica had used confidential information obtained from EF to assist in obtaining this pricing information in violation of confidentially agreements executed by former EF employees now working for Explorica.

seems like it's the using confidential information part that got the scrapper capped.

I don't see why accessing *public* information be problematic.

the only thing that may be of trouble is the website EULA, but then the EULA would be saying the same thing as "don't visit my store unless you intend to buy," which would be rediculous in brick-and-mortar world (and should be similarly in cyberspace).

last question, though - why the heck would you ask this kind of stuff HERE? wouldn't a law-forum be a better choice?

Seek real legal advice.

[Gerad (86818)]

I am not a lawyer.

Slashdot is not a lawyer.

Slashdot is not a replacement for a lawyer.

Individual posters on slashdot may be lawyers, but are you really willing to trust your future to what some random person online says, when they could be a lawyer, but could also be some 14 year old kid who thinks it's amusing to screw with people?

Repeat after me:

I will seek proper legal advice.

Seriously, this comes up time and time again. If you're in a situation where you need actual concrete legal advice, SLASHDOT IS NOT THE PLACE TO GO. Sending in an Ask Slashdot is fine for theoretical questions, but when your ass is at stake if a lawsuit comes around, do you really want to trust your future to the legal advice given to you by Anonymous Cowards and karma whores?

you are stating the obvious

[g4dget (579145)]

Every time someone asks a legal question on Slashdot, people like you come out of the woodwork and point out that Slashdot is the same as going to a lawyer. You don't say. Do you really think everybody else is stupid?

Discussing legal issues is not just a business for lawyers. Non-lawyers can give each other useful pointers. And non-lawyers actually have an obligation to determine whether their legislators are doing a good job with the laws they enact and judges they appoint, and a healthy discussion i

Why waste time in the legal system?

[x00101010x (631764)]

Filing lawsuits to protect your price information is just dumb, not to mention waste (if not abuse) of the legal system.

Personal feelings about freedom of information aside, and just from a coder's POV, here's my solution.

If they really want to avoid getting scraped, they should just get their existing, underpaid web developers to create a backend setup that generates the prices as gif's that give OCR hell (such as those used to prevent automated registration of say Yahoo! email accounts).

Coders are cheaper than lawyers (at least those needed to write such code as this).

Sure, the compition could pay more money to get somebody to develop better OCR to read each and every dynamically generated GIF, but most people require proof reading of OCR data, which leads to even more cost.

Something I learned from my Uncle who works with the DOD is this: Any lock can be picked; Any encryption can be broken. It's just a matter of if it's worth the time and money to get what's inside.

In short, with a little one time cost, the company that doesn't want it's prices scraped can just make it so hard to scrape their prices that it's not worth it. The price of scraping the graphically displayed price tags would also be an ongoing cost of software and proofreaders that would dip into profit margins, which management at the company that desires the scraping won't like.

It's not perfect, but it's better (and more bankable) than going whining to the legal system. (Especially since coders are generally cheaper than lawyers).

Same as in the Physical World

[serutan (259622)]

Anybody who publishes information about their business runs the risk that a competitor will get hold of the information and use it in some way. This has always been a fact of life in the physical world. As computers came online in recent decades many companies have maintained databases of information about competitors' products. The Internet doesn't change any of this.

Read the case...

[anubis (87418)]

Read the case...EF Cultural Travel BV v. Explorica hinges on the fact that the defendant company hired an ex-programer from the plaintif company. The programmer had special knowledge of codes used in the pricing (which he had signed a confidentiality agreement not to disclose). When he made the scrapper program he violated the confidentiality agreement.

It was the violation of the confidentiality agreement that the court held was illegal.

As for whether you should tell your employer, it depends on your employment agreement! :) Depending on how the contract is written you could be jointly liable.

While this is a 1st Circuit case, it has been followed by the 5th Circuit (Ingenix, Inc. v. Lagalante) and cited in cases in the 7th and 9th Circuit.

Hope this helps.

--me

Screen scraping is not data mining

[Call Me Black Cloud (616282)]

Screen scraping is data gathering. Data mining is looking for trends or patterns in data you already have. Getting the nuggets out of your data to continue with the mining analogy. From this [thearling.com] presentation titled "An Introduction to Data Mining Technology" data mining is defined as "The automated extraction of hidden predictive information from (large) databases".

The bottom line is this: when you put this work experience down on your resume don't say you were data mining. Companies looking for that experience will ask you hard questions you don't know the answers to and you will be embarrassed.

here is a similar case

[mcguyver (589810)]

Here is a related incident:

http://news.com.com/2110-1017-944258.html [com.com]

Bargain Network spidered real estate prices on homestore.com/realtor.com and posted them on the bargain.com website. Homestore sued and the case was settled out of court. I wish it was not settled out of court because that would set up a precident.

In my opinion you are asking for the problems. Taking a case like this to court and winning would be difficult. At the very least it would be a serious legal expense.

The last time I checked the rules for Froogle you had to be the actual merchant that ships the product in order to show up in their index. If you are spidering a merchant then you are an affiliate, the products do not originate from you so you would be exluced from Froogle. Froogle does not allow you to sort products by price - so obviously what you plan on doing is different. Froogle also gives merchants the option to be excluded from their index.

My advice is this - get a lawyer because one will surely be contacting you. Familiarize yourself with these phrases: false advertising, breach of contract, and unfair competition.

Re:let me ask you this.

[slamb (119285)]

An Anonymous Coward asked: Is the comma near the end of this question, necessary?

Of course it is. Let's dissect this sentence:

Is Data Mining for Product Pricing, Illegal?

• Subject: Data (from Star Trek, I suppose)
• Verb: Mining
• Object: Product pricing
• Listener: Illegal

Not having the comma would completely distort the meaning of the sentence.

Ah, just the comment I was looking for.

[ahfoo (223186)]

I think what you have to look at is the media context in which the prices are displayed.
It's quite true that many stores will try to prevent you from making recordings of any kinds on their physical premises. I've been reprimanded by store managers many times for taking photos in the store. But their right to prevent me from creating media on their premises is based on their property rights, not any some legally backed authority to censror the media.
The web is a totally different story. I use web scrapers all the time and a site that doesn't like it can kindly take its ass off the web. Once you place material on the web, it is published. If you don't want to publish your prices, you don't have to. That's like publishing a book and complaining the readers read it too fast.
The people who compain about such things are the idiots who create unworkable business plans based on their own assumptions about how people are going to use the resource. This is an interesting issue with news media that want to sell access to their archives. There's no way they can both publish to the web and prevent me from caching old copies. If that's the business plan then web publishing is an inappropriate business decision and guess who should pay for bad business decisions: the consumer, or the fool who pursued an ignorant business plan?

er, BS:Re:Best buy is a really really bad example.

[randyest (589159)]

Sorry, I call bullshit.

Over the past 6 or 7 years I've used a palm (handspring visor, to be more precise) hundreds of times, in every Best Buy (and Circuit City, MicroCenter, etc.) in the Boston area to record prices. I've never had anyone even look at me funny.

Maybe it's related to how guilty (or difficult to remove) you look, but I really doubt that happened to anyone ever (note the once-removed story -- it's always a 'friend of mine' in these types of stories.)

In any case, what kind of wuss would leave without making a fuss and forcing them to call the police over something so ridiculous? I could be using my palm to look up my friend's number to call and ask which video card to buy. Fsck them if they don't like it.

Or, maybe this particular Best Buy was located in an airplane and the event happened during takeoff or landing. Or your friend lied to you. One or the other.

Re:Easy fix.

[canajin56 (660655)]

Scott Adams refers to this as a "confusopoly" Telephone companies and airlines use them. Since you can never tell who has the best price, they can all remain in business.

Re:I'm reminded

[shumacher (199043)]

Both Best Buy and the judge were off their rocker. Best Buy can ask him to leave. The store is private property. If he is a customer, they should have allowed him to continue to calculate prices. If Best Buy asked him to put his laptop away, he should have complied. Further, the article mentions other stores that claim to not have a policy against Mr. Kahlow's behavior. I know for a fact that one of them actually is okay with Mr Kahlow's behavior, but does have a policy (and not just an "unwritten" one)against competitor's doing what Mr. Kahlow was doing. (Which is a different thing - and more on topic - one means a potential sale for them, the other means potential sales for their competition) Their policy is to ask the suspected competitor what they are doing, and then if confirmed as a competitor, they are to make a specific statement essentially revoking that individual's right to visit the store, and then, they are to ask them to leave. When that store shops a competitor, policy compels them to wear normal street clothes, to not use any automated devices, and to admit their affilliation and leave if asked.