Slashdot stories can be listened to in audio form via an RSS feed, as read by our own robotic overlord.

 



Forgot your password?
typodupeerror
Stats Your Rights Online

OKCupid Experiments on Users Too 161

Posted by Unknown Lamer
from the statistics-are-only-skin-deep dept.
With recent news that Facebook altered users' feeds as part of a psychology experiment, OKCupid has jumped in and noted that they too have altered their algorithms and experimented with their users (some unintentional) and "if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work." Findings include that removing pictures from profiles resulted in deeper conversations, but as soon as the pictures returned appearance took over; personality ratings are highly correlated with appearance ratings (profiles with attractive pictures and no other information still scored as having a great personality); and that suggesting a bad match is a good match causes people to converse nearly as much as ideal matches would.
This discussion has been archived. No new comments can be posted.

OKCupid Experiments on Users Too

Comments Filter:
  • Flash panic (Score:5, Insightful)

    by LordLucless (582312) on Monday July 28, 2014 @06:18PM (#47553877)

    World discovers A/B testing
    Freaks out
    Until the next reality tv show comes on

    • Re:Flash panic (Score:5, Insightful)

      by stranger_to_himself (1132241) on Monday July 28, 2014 @07:09PM (#47554135) Journal

      World discovers A/B testing
      Freaks out
      Until the next reality tv show comes on

      When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc. I think its skipping that part that people are uncomfortable about. Of course that happens every day in the business world (and even did before computer scientists rediscovered basic experiments and called it A/B testing), but in some of these cases it does start to look like an academic psychology experiment. Perhaps use of OK Cupid implies consent to be experimented on but I doubt that consent is collected in a transparent way.

      • by perpenso (1613749) on Monday July 28, 2014 @07:39PM (#47554339)
        That probably depends upon whether you consider the terms of use of the online service, grocery store loyalty card, casino player's card, etc to be transparent. Those terms of use that no one reads.

        There is also consent by action. The casino does A/B testing by offering some a $40 steak dinner plus $40 in chips while it offers others $80 in chips. You clicked on the advertisement/offer, or you opened the envelope that arrived in your postal mail, etc.

        Similarly the coupons a grocery store offers you are often part of an experiment. Hell, changing the items on the isle end caps are sometimes part of an experiment.

        My marketing processor thought that grocery store loyalty cards were the greatest invention ever in the history of marketing. The data collected and opportunity for experiments enormous.
        • by Xest (935314)

          "There is also consent by action. The casino does A/B testing by offering some a $40 steak dinner plus $40 in chips while it offers others $80 in chips. You clicked on the advertisement/offer, or you opened the envelope that arrived in your postal mail, etc."

          Well I think this is the difference, when you sign up to OKCupid you're signing up to a service that's explicitly designed to optimise your chance of meeting someone, so almost by definition you're going to expect them to play around with your profile,

        • by tomhath (637240)

          That probably depends upon whether you consider the terms of use

          I use depends, you insensitive clod.

          • by perpenso (1613749)

            That probably depends upon whether you consider the terms of use

            I use depends, you insensitive clod.

            We know, we have your supermarket data. :-)

      • When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc. I think its skipping that part that people are uncomfortable about.

        The operative word being "usually", which implies there exist cases where you don't. The discomfort come from people not grasping the existence of the "usually", and that businesses are not academics and product testing is not held to the same standard.

        • by billstewart (78916) on Tuesday July 29, 2014 @01:32AM (#47555479) Journal

          Facebook's experiments bother me more than OKCupid's. They're deliberately manipulating which news stories their readers see in order to affect their mood, and seeing how that affects the readers' behavior. That seems mean and dishonest. (Of course, I didn't know Facebook had news, so I'm not in their target market anyway, but it still seems mean.)

          OKCupid's a dating site, which means that all their "compatibility" scores are pretty much guesswork anyway, assisted by a lot of measurement, so an occasional suggestion of "maybe you two should see if you want to date" to people they normally wouldn't match up isn't that much perturbation of their approach anyway, and "whoops, pictures are broken, why don't you try talking first instead of just looking at pictures" is just fine, and both of them give them a bit of data outside the ranges they'd normally be collecting from - perhaps there are people that would get along well who they haven't been matching up. (I'm not in their market either, fortunately.)

          • Facebook's experiments bother me more than OKCupid's. They're deliberately manipulating which news stories their readers see in order to affect their mood, and seeing how that affects the readers' behavior. That seems mean and dishonest. (Of course, I didn't know Facebook had news, so I'm not in their target market anyway, but it still seems mean.)

            I agree that the Facebook incident is much more worrying than what OKCupid is doing.

            OKCupid's a dating site, which means that all their "compatibility" scores are pretty much guesswork anyway, assisted by a lot of measurement, so an occasional suggestion of "maybe you two should see if you want to date" to people they normally wouldn't match up isn't that much perturbation of their approach anyway, and "whoops, pictures are broken, why don't you try talking first instead of just looking at pictures" is just fine, and both of them give them a bit of data outside the ranges they'd normally be collecting from - perhaps there are people that would get along well who they haven't been matching up. (I'm not in their market either, fortunately.)

            Your post raises another interesting (and IMO ridiculous) issue though, which is that just because providing either of two different services (either pictures or no pictures) to all of your clients is perfectly fine - randomizing people to receiving one or the other is often considered not to be ethically sound.

      • by bondsbw (888959)

        When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc.

        Academic experiments have external results, publishing findings as scientific research. Business experiments have internal results, data mining with the goal of increasing profits (via providing better value to the consumer, at least in capitalist theory).

        Well, at least, I can hope the results stay internal to the business. As with data mining in general, that's not always the case. But perhaps this becoming a mainstream topic will end with a framework on which to judge companies that release "experiment

        • by pipedwho (1174327)

          The problem with most 'commercial experimentation' is that it isn't about getting better value for the consumer, but about how to to best convince the consumer to pay more for something, or buy something, that they otherwise would not have.

          Loyalty cards are a way for a business to encourage a customer to return whether or not it is really in their best interest. Phone contracts, transaction 'fees' and 'licensing' are other ways to get people coming back for more of a beating. If you make the fine print and

          • Specifically regarding getting you to purchase something you wouldn't have, well, I don't see that as bad. You buy stuff because that stuff is worth more to you than the money is sitting in your wallet. That is what every honest transaction is...an exchange that favors both parties.

            Shady practices like making it difficult to redeem a coupon are a different story, and frankly should be illegal.

            • by pipedwho (1174327)

              I have to agree that buying something useful that you otherwise would not necessarily have bought in the first place is not a bad thing for either yourself or the economy as a whole.

      • by taustin (171655) on Monday July 28, 2014 @08:49PM (#47554597) Homepage Journal

        It's hard to imagine how anyone could find this to be scientific experimentation, rather than some random crap done in hopes of finding a way to sell more advertising.

        • The two are not mutually exclusive. Something doesn't become unscientific just because you happen to think it's trivial.

      • by Solandri (704621)

        When we (academics) do experiments on people however trivial we usually have to go through ethical clearance, get informed consent etc. I think its skipping that part that people are uncomfortable about.

        You do realize that you yourself conduct such "experiments" on your friends every day? While making conversation in the lunch room you ask, "Hey, anyone wanna see Planet of the Apes tonight?" That elicits a lukewarm response, so you then ask "Well what about How to Train your Dragon?" You get a lot of in

        • I think the dividing line between when you need to get informed consent is when the experiment begins to make people do things they wouldn't have done anyway. Tweaking how people get paired up for dates is fine if they were looking for a date anyway. Forcing them to go on a date when they weren't planning to would require informed consent (and probably compensation).

          Not really - even purely observational academic studies need ethical approval and informed consent. I really am confused about where the diving line should be between academic and commercial work.

          • even purely observational academic studies need ethical approval and informed consent.

            In what jurisdiction?

            That's interesting, because a lot of "purely observational academic studies" have been done with no informed consent at all.

            Examples: - Robert Levine's experiments [google.co.uk] linking a city population's average walking speed with their degree of helpfulness and their health (actually, this is not merely observational - parts of those experiments involved getting people to pick up dropped pens, return lost letters, etc). That was done in many parts of the world including the USA.

            - In a car, sitti

            • You are right. I just checked my School's ethical research code, and there is an exemption under the 'exceptional circumstance' that taking informed consent is not possible because of a need for concealment. These studies still require ethical review though and the ethics panel will decide whether or not the concealment is justified. I had not encountered this before in my own work and I should not have generalized.

      • Do academic demographers get "informed consent" before processing census data? What about crime statistics? Network security incidents?

        Showing a page with and without images and then processing access_log is not the same as monitoring someone's eating habits and stress levels for a week. Just because you call something "an experiment" (a) doesn't mean it is one, and (b) doesn't mean it's the same as all other "experiments".

        • Do academic demographers get "informed consent" before processing census data? What about crime statistics? Network security incidents?

          You are talking about aggregated data which is a bit different. Its a current debate as to whether anonymous routine data should be available to researchers at the individual level without the explicit consent of the people involved. I would say it should be but many argue otherwise.

          • I think you mean de-identified data, not necessarily aggregated data. But I understand your point. I am not sure that the outrage of the interwebs turns on that, though, as there are plenty of examples of data collection that cannot be tied to a real-life identity that gets their panties in a wad.

        • by AmiMoJo (196126) *

          According to TFA they correlated certain reactions with attractive photos, which means they must have looked at the photos to determine if they were attractive or not. It isn't clear how far they went into people's private profiles to do this. Maybe it was all innocent, but it seems creepy because they were not transparent about it and probably would have kept quiet if Facebook hadn't been caught.

          I know what people will say, they uploaded their photos to a web site and have no expectation of privacy. Well,

          • > which means they must have looked at the photos to determine if they were attractive or not

            That is quite an assumption. I can think of a ton of ways they could have an attractiveness measure without themselves digging into people's personal profiles. In fact, I did 5 seconds of googling and found this, which clearly suggests that they are asking other members to rate attractiveness of profile pictures: http://blog.okcupid.com/index.... [okcupid.com]

            > I know what people will say, they uploaded their photos to a we

      • by EL_mal0 (777947)
        For me, it's not the testing so much, but in Facebook's case, publishing those results as if the participants had given informed consent. I expect to be subject to usability testing. I don't expect to be the subject of psychological testing. I don't know where the line is, but there's a line somewhere in there that was for sure crossed in the Facebook case. It's less clear to me what side this OKCupid case is on.
        • by Lennie (16154)

          The problem is: usability testing is psychological testing.

          With usability testing you are testing what people would do in different situations. That to me sounds like a psychological test.

          • by EL_mal0 (777947)
            I get that. I just think there's some important difference of degree between, "Let's see if the blue button gets pressed more than the green button," and "Let's see if people become sad if we show them sad posts from their friends." My opinion is that there is a line somewhere there. I just don't know quite where it is.
  • A/B Testing (Score:4, Insightful)

    by Mr Z (6791) on Monday July 28, 2014 @06:18PM (#47553879) Homepage Journal
    Isn't that what A/B testing is all about?
    • People being manipulated, even if the manipulation is only demonstrated by the experiment, without knowing they are being A/B tested, is not what A/B testing is about.

      Knowingly participating in an A/B test is kinda part of A/B testing. Is this lens better, or worse? Which of these televisions side by side looks better to you?

      You are looking for love, your soulmate, or someone who will put up with your desire to have cough medicine inserted into your rectum by someone dressed as a Teletubby.

      You don't find

      • Fuck me, I made the point using the wrong experiment. Otherwise, the argument still stands. People trust that the number is as correct as the website can be. Given that it doesn't know whether you like Teletubbies putting cough syrup in your ass.

        Still, not A/B testing in any but the most ignorant sense of words.

    • A/B testing, as a concept, is fine. The issue here is that A was "truth" and B was "deception", and that's something you shouldn't be A/B testing (at least not without getting ethics waivers signed). Facebook provided feeds that were not representative of what was actually going on and OKCupid flipped bad matches to good matches, both of which compromised their relevant services by misleading users or misrepresenting information. You can't do stuff like that in most (all?) ethical systems, and it may even o

      • >OKCupid flipped bad matches to good matches

        To be fair on this point is there any objective measure on what a good or bad match is? The entire system on OKCupid is made and defined by OKCupid, there is no objectivity. Therefore a good match == bad match == imaginary purple dinosaur. There are plenty of writeups online about just this subject. Human happiness in relationships is not as formulaic as OKC would like you to believe.

        The FB thing is definitely more objectively definable. As in many bits of info

        • You're correct that the matches are not objective, though that really doesn't matter in the end. If I say, "I'll make my best guess," and then knowingly provide you with the choice that is as far away from my actual best guess as possible, there's nothing subjective about the fact that I've intentionally misled you. My guesses may be subjective, but you were expecting my best one, and instead got my worst one. That's a lie.

      • by Lennie (16154)

        What OKCupid is doing: is checking if their algorithm works.

        How else then changing variables do you check if it works ?

        • There are ways to test the algorithm that don't involve flipping the results entirely. Suggesting, as you have, that this is the only means available to them to do so is a bit disingenuous. Moreover, if you are going to intentionally give people results that you believe to be incorrect, contrary to what you have promised them, you have an ethical responsibility to get their permission in advance via some form of opt-in release.

          • by Lennie (16154)

            I'm sure it's somewhere in the terms-of-service. ;-)

            • As am I. :P

              Even so, you get what I mean, and I doubt most people would consider it ethical to have a blanket ToS that covered all use, including experimental uses. Rather, they'd expect to have to do a separate opt-in to a beta or experimental version of the product, I'd wager.

              • by Lennie (16154)

                I honestly don't know.

                They do say: opposites attract. :-)

                Really: we don't know how well this online dating thing really works. Isn't really all that clear cut as people make it out to be. They are just guessing. And they know it this. So them trying out different approaches isn't as different as what they normally do as you think.

  • All they did was discover that everyone on a dating site places physical attraction (based on a photo) above everything else by a wide margin. Reported "compatibility" and profile data are largely irrelevant. Basically, Hot or Not should be as effective for online dating as eHarmony.
    • Re: (Score:2, Funny)

      by Anonymous Coward

      "People are shallow and place too much value on appearances," said the fat, ugly nerd. "They should be like me, and value what's in peoples' souls. For instance, I can tell Jennifer Lawrence and Scarlett Johansson are perfect for me, because they have such beautiful souls."

      • by Falos (2905315)
        Either you're lamenting in monologue or you're trying to shoehorn in some kind of distance to keep yourself safely apart and hopefully polar from a stereotype you're assembling ad hoc. Need to stay distinct to protect that fragile ego.

        Happens during vilifying too.
  • by slaker (53818) on Monday July 28, 2014 @06:37PM (#47553973)

    The people who run OKC were a bunch of statistics nerds. It runs (ran, anyway) on a custom web server that performs a lot of real time analysis. Their blog is chock full of incredibly detailed information about their users. This shouldn't be news to anyone who has even the slightest clue as to how OKCupid actually works.

  • by egr (932620)

    Is that some sort of advertisement?

    I see no other reason to "jump in" with that kind of information.

  • by thisisauniqueid (825395) on Monday July 28, 2014 @06:53PM (#47554063)

    Findings include that ... suggesting a bad match is a good match causes people to converse nearly as much as ideal matches would.

    All this means is that OKC's match algorithms suck: there's only a weak correlation between match scores and real-world compatibility (like with every other dating site).

    • Findings include that ... suggesting a bad match is a good match causes people to converse nearly as much as ideal matches would.

      All this means is that OKC's match algorithms suck: there's only a weak correlation between match scores and real-world compatibility (like with every other dating site).

      No, it means that:
      1) People trust OKCupid's rating system enough to try harder when it suggests a good match
      2) OKCupid has to take into account their stated match rating, not just length of conversation, when trying to use conversation length as data to improve their algorithm.

      • edit: 1) Users of OKCupid trust OKCupid's rating system enough to try harder when it suggests a good match

    • It's called the "tyrrany of dimensions". The more variables you have, the more data points you need exponentially to derive meaningful partitioning analysis from it, regardless of how clever your distance algorithms are.

      And they have hundreds of questions when a dozen would be about all the entire population of Earth could support.

      • It's called the "tyrrany of dimensions". The more variables you have, the more data points you need exponentially to derive meaningful partitioning analysis from it, regardless of how clever your distance algorithms are.

        Indeed, but only if you insist on carrying along in your analysis all the irrelevant and correlated dimensions.

        And they have hundreds of questions when a dozen would be about all the entire population of Earth could support.

        So do surveys, for significantly smaller sample sizes. I wouldn't be surprised if a non-trivial percentage of those questions are intentionally redundant - you know, to check *ahem* consistency, improve accuracy, etc. If, say, you have 100 questions grouped into 10 categories with 10q/cat, you have just dropped the dimensionality significantly while at the same time having more confidence in your d

    • by wvmarle (1070040)

      Well, of course. They're digging too deep in that. In the real world, I believe that there is no such thing as "the one" or "the perfect match". Maybe it feels like it, but that's in part thanks to the "pink glasses" effect of being in love and because both parties tend to adopt to one another, especially when a relationship lasts long (years, decades).

      People probably can form lasting romantic relationships with a large number of other people, after the following basic matches are followed (assuming heteros

  • That was STUPID!

    Conduct your little experiment if you have to, just keep your mouth shut about it.... At least until you have notified ALL your users that such experiments *might* be taking place (Or if you intend to issue refunds from the resulting class action suit.)

    • As opposed to just randomly matching people so as to not have to learn things about people in general nor their users in particular?

    • The particularly stupid part was messing with their match algorithm. If they imply that their algorithm has any value, then their users will feel at least ripped off (since the algorithm doesn't seem to work well), and possibly angry because they were given incorrect information .

      Blocking pictures was visible to users and I don't have any problem with that .

      • >The particularly stupid part was messing with their match algorithm.

        You are making a false assumption because you are dealing with a biological system.

        If you made a screen to sift sand, that screen will reliably sift sand of a certain size because they sand has no choice in the matter and does not evolve.

        On the other hand if you make an antibiotic that kills bacteria X you will quickly find out that in just a few generations almost all of bacteria are resistant to the antibiotic.

        Culture evolves, religio

  • by nikhilhs (1292298) on Monday July 28, 2014 @07:15PM (#47554177)
    FB's experiment was to see if they could alter the mood of their user. OKC tried to see if they could get more conversations going. Intent matters. OKC's is fairly harmless. FB's experiment could have a ripple effect and cause negative consequences.
    • by thegarbz (1787294)

      FB's experiment was to see if they could alter the mood of their user.

      They should have hired the DICE Beta department.

    • Er, why? Facebook presumably wants users to feel happy when they visit their website - exploring how to do that doesn't seem substantially different to OKCupid wanting to make lots of successful relationships.

  • There's a huge difference between A/B testing, designed to optimize your website with the direct intent to improve sales, and performing experiments on how different news feeds affect your users' moods. A/B testing typically comprises changes in button size and color, website layout, font variations, etc; should we lead with the price, or with the benefits, or with something else? On the other hand, what FB and OKC are doing - admitting to, and proudly! - amounts to wholesale experimentation on their user

  • Marketing (Score:5, Insightful)

    by Bo'Bob'O (95398) on Monday July 28, 2014 @07:25PM (#47554253)

    The fact is that the experiment they Facebook conducted was mild to what other corporations do every day under the umbrella of "marketing".

    They use control groups and try every trick they can to manipulate your mood, feelings, impressions of their products. They carefully script interactions to take advantage of your feelings and social norms. Also take the recent example in the past few weeks of the scripts that Verizon's 'account retention' departments use to try and wedge people into keeping their account longer. Those weren't just thrown together, those were made with careful research and years of experiments on customers and focus groups.

    The only difference with what Facebook did and the rest do is that they shared their results with everyone. Was Facebook Unethical manipulating people the way they did? I think so, and I'm only less interested in the service after that scandal, but what they got them in trouble was sharing it with the rest of the world in a way that might have also done some honest good. Now they will learn from their mistakes, keep it to themselves, and use that research purely to manipulate people for higher profit and no one will say a thing.

    • There are three professions where being untruthful is the key to success: Lawyers, salespeople, and marketing. All three are hired to portray their client in the most favorable light possible, and the very best ones lie through their teeth. The worst of these three are the marketers because they have legions of psychologists and scientists trying to figure out the best way to lie to people.
  • by Kris_J (10111) * on Monday July 28, 2014 @08:25PM (#47554527) Journal
    Try this the next time you want to try an online dating site: Create two profiles, a "real" one and a fake perfect match to your real profile and see how long it takes for the site to claim that your fake perfect match has attempted to contact you and for only $4.95 you can sign on to the paid service and reply.
    • by Ksevio (865461)
      Funny you should mention that since this very blog had a length article discussing that topic. It's gone now that match.com bought them out, but you can still find it in archives.
    • by Anonymous Coward

      That's not actually how okcupid works. You aren't charged for talking to your perfect match. You are charged if you go over the limit of 100 messages and don't want to delete any. And, you are charged if you want to remove advertisements. If you use adblocker their ads actually ask you to turn it off for their site because they get most of their revenue that way.

  • I think Firefox should boycott the site.... display a message about it being possibly malicious/dangerous to all users attempting to visit OKCupid, showing a link to the article as a warning message in bright red... (Just kidding <EG>).

  • "That’s how websites work." Whoa so OKCupid was retarded enough to hire someone who have NEVER, EVER been on the internet? Some heads need to roll ...
  • very few successful businesses are doing what they were originally founded to do. business is all about experimentation. you tweak and reset and change and reset again until you see the numbers going in the right direction at the desired speed. unsuccessful businesses usually do the same thing, too; they just don't ever find a combination that works.

    see also: "Getting to Plan B: Breaking Through to a Better Business Model" http://www.amazon.com/Getting-... [amazon.com]

    "To succeed, you must change the plan in real time a

  • OkCupid will start pairing everyone with the worst possible match. Once and for all, we'll be able to prove that opposites attract!
  • I have a completely different opinion of online dating services at all. When you are trying them in the trial period, which of trial is a waste of time, as just it lets you browse the public profiles and receive messages, you are most certain to receive one or two messages, often in english, no matter what your mother tongue, of someone VERY INTERESTED in meeting you, just to make sure you sign up for the service. Do those people think we are dumb?
  • The Internet is not powered by experiments on humans. Not even in the DARPA days.

    No, websites do NOT experiment on users. Users may experiment on websites, if there's customization, but the rules for good design have not changed either in the past 30 years or the past 3,000. And, to judge from how humans organized carvings and paintings, not the past 30,000 either.

    To say that websites experiment on people is tripe. Mouldy tripe. Websites may offer experimental views, surveys on what works, log analysis, etc

Loose bits sink chips.

Working...