Is the panel research business model creating a gold farming problem?

Greenfield must be having trouble getting panelists to complete research these days. Maybe it’s the summer blahs, with respondents too hot, too sweaty, or just too on vacation to be bothered.

Then again, maybe it’s something to do with people just getting sick of trying to imagine their orange juice has come to life and is displaying personality traits.

In any case, I’m sure this is the answer:

50 cents per survey! At 20 minutes per survey, that’s like, $1.50 an hour! This will totally solve all of Greenfield’s problems, and can only lead to amazing data quality.

Right?

But let’s turn this repetitive Greenfield mockery into a real question:  what are the odds that this sort of incentive (and incentives in general, really) has already led to or soon will lead to the market research version of “gold farming?”

Gold farming, if you don’t want to bother reading the Wikipedia entry, is an exploit carried out within massive online role-playing games, like World of Warcraft. I’m no expert in it, but as I understand it, people hire low-wage workers (this has apparently been an issue in China) to sit in front of multiple computer terminals logged into the online game. The workers don’t actually play the game as it’s intended to be used, but they instead perform repetitive actions, generally using automated scripts, to earn (or, colloquially, to farm for) in-game cash — virtual money, essentially, that can be spent on in-game items like better weapons and the like. The folks behind the operation then sell the virtual currency online, to actual players of the game who want to buy a really cool sword or whatever but who can’t be bothered spending weeks building up the in-game cash to buy it.

So, since Greenfield is paying 50 cents for 20 minutes worth of human labor here, it occurs to me that someone has probably already figured out that they write some scripts to blast through these things in (let’s say) five minutes each — 12 per hour, as opposed to 3 per hour. And that’s per computer. So you sit a guy in front of five screens, each logged in on a different Greenfield account, each earning $6 an hour — so $30 an hour across those five screens — you know, if your labor only needs to make around $3 an hour, that’s $216 a day in pure profit for the guy in charge. And that’s assuming he’s only got one guy doing this on only five accounts at once.

Now, I’m sure I can’t be the first person this sort of thing has occurred to, and I’m sure Greenfield and the other panel outfits are trying their hardest to make this impossible, limiting the number of surveys one respondent can complete in a day, maybe checking for a total elapsed time and invalidating surveys that move too quickly — but, I don’t know, that strikes me as sort of being similar to making the roads near the bars really wide and straight instead of outlawing 24-hour happy hours, or some similarly goofy comparison.

If we want honest answers from real people, maybe we should rethink this entire insulting “we’ll pay you fifty cents to answer 120 repetitive questions about the minute differences between four brands of orange juice” business model.

Health Survey!

You know what works better than an incentive? For me, anyway?

Exclamation points! The enthusiasm is contagious! I can’t wait to click the link and take the survey.

Of course, I’ll bet the staid researchers who programmed the survey itself aren’t so excited about it, though.

Oh my gosh! They totally are! This is going to be the best survey ever!

…

Or not. But it was refreshingly matrix-free and fairly speedy. And oddly, I know I’m sort of making fun of it, but — the exclamation point actually worked, unintentional though I suspect it must be.

Is it me?

Is it just me, or do orange juice brand managers seem more likely than just about anyone else to think their brands have the ability to come to life with distinct human personality traits and characteristics? Is it just one crazy person who keeps using Greenfield for this? Is it the whole industry? Someone help me understand this.

This is at least the third time I’ve seen one of these:

(Also, “warm?” You really want me to think about whether or not any of these refrigerated products could best be described as being “warm?” Because, ew.)

ETA:

OH, COME ON. This is just ridiculous. I’m almost too fatigued looking at this to copy and paste it here, let alone fill it out. In fact, I think the only reason I’m continuing at all is so I can see what fresh horror awaits. How do they expect real people to answer things like this? Oh, right — because they promise us a sweepstakes entry in exchange for 35 minutes of our time. (No, really, they do:)

Anyway, here’s what I’m yelling about now:

Lolcentives

I’m not a fan of incentives in general, though I think they may sometimes be necessary.

The other day, I did some account management at Sprint’s website, which required a (pretty unsatisfying) live chat with a representative. Afterwards, the system asked me to take a survey about my experience, which I of course did. I didn’t notice them telling me there’d be an incentive — turns out they wanted to give you a free ring tone if you took the survey — most likely I just sped past that bit to get started.

I did, however, notice the details of what exactly I’d need to do in order to actually get my ring tone:

What could be easier?

From my standpoint, this is just another tiny nail in the coffin of online research. When you can’t get the incentive you were promised without jumping through seventeen hoops, how likely are you to believe the next researcher who claims you’ll be compensated for your time?

(Adding insult to injury: you’re not going to see this survey unless you interact with a Sprint representative in a chat. You’re not going to need to interact with a Sprint representative unless you’re already having a problem. Sprint already knows how bad its customer service is, so they can be reasonably sure the only people who take this survey were angry when they first got to the website and that a healthy percentage of them got even angrier after they had their chat. So now you’re going to ask them questions and then “reward” them with a nonredeemable incentive? Classy.)

R-E-S-P-E-C-T.

I feel like Seth Godin here, talking about permission-based marketing. (Which is worth learning about, if you’re unfamiliar.)

Everything we do, as researchers, is dependent on the goodwill granted us by our respondents. If we abuse that goodwill — if we contact them too often, if we keep them on the phone/online too long, if we ask them too many questions that make them groan in frustration — well, that all costs us goodwill.

Whether our research is product-based or political, we cannot depend on the results telling us how to proceed if our respondents aren’t being truthful with us. If we disrespect them, they’ll disrespect us. If we don’t care enough to make the process of taking the survey pleasurable, why should we expect them to be patient and honest when we tie them up for half an hour asking redundant questions?

If you view your research as something the respondent will endure in order to get their guaranteed $2 or their SweepLand points or whatever the incentive is, you will not only produce sub-par results, but you will continue to contribute to the overall problem of over-used, disengaged respondents.

Shorter surveys. Smarter questions. Respect.

Get to it.

Rethinking Incentives

Here’s something to think about:

How much of an impact does this…

…have on which button I click here:

If I really want that $5, I’m not going to take the chance and click either of those endpoint answer choices — I’m going to play it safe and put myself right in the middle of the range.

I wonder if Greenfield has ever run a side by side test where they offer $5 to a third of the respondents, $10 to another third, and nothing but a sweepstakes entry to the remaining third. I’m willing to bet that if they have, the percentage of respondents working for large companies increases hand-in-hand with the compensation figure.