Straightlining vs. Answering Your Stupid Question Honestly

OK, this is something I hadn’t thought of before.

When I’m staring at a bad survey question — asking me to compare two absolutely identical companies in a matrix, for instance — my tendency is to do this:

They’re equal. There’s no difference between Visa and MasterCard in my mind. Discover and American Express, those are different, both from one another and from these two brands, but Visa and MasterCard might as well just merge, as far as I’m concerned. Of course, there’s no way to provide that answer in the framework provided here, so I decided to simply give each company a score of “5” for each item. That seemed to get the message across, as far as I was concerned. Of course, as soon as I clicked the button, I got booted, with the same generic non-qualified message you get when you tell them you don’t have kids or haven’t seen a movie in the past two months or whatever it is. We all know the truth: they booted me for straightlining.

Which I wasn’t.

At the very least, wouldn’t it be smarter to keep me in and see what the rest of my answers looked like? With the amount of amply-documented badly designed questionnaires out there, shouldn’t we maybe consider that a respondent will occasionally need to do something to get around a poorly framed question, or an item that simply doesn’t apply to them?

Simply ending the survey as soon as someone gives all items on a page the same value seems both too simplistic and too drastic a solution to me.

Another Fine Matrix

First, look at this full-size. See how there are 14 brands of cat food going across the top? I already told it I’d never heard of five of them, and yet here they all are again. It’s one thing to ask me if I’ve ever heard of a brand and to then, even if I haven’t, show me an ad for the brand and ask if I’ve seen that ad — I very well could have forgotten about it, or misremembered what brand it was for.

This is just stupid…

Worse, though, it’s endless. Here’s a reduced-size capture so you can see how long it is:

This is what I’m referring to in the comments on Gary Langer’ post here — what the hell sort of non-representative person is going to sit through this? This kind of garbage really is the cancer that is killing market research. Stop pulling this crap, and then go and worry some more about probability samples.

Is the panel research business model creating a gold farming problem?

Greenfield must be having trouble getting panelists to complete research these days. Maybe it’s the summer blahs, with respondents too hot, too sweaty, or just too on vacation to be bothered.

Then again, maybe it’s something to do with people just getting sick of trying to imagine their orange juice has come to life and is displaying personality traits.

In any case, I’m sure this is the answer:

50 cents per survey! At 20 minutes per survey, that’s like, $1.50 an hour! This will totally solve all of Greenfield’s problems, and can only lead to amazing data quality.

Right?

But let’s turn this repetitive Greenfield mockery into a real question:  what are the odds that this sort of incentive (and incentives in general, really) has already led to or soon will lead to the market research version of “gold farming?”

Gold farming, if you don’t want to bother reading the Wikipedia entry, is an exploit carried out within massive online role-playing games, like World of Warcraft. I’m no expert in it, but as I understand it, people hire low-wage workers (this has apparently been an issue in China) to sit in front of multiple computer terminals logged into the online game. The workers don’t actually play the game as it’s intended to be used, but they instead perform repetitive actions, generally using automated scripts, to earn (or, colloquially, to farm for) in-game cash — virtual money, essentially, that can be spent on in-game items like better weapons and the like. The folks behind the operation then sell the virtual currency online, to actual players of the game who want to buy a really cool sword or whatever but who can’t be bothered spending weeks building up the in-game cash to buy it.

So, since Greenfield is paying 50 cents for 20 minutes worth of human labor here, it occurs to me that someone has probably already figured out that they write some scripts to blast through these things in (let’s say) five minutes each — 12 per hour, as opposed to 3 per hour. And that’s per computer. So you sit a guy in front of five screens, each logged in on a different Greenfield account, each earning $6 an hour — so $30 an hour across those five screens — you know, if your labor only needs to make around $3 an hour, that’s $216 a day in pure profit for the guy in charge. And that’s assuming he’s only got one guy doing this on only five accounts at once.

Now, I’m sure I can’t be the first person this sort of thing has occurred to, and I’m sure Greenfield and the other panel outfits are trying their hardest to make this impossible, limiting the number of surveys one respondent can complete in a day, maybe checking for a total elapsed time and invalidating surveys that move too quickly — but, I don’t know, that strikes me as sort of being similar to making the roads near the bars really wide and straight instead of outlawing 24-hour happy hours, or some similarly goofy comparison.

If we want honest answers from real people, maybe we should rethink this entire insulting “we’ll pay you fifty cents to answer 120 repetitive questions about the minute differences between four brands of orange juice” business model.

Greenfield: They’re Just Weird. (And why are they plagiarizing recipes from the BBC?)

A typical survey invite email from Greenfield Online:

I don’t get it. The recipe, the quotation from Aristotle, the mountain climber — what does any of this this have to do with orange juice? (I haven’t clicked to see what the survey is about yet, but I’m sure it’ll end up being about orange juice. Again. BTW, I just bought a 4-pack of Tropicana at Costco, because it’s what they have, not because I thought it was the most likely to be “outgoing” if it happened to “come to life as a person with distinct personality traits and characteristics.” And just scroll down a bit if you don’t know what I’m talking about.)

So anyway, that’s weird enough, and it was going to be a post on its own, but I figured I’d take a quick look through my in box and see what else Greenfield was using so I could mock them a little more. The very next invite I had included a recipe for monkfish pasta, and here’s where this starts to get pretty funny:

Greenfield is based in the US state of Connecticut. We don’t much use the metric system in America, and yet here’s a recipe that not only includes metric portions at all, but lists them first … and also refers to a type of fish (huss?) I’d never before heard of, as well as to  “double cream,” which despite sounding delicious, does not exist here. At all.

That all seemed odd enough that I copied and pasted the first line of ingredients into Google, and guess what the first hit was?

This:

I’m no lawyer, but it’s probably legal for Greenfield to include this recipe, especially as it seems to be coming from the “Sea Fish Authority,” which seems to be an official UK agency. It’s just deeply weird, no? Why would you do this? What is it adding to the respondent experience? Was there research done on what consumers want in a survey invite, and did the results actually end up ranking “Strange recipes from other countries” slightly higher than “trivial facts about adorable animals?”

(from another invite:)

(I can’t tell what the original source for this one is, though it’s surprisingly not Wikipedia, although that sentence about the British Isles does appear in the Red Fox entry there.)

Oh, Greenfield. Don’t ever change.

Is it me?

Is it just me, or do orange juice brand managers seem more likely than just about anyone else to think their brands have the ability to come to life with distinct human personality traits and characteristics? Is it just one crazy person who keeps using Greenfield for this? Is it the whole industry? Someone help me understand this.

This is at least the third time I’ve seen one of these:

(Also, “warm?” You really want me to think about whether or not any of these refrigerated products could best be described as being “warm?” Because, ew.)

ETA:

OH, COME ON. This is just ridiculous. I’m almost too fatigued looking at this to copy and paste it here, let alone fill it out. In fact, I think the only reason I’m continuing at all is so I can see what fresh horror awaits. How do they expect real people to answer things like this? Oh, right — because they promise us a sweepstakes entry in exchange for 35 minutes of our time. (No, really, they do:)

Anyway, here’s what I’m yelling about now:

Sweet Fancy Matrix

This one, from the nameless partner of a nameless partner of Greenfield the Matrix King, may just take the cake.

Maybe it’s just that it’s late at night and my glasses are dirty and my eyes are tired, but this is incredibly off-putting to me:

I especially enjoy the “100% Fruit Juice In Any Flavor” column, and the way each word is on its own row.

I’m going to try to persevere and get through this one, mainly because I think it’s sure to get worse as I go, but if it weren’t for you, my loyal readers, I swear I’d just to to bed.

In all seriousness, though: who decided that this was the way to determine what consumers truly think?

Greenfield Solves the Wrong Problem

So, I see Greenfield Online has gone and added a “security” question to their matrices to weed out people who pay no attention whatsoever. This, of course, is a predictable response to the somewhat misplaced industry obsession with the quality of online survey responses.

Oddly, though, they’ve gone and inserted this into a survey where I’m personally finding the questions — even in this matrix format — to be pretty interesting. For once, I’m not feeling like this is a tedious waste of my time:

Eats, Shoots & Writes Online Polls for Greenfield

Three things I was going to do, but didn’t:

1) Leave out the Greenfield logo. I’m acutely aware that I pick on them quite a bit, so I wasn’t going to say who was behind this one. Then I realized it was probably obvious.

2) Write an entire rant about the complete lack of proofreading, quality assurance, and consideration for user experience that has to exist in a corporate culture for something like this to ever be exposed to the general public. Consider the previous sentence to be the rant; I don’t think I need to expand upon that.

3) Fired off an email to Greenfield offering to solve these constantly recurring problems for them for a very affordable ongoing consulting fee. I might still do that, come to think.

Enjoy the rest of your Sunday, folks.

Why We Fight

I think I need the occasional reminder that the point of this isn’t so much to point and laugh at Greenfield and other worthy targets, but instead that bad research needs to be eliminated, because it actually hurts us all.

What’s the long-term effect of Greenfield getting respondents to take survey after survey, in a never-ending chain of sweepstakes entries?

What’s the long-term effect of Polling Point asking respondents to mark screen after screen of company names as green or red?

What’s the long-term effect of untrained telemarketers asking respondents who they’ll vote for — and mispronouncing the name of each candidate?

What’s the long-term effect of Zogby’s willingness to get in bed with and take research on just about any topic for just about any client?

As the pool of willing cooperators dwindles, how much longer can we rely on the remaining respondents to fill in the gaps?

Short-term, there are ways to compensate, but at some point, we’re going to regret that we let this go on as long as we did.

Define “use.”

OK, 168 is probably an exaggeration; I do put my laptop to sleep when I go to bed. On the other hand, I have machines in the house that are always running, and generally serving something to somebody somewhere, so in that respect, I am using the internet.

Harris asks something similar, which I thought I’d complained about previously but now can’t locate, so maybe I dreamt about it. Look, I “use the internet” all day long at work, and am logged onto IM from the moment I get home until the moment I go to bed, even if I’m doing something else at the time like watching television. Lately, I’ve been logging off IM and email and getting in bed with my laptop to get sleepy by reading wikipedia articles on interesting topics — I’ve always liked non-fiction. Except for the hours I’m actually asleep, I really AM on the internet all the time — so maybe you could be asking a more relevant measure of my internet usage than the specific number of hours I’m using it.

Incidentally, I never figured out what the top allowable number was; I subtracted 6 hours a night for sleep from 168 and tried 126, which it also rejected, and then tried 100 (assuming it would also fail and I’d end up using 99), but it did in fact accept it.