Just Say No Already.

Annie Pettit this morning tweeted from the Net Gain 4.0 Conference in Toronto:

Clients still want 1 hour surveys and we can’t do anything about it : I say turn it down!!

I’ll go further than that: I say turn it down and make it clear to the client that they are the cancer that is killing market research. What in the world can you learn from a sixty minute survey that you can’t learn from a 5-minute one? (I’m not talking about an in-depth qualitative research project, or something where you hook someone up to an EEG and have them watch an episode of CSI: Miami to see what their brain has to say. I’m talking about asking questions, on the phone or on a screen. 60 minutes is 55 minutes too long!

Do we really think the respondents still on the phone (or on the web) at the one-minute mark, the ten-minute mark, and the 60-minute mark are identical?

Another Fine Matrix

First, look at this full-size. See how there are 14 brands of cat food going across the top? I already told it I’d never heard of five of them, and yet here they all are again. It’s one thing to ask me if I’ve ever heard of a brand and to then, even if I haven’t, show me an ad for the brand and ask if I’ve seen that ad — I very well could have forgotten about it, or misremembered what brand it was for.

This is just stupid…

Worse, though, it’s endless. Here’s a reduced-size capture so you can see how long it is:

This is what I’m referring to in the comments on Gary Langer’ post here — what the hell sort of non-representative person is going to sit through this? This kind of garbage really is the cancer that is killing market research. Stop pulling this crap, and then go and worry some more about probability samples.

Needs Moar Choices.

Seriously? Shouldn’t they also have broken out high school by year, or something? Maybe included a radio button for each individual year from kindergarten through law school? No, really, I just can’t imagine how such fine distinctions are useful to anyone. Is someone really looking at this and saying, “Wow, the 7 respondents with some advanced degree work are slightly more likely to say x than the 11 respondents who are currently in advanced degree work! Fascinating! Oh, wait, the margin of error is +/- 37.8%.”

I get that there’s value in collecting more, not less data; I’m a firm believer in asking respondents for their actual ages, actually, instead of for a range — because when you have the actual data, you can put it back together in whatever groupings you want, which may not be the groupings you think make sense before you see the results — but this here is just a mess.

Is the panel research business model creating a gold farming problem?

Greenfield must be having trouble getting panelists to complete research these days. Maybe it’s the summer blahs, with respondents too hot, too sweaty, or just too on vacation to be bothered.

Then again, maybe it’s something to do with people just getting sick of trying to imagine their orange juice has come to life and is displaying personality traits.

In any case, I’m sure this is the answer:

50 cents per survey! At 20 minutes per survey, that’s like, $1.50 an hour! This will totally solve all of Greenfield’s problems, and can only lead to amazing data quality.

Right?

But let’s turn this repetitive Greenfield mockery into a real question:  what are the odds that this sort of incentive (and incentives in general, really) has already led to or soon will lead to the market research version of “gold farming?”

Gold farming, if you don’t want to bother reading the Wikipedia entry, is an exploit carried out within massive online role-playing games, like World of Warcraft. I’m no expert in it, but as I understand it, people hire low-wage workers (this has apparently been an issue in China) to sit in front of multiple computer terminals logged into the online game. The workers don’t actually play the game as it’s intended to be used, but they instead perform repetitive actions, generally using automated scripts, to earn (or, colloquially, to farm for) in-game cash — virtual money, essentially, that can be spent on in-game items like better weapons and the like. The folks behind the operation then sell the virtual currency online, to actual players of the game who want to buy a really cool sword or whatever but who can’t be bothered spending weeks building up the in-game cash to buy it.

So, since Greenfield is paying 50 cents for 20 minutes worth of human labor here, it occurs to me that someone has probably already figured out that they write some scripts to blast through these things in (let’s say) five minutes each — 12 per hour, as opposed to 3 per hour. And that’s per computer. So you sit a guy in front of five screens, each logged in on a different Greenfield account, each earning $6 an hour — so $30 an hour across those five screens — you know, if your labor only needs to make around $3 an hour, that’s $216 a day in pure profit for the guy in charge. And that’s assuming he’s only got one guy doing this on only five accounts at once.

Now, I’m sure I can’t be the first person this sort of thing has occurred to, and I’m sure Greenfield and the other panel outfits are trying their hardest to make this impossible, limiting the number of surveys one respondent can complete in a day, maybe checking for a total elapsed time and invalidating surveys that move too quickly — but, I don’t know, that strikes me as sort of being similar to making the roads near the bars really wide and straight instead of outlawing 24-hour happy hours, or some similarly goofy comparison.

If we want honest answers from real people, maybe we should rethink this entire insulting “we’ll pay you fifty cents to answer 120 repetitive questions about the minute differences between four brands of orange juice” business model.

SSI: Matrixes Make Us Cry, Too.

“We know respondents don’t like grids,” Jackie Lorch, SSI VP, Global Knowledge Management, points out. “They’ve been telling us that for years in focus groups and feedback, but we’ve always thought of grids as a necessary evil in questionnaire design. Now, we’re beginning to learn that not only are grids frustrating for respondents – they actually produce inferior data.”

THANK YOU.

Here’s a link to the full report.

L0t there to digest. What do you all think?

Lolcentives

I’m not a fan of incentives in general, though I think they may sometimes be necessary.

The other day, I did some account management at Sprint’s website, which required a (pretty unsatisfying) live chat with a representative. Afterwards, the system asked me to take a survey about my experience, which I of course did. I didn’t notice them telling me there’d be an incentive — turns out they wanted to give you a free ring tone if you took the survey — most likely I just sped past that bit to get started.

I did, however, notice the details of what exactly I’d need to do in order to actually get my ring tone:

What could be easier?

From my standpoint, this is just another tiny nail in the coffin of online research. When you can’t get the incentive you were promised without jumping through seventeen hoops, how likely are you to believe the next researcher who claims you’ll be compensated for your time?

(Adding insult to injury: you’re not going to see this survey unless you interact with a Sprint representative in a chat. You’re not going to need to interact with a Sprint representative unless you’re already having a problem. Sprint already knows how bad its customer service is, so they can be reasonably sure the only people who take this survey were angry when they first got to the website and that a healthy percentage of them got even angrier after they had their chat. So now you’re going to ask them questions and then “reward” them with a nonredeemable incentive? Classy.)

PROTIP: If I Have to Google to Understand Your Question, It’s a Bad Question

Ignoring completely (well, almost completely) the very strange Yes/No selection mechanism on this question, let’s focus on the gibberish instead:

Seriously. I may have, at some distant point in the past, told this particular research outfit that my company uses computers, but I don’t remember being more specific. No idea whatsoever why someone thought defining this term (or, better, avoiding using it at all and actually saying what they meant) was a good idea. Is it the old “if you don’t know what it means, you don’t matter” standpoint? Because while I may not matter for this particular project, presumably I do matter in general because they keep emailing me links to their research projects…

Wow. Just wow.

You’ll have to excuse the low quality of the image here; I snagged it off a WebEx meeting where the presenter flashed it by as part of a PowerPoint he was sharing. I don’t know the source, I don’t know the rationale, I just know it’s probably going to be the worst matrix you ever see:

Click to see it full size. You might want to print it out and tape it up to remind you to never do this, though I suspect you wouldn’t be reading this if you would.

Eats, Shoots & Writes Online Polls for Greenfield

Three things I was going to do, but didn’t:

1) Leave out the Greenfield logo. I’m acutely aware that I pick on them quite a bit, so I wasn’t going to say who was behind this one. Then I realized it was probably obvious.

2) Write an entire rant about the complete lack of proofreading, quality assurance, and consideration for user experience that has to exist in a corporate culture for something like this to ever be exposed to the general public. Consider the previous sentence to be the rant; I don’t think I need to expand upon that.

3) Fired off an email to Greenfield offering to solve these constantly recurring problems for them for a very affordable ongoing consulting fee. I might still do that, come to think.

Enjoy the rest of your Sunday, folks.

Please Stop Doing This.

I really should have called this thing “matrixes are evil,” instead of “researchcrants.”

Oh well.

STOP IT WITH THIS:

If the question make me groan, screengrab it, and blog about it, you can guarantee it’s making some percentage of your respondents close the tab and get on with their lives.

You are the cancer that is killing market research Questions like these are directly responsible for the decreasing quality of market research.