Straightlining vs. Answering Your Stupid Question Honestly

OK, this is something I hadn’t thought of before.

When I’m staring at a bad survey question — asking me to compare two absolutely identical companies in a matrix, for instance — my tendency is to do this:

They’re equal. There’s no difference between Visa and MasterCard in my mind. Discover and American Express, those are different, both from one another and from these two brands, but Visa and MasterCard might as well just merge, as far as I’m concerned. Of course, there’s no way to provide that answer in the framework provided here, so I decided to simply give each company a score of “5″ for each item. That seemed to get the message across, as far as I was concerned. Of course, as soon as I clicked the button, I got booted, with the same generic non-qualified message you get when you tell them you don’t have kids or haven’t seen a movie in the past two months or whatever it is. We all know the truth: they booted me for straightlining.

Which I wasn’t.

At the very least, wouldn’t it be smarter to keep me in and see what the rest of my answers looked like? With the amount of amply-documented badly designed questionnaires out there, shouldn’t we maybe consider that a respondent will occasionally need to do something to get around a poorly framed question, or an item that simply doesn’t apply to them?

Simply ending the survey as soon as someone gives all items on a page the same value seems both too simplistic and too drastic a solution to me.

Another Fine Matrix

First, look at this full-size. See how there are 14 brands of cat food going across the top? I already told it I’d never heard of five of them, and yet here they all are again. It’s one thing to ask me if I’ve ever heard of a brand and to then, even if I haven’t, show me an ad for the brand and ask if I’ve seen that ad — I very well could have forgotten about it, or misremembered what brand it was for.

This is just stupid…

Worse, though, it’s endless. Here’s a reduced-size capture so you can see how long it is:

This is what I’m referring to in the comments on Gary Langer’ post here — what the hell sort of non-representative person is going to sit through this? This kind of garbage really is the cancer that is killing market research. Stop pulling this crap, and then go and worry some more about probability samples.

Needs Moar Choices.

Seriously? Shouldn’t they also have broken out high school by year, or something? Maybe included a radio button for each individual year from kindergarten through law school? No, really, I just can’t imagine how such fine distinctions are useful to anyone. Is someone really looking at this and saying, “Wow, the 7 respondents with some advanced degree work are slightly more likely to say x than the 11 respondents who are currently in advanced degree work! Fascinating! Oh, wait, the margin of error is +/- 37.8%.”

I get that there’s value in collecting more, not less data; I’m a firm believer in asking respondents for their actual ages, actually, instead of for a range — because when you have the actual data, you can put it back together in whatever groupings you want, which may not be the groupings you think make sense before you see the results — but this here is just a mess.

Is the panel research business model creating a gold farming problem?

Greenfield must be having trouble getting panelists to complete research these days. Maybe it’s the summer blahs, with respondents too hot, too sweaty, or just too on vacation to be bothered.

Then again, maybe it’s something to do with people just getting sick of trying to imagine their orange juice has come to life and is displaying personality traits.

In any case, I’m sure this is the answer:

50 cents per survey! At 20 minutes per survey, that’s like, $1.50 an hour! This will totally solve all of Greenfield’s problems, and can only lead to amazing data quality.

Right?

But let’s turn this repetitive Greenfield mockery into a real question:  what are the odds that this sort of incentive (and incentives in general, really) has already led to or soon will lead to the market research version of “gold farming?”

Gold farming, if you don’t want to bother reading the Wikipedia entry, is an exploit carried out within massive online role-playing games, like World of Warcraft. I’m no expert in it, but as I understand it, people hire low-wage workers (this has apparently been an issue in China) to sit in front of multiple computer terminals logged into the online game. The workers don’t actually play the game as it’s intended to be used, but they instead perform repetitive actions, generally using automated scripts, to earn (or, colloquially, to farm for) in-game cash — virtual money, essentially, that can be spent on in-game items like better weapons and the like. The folks behind the operation then sell the virtual currency online, to actual players of the game who want to buy a really cool sword or whatever but who can’t be bothered spending weeks building up the in-game cash to buy it.

So, since Greenfield is paying 50 cents for 20 minutes worth of human labor here, it occurs to me that someone has probably already figured out that they write some scripts to blast through these things in (let’s say) five minutes each — 12 per hour, as opposed to 3 per hour. And that’s per computer. So you sit a guy in front of five screens, each logged in on a different Greenfield account, each earning $6 an hour — so $30 an hour across those five screens — you know, if your labor only needs to make around $3 an hour, that’s $216 a day in pure profit for the guy in charge. And that’s assuming he’s only got one guy doing this on only five accounts at once.

Now, I’m sure I can’t be the first person this sort of thing has occurred to, and I’m sure Greenfield and the other panel outfits are trying their hardest to make this impossible, limiting the number of surveys one respondent can complete in a day, maybe checking for a total elapsed time and invalidating surveys that move too quickly — but, I don’t know, that strikes me as sort of being similar to making the roads near the bars really wide and straight instead of outlawing 24-hour happy hours, or some similarly goofy comparison.

If we want honest answers from real people, maybe we should rethink this entire insulting “we’ll pay you fifty cents to answer 120 repetitive questions about the minute differences between four brands of orange juice” business model.

Resident of What, Now?

Does anyone else have trouble answering this? I see it on nearly every Zogby Interactive poll I get, and I never know what to do with it:

I just don’t get it.

Maybe I’m overthinking it — it happens — but I can’t figure out exactly what the underlying idea of the question is. I get why he asks if I have a passport, if I watch NASCAR, and how often I shop at Wal*Mart — it’s his “elites vs. normals” crosstabs, or whatever, and I assume this is supposed to be the same sort of thing, but I can’t figure it out.

Worse, I think the one choice I WOULD pick — that I think of myself as a resident of a particular region of the United States — isn’t listed.

Anyway, just wondering if anyone else is over-thinking this one and has trouble answering it as well.

Blank = Zero

Can we please just all agree that blank = zero?

Greenfield, unsurprisingly:

Come on, guys. Stop making things hard for respondents. It’s not like you have enough of them to begin with.

Zogby Motorcycle Emptiness

What fresh hell is this? John Zogby has discovered the matrix? And somehow made it uglier than any other matrix in the history of research? All true, and more on that in a moment, but first, this unusual use of the ticky box:

That’s probably illegible unless you click on it, but the gist is I’m being asked, in a pretty wordy fashion at that, which of the following things I’m interested in or knowledgeable about; nothing inherently wrong with that setup, except this: I’m then given exactly one box to check (or not check, as the case may be) : “motorcycles.”

Let’s type it out and count, OK?

John’s Way: (53 words)

“Now for some questions about consumer goods categories. Please choose the categories in which you feel you have a particular interest and knowledge. This means you actively seek out information on these products and services (for example you watch TV shows/read magazines and websites/attend exhibitions/discuss with friends and colleagues etc.)

My Way: (6 words)

“How interested are you in motorcycles?”

Now, it’s possible the reason I only saw the single checkbox for motorcycles is because my answers to the previous series of questions disqualified me from everything else, although I don’t really think my responses would have pointed in the direction of motorcycles,  but who am I to judge? Anyway, as promised, here’s what the first screen of these looked like:

Maybe I shouldn’t say this is uglier than any other matrix — maybe it’s just that it’s simpler, in that it uses less newfangled HTML and is therefore, I don’t know, easier to access via mobile browsers, which isn’t a bad design goal to have — it’s just very strange looking to me, and pretty hard to take in at a glance. By the time you get down to the last button on the right, it’s not immediately obvious to me if that “1″ radio button is for “Pessimistic” or for some other word that I perhaps need to scroll down for, or that’s just not appearing for some reason.

Are mobile/degraded browsers a big factor in the panel research industry? Are there a lot of folks on the Greenfield panel using Netscape 2.0 on Mac IIci’s or something? Because I seriously don’t get why this hasn’t all been replaced — and I’m talking about everyone here, not just Zogby — with some well-designed Flash code. Seems to me a freshman design student could pretty quickly mock up something vastly superior to anything being used in the industry today, no?

One other thing I found interesting: on some (but not all) screens of this survey, when I clicked the final radio button, I was automatically advanced to the next page. Despite the pages having a “continue” button on them, by the way. I don’t have a problem with auto-advancing in general, though I think it needs to either consistently happen 100% of the time or 0% of the time — but I’m curious what y’all think about it. Is the auto-advance, which would make it difficult or impossible to go back and fix an error you made, a good thing or a bad thing?

Excuse me while I vomit.

Protip: just because you’ve somehow managed to combine some letters to form some words and you’ve placed radio buttons alongside them doesn’t mean you’ve created a poll.

Bonus protip: you can see research that’s just as bad (but generally a lot less offensive) on the home pages of most every local TV station, newspaper, and radio station in America.

Health Survey!

You know what works better than an incentive? For me, anyway?

Exclamation points! The enthusiasm is contagious! I can’t wait to click the link and take the survey.

Of course, I’ll bet the staid researchers who programmed the survey itself aren’t so excited about it, though.

Oh my gosh! They totally are! This is going to be the best survey ever!

…

Or not. But it was refreshingly matrix-free and fairly speedy. And oddly, I know I’m sort of making fun of it, but — the exclamation point actually worked, unintentional though I suspect it must be.

Meanwhile, Deep in John Zogby’s Secret Underground Lair/Science Lab…

I’m not the first one to see this, but that never stopped me before; everyone loves a good dogpile.

The Borg Collective fine people at Zogby Interactive just asked me this interesting set of questions:

What the hell are they working on up there in Utica?