Straightlining vs. Answering Your Stupid Question Honestly

OK, this is something I hadn’t thought of before.

When I’m staring at a bad survey question — asking me to compare two absolutely identical companies in a matrix, for instance — my tendency is to do this:
straightline

They’re equal. There’s no difference between Visa and MasterCard in my mind. Discover and American Express, those are different, both from one another and from these two brands, but Visa and MasterCard might as well just merge, as far as I’m concerned. Of course, there’s no way to provide that answer in the framework provided here, so I decided to simply give each company a score of “5” for each item. That seemed to get the message across, as far as I was concerned. Of course, as soon as I clicked the button, I got booted, with the same generic non-qualified message you get when you tell them you don’t have kids or haven’t seen a movie in the past two months or whatever it is. We all know the truth: they booted me for straightlining.

Which I wasn’t.

At the very least, wouldn’t it be smarter to keep me in and see what the rest of my answers looked like? With the amount of amply-documented badly designed questionnaires out there, shouldn’t we maybe consider that a respondent will occasionally need to do something to get around a poorly framed question, or an item that simply doesn’t apply to them?

Simply ending the survey as soon as someone gives all items on a page the same value seems both too simplistic and too drastic a solution to me.

Advertisements

5 Comments

Filed under answer choices, bad user experiences, data quality, Greenfield, Market Research, matrixes make me cry, web research

5 responses to “Straightlining vs. Answering Your Stupid Question Honestly

  1. There should be a box for feedback on what the respondent (dare I care them that anymore?) thought of the question and/or allowing space to further explain their answer (spot the qualitative researcher! ; P).

    Love the automatically generated “possibly related post”. Sums it all up really, doesn’t it!

  2. Oh gosh, I didn’t see the possibly related posts until now. CLASSIC.

  3. Cross Man

    and…. the answer to: “is the most convenient” is a single brand (or maybe 2 equal brands in your case) not a number… the question that has a number as the answer is “is convenient” – then you can say how convenient it is…. it’s only english after all, we all speak it..kinda. sorta

  4. Ditto. My experience with one research company is that you would not be kicked out for that. Your full series of answers would be reviewed and if enough indicators pointed to carelessness, then you would be kicked out. It’s far too easy, and wasteful, for companies to terminate based on one inappropriate indicator. Too many companies do this.
    Maybe the new ARF research findings will change this!

  5. Rick Frank

    Totally agree, we flag grids with the same answers & review manually while in field.

    Sometimes straight-lining IS valid as your example points out.

    The usual test of good vs. bad is straight lines PLUS something like gfhgfhgfhgfh in a verbatim field. If you have a well answered verbatim I will accept whatever numeric data you have submitted because cheaters don’t wrote thoughtful answers.

    Cheers

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s