Where and how do we begin?
I’ve been working in the research industry for about fifteen years. My career path did not travel a straight line, and my undergraduate degree is in another field. I have no graduate degree. My professional experience comes under the tutelage of others who similarly were trained outside of the ivory towers of academic polling. I’ve learned and improved by doing. I’m not in any way a math or statistics expert. I haven’t done regression analysis or chi-square tests since college.
To some of you, I’ve just disqualified myself as a researcher and critic. That’s a shame, because you’re probably the most in need of the following advice:
Hire someone who knows how to use words effectively, and let them edit your surveys down to their core. (That person, by the way, probably isn’t currently a researcher. That’s OK. It’s much easier to teach someone how to write good questions than it is to teach them to write in the first place.) Stop asking the same question six different ways; ask it once — correctly — and move on. Stop abusing your respondents by subjecting them to telephone surveys that go on forever and web panel studies that present page after page of matrices that practically beg respondents to click random choices just to get it over with.
It drives me crazy that so much of the research I see is so very bad. The reasons vary, but the outcome is the same — frustrated respondents and unreliable results.
My purpose here is simple — to display instances of what I consider to be bad research in the hope that we all can learn by example and improve. Some of what we’ll see is flat-out awful; other examples will be less egregious. There also are, of course, researchers who are doing great, innovative work, and I hope to hold some of that up as examples of what we all should be aiming for.
The better we make our questions — the better job we do of survey design — the better our results will be. I hope this blog can become a home for what I see as a movement — the movement to make our research instruments more friendly to respondents, which in turn will make our data better.