Category Archives: databases are your friend

Yet Another Example of How Not To Use The Internet To Conduct Research

(edit, April 6, 2011: Over a year since I posted this, and I just took another Zogby poll (now an “Ibope Zogby poll,” by the way), and they’re still asking this question the same way. And I still, despite being pretty politically aware, knowing my congressman’s name, and having even written the guy and gotten a response on at least one occasion, have absolutely no idea what district number I live in. Everything below this parenthetical addition is old content, so if you have seen it before, sorry.)

This is from a couple of weeks ago, and I’m just now getting a chance to post it.

88% of Americans live in a state with fewer than 53 US congressional districts in it. Only California has that many; Texas comes in second with 32.

And yet, here’s how the good folks at Zogby Interactive ask what congressional district you live in:

That’s right. Zogby asks what state you live in, and then asks you, regardless of how many districts your state contains, which of 53 districts you live in. This is terrible for a lot of reasons, beginning with what should be obvious to everyone: it’s really lazy.

Looking at this from a practical political standpoint, though, it’s a mess. Folks just don’t think about their congressional district that way. Many (certainly not all) will know the name of their representative — or at least be able to pick the name from a short list of possibilities — but the odds of them knowing the actual district number aren’t great.

That being said: it can be problematic to ask people who their representative is if you’re then going to ask them if they’re going to vote for that person — it creates a priming effect and reminds (or informs, if the respondent is less focused on politics) of incumbency and makes it difficult to do a clean “would you vote for x or y” question. While I didn’t get that question as a follow-up, it’s possible some respondents did, though I somewhat doubt it this far out.

A much better way to ask this question is to ask for zip code, which will let you look up the right district in most cases; a simpler method (for the respondent), and one that might feel less personally intrusive, is to remember that this is the internet and present a state map, on which the respondent can zoom in and actually CLICK WHERE HE LIVES.

And, saying what should be obvious, but maybe isn’t: if you structure your research in such a way that only the very very very top-most super-engaged respondents are qualified to answer a follow-up, your results are only going to reflect that tiny slice of the population.

Pathetic, and sadly, about what one would expect.

Leave a comment

Filed under abject stupidity, bad user experiences, databases are your friend, election polling, Market Research, Politics, quality of responses, the web is a visual medium, Zogby

Databases Are Your Friend!

I’ve ranted about this so many times that it’s a true pleasure to see it being done the right way by YouGov/Polling Point here:

(obviously, I blanked out the zip code.)

Compare and contrast with other folks’ ongoing aggravation of asking me to pick which country I live in (from a list of about a billion, though the US at least is at the top), then pick my state, and then entering my zip code. Harris does this pretty frequently, though I just saw an example the other day where they instead asked if I still lived in the United States, and then (without asking for state or zip) asked me to type the name of the city where I live, which I found somewhat unusual.

Anyway, nice to see this happening. We’re already on the panel, so you already know all this background info!

1 Comment

Filed under databases are your friend, Doing it right, Market Research, web research, YouGov

Obscure AND Potentially Personally Identifying? Let’s Ask It!

Sent in by a reader; click to embiggen:

Bad enough they’re asking for something few people would know offhand — and who wants to go fetch a piece of mail to get the answer  — but I think there’s an equally bad issue here regarding respondent confidentiality, at least theoretically.  A quick search of census data for some five-digit zip codes chosen at random from among those I’m familiar with around the country shows between about 8,500 and 16,000 occupied households in each. (I wouldn’t call that an average, as it’s practically anecdotal, but it’ll do for now, since I can’t find exactly what I’m looking for.)  A zip+4, though, is designed to be reflective of a much, much smaller geography. According to the US Postal Service:

The 4-digit add-on number identifies a geographic segment within the 5-digit delivery area, such as a city block, office building, individual high-volume receiver of mail, or any other unit that would aid efficient mail sorting and delivery

How small are those “geographic segments?” You can use this USPS lookup tool to get a sense of it. I live on a suburban street; my house is on a corner. My immediate neighbor around the corner has a different zip+4; the people across the street have a different zip+4; the house immediately behind me has a different zip +4. The house next door to me, though, and the two houses that follow it going down to the end of the block — those all have the same zip+4 data. Apparently, my personal zip+4 will narrow you down to one of four homes.

Now, presumably, you gave your full mailing address when you signed up for this panel, so it’s not as if the research company) doesn’t already know exactly who you are and where you live — and it’s not as if telephone research doesn’t contain your even more personally identifiable phone number right there in the data — but still, this makes me uncomfortable. Rather than using back-end databases to append that information in post-production (which, for the millionth time, would be the ideal way to deal with this situation), we’re instead outright asking for something that both makes your data pretty easy to tie back to you and which you don’t know in the first place. (I actually thought I knew mine, and I don’t, though I was fairly close.)

All in all, this strikes me as a really bad question. What do you think?


Filed under bad user experiences, data quality, databases are your friend, ethics, Market Research, redundant questions, web research

If you’re going to use the web, USE THE WEB.

Why is everyone still doing this?

Yes, when you report research results to your client, you’re going to want to group the responses into age groupings. Makes sense.

Why, though, are you asking your respondents to enter their answer using those same groupings? Three real issues I have with this approach:

1) When I see groups, I immediately think that some of the groups will qualify to continue, and others won’t. I want that $2, or that sweepstakes entry, or, frankly, I want to kill fifteen minutes; I don’t want to get told I don’t qualify. Therefore, I’m not telling it I’m in the lowest group or the oldest group, and really, probably not the next ones in, either. I’m playing it safe and saying I’m 30-59; that’s probably the safe range.

2) Down the road, if the client wants to see the data broken out in different ways — let’s say the client comes back and asks to see the 25-54 year-olds broken out — there’s no way to provide that data.

3) Why limit yourself to nothing but radio buttons? It’s not difficult to program a text-entry field that’s limited to accepting only digits; it’s not especially difficult to program a slider that lets respondents navigate to their exact age or year of birth, either.

4) Yes, I said three real issues; this is the bonus, fourth issue. This isn’t RDD research — this is web panel work. The panel company already knows my age, as well as my gender, my race, my current mode of internet connectivity, and a thousand other things. Why can’t it use that stored data to fill in the correct responses to these questions without ever even showing them to me? Yes, that requires a lot more work — that’s why I consider this a “bonus” item — but the panel company that figures out how to do this reliably and consistently will reap the benefits of significantly shorter surveys — especially on the qualifications end. Greenfield is particularly bad about this, asking me the same five or ten initial questions before every piece of research and then telling me I’m not qualified for it — so why did they present it to me in the first place?

Leave a comment

Filed under databases are your friend, the web is a visual medium, thinking ahead for client needs