Obscure AND Potentially Personally Identifying? Let’s Ask It!

Sent in by a reader; click to embiggen:

Bad enough they’re asking for something few people would know offhand — and who wants to go fetch a piece of mail to get the answer  — but I think there’s an equally bad issue here regarding respondent confidentiality, at least theoretically.  A quick search of census data for some five-digit zip codes chosen at random from among those I’m familiar with around the country shows between about 8,500 and 16,000 occupied households in each. (I wouldn’t call that an average, as it’s practically anecdotal, but it’ll do for now, since I can’t find exactly what I’m looking for.)  A zip+4, though, is designed to be reflective of a much, much smaller geography. According to the US Postal Service:

The 4-digit add-on number identifies a geographic segment within the 5-digit delivery area, such as a city block, office building, individual high-volume receiver of mail, or any other unit that would aid efficient mail sorting and delivery

How small are those “geographic segments?” You can use this USPS lookup tool to get a sense of it. I live on a suburban street; my house is on a corner. My immediate neighbor around the corner has a different zip+4; the people across the street have a different zip+4; the house immediately behind me has a different zip +4. The house next door to me, though, and the two houses that follow it going down to the end of the block — those all have the same zip+4 data. Apparently, my personal zip+4 will narrow you down to one of four homes.

Now, presumably, you gave your full mailing address when you signed up for this panel, so it’s not as if the research company) doesn’t already know exactly who you are and where you live — and it’s not as if telephone research doesn’t contain your even more personally identifiable phone number right there in the data — but still, this makes me uncomfortable. Rather than using back-end databases to append that information in post-production (which, for the millionth time, would be the ideal way to deal with this situation), we’re instead outright asking for something that both makes your data pretty easy to tie back to you and which you don’t know in the first place. (I actually thought I knew mine, and I don’t, though I was fairly close.)

All in all, this strikes me as a really bad question. What do you think?

3 Comments

Filed under bad user experiences, data quality, databases are your friend, ethics, Market Research, redundant questions, web research

LA Times: What the What?

So for a couple of weeks now, I’ve been getting emails from the Los Angeles Times about how my email newsletter subscriptions are about to end. I’ve been ignoring them, because I don’t think I actually get any emails from the Los Angeles Times. I suppose I must have registered with a real email address on their site to read a story once, years ago, before BugMeNot and their Firefox extension made such things unnecessary. In any case, I don’t care, fine, whatever, stop sending me those newsletters you’re not actually sending me, I’ll find a way to survive, despite the longing I shall forever feel in my heart.

Just now, though, I got this brilliant piece of email from them:

“Why have we stopped sending you emails?” WHAT DO YOU THINK THIS THING IS? IT’S AN EMAIL! THAT YOU’RE SENDING ME! ABOUT HOW YOU’VE STOPPED SENDING ME EMAILS WHICH IN ACTUALITY YOU NEVER WERE SENDING ME IN THE FIRST PLACE!”

It boggles the mind.

Leave a comment

Filed under abject stupidity, bad user experiences, non-research, silly nitpicking, what

Research Lifestreaming

Harris (click to embiggen):

I’m fascinated, but I think the universe might collapse in on itself in some sort of divide-by-zero error if I were to sign up with @researchrants.

In any case, I want to hear from anyone who does sign up … and I’d love to see any examples of gaming this system. I mean, there have got to be brand managers salivating over this, right?

11 Comments

Filed under a challenger appears, Harris, Market Research, social media, web research

Probability Sampling vs. Web Panel Sampling vs. Interviewing People Completely By Accident

A reader sent me this a number of months ago, and embarrassingly, I’m just getting around to posting it now, as I’ve been reminded by something @Lovestats posted the other day.

Another matrix with multiple problems — “I felt expected,” really? But the focus here actually isn’t the matrix, believe it or not: it’s that my reader has never actually stayed in the hotel that sent him the survey, with the familiar “please rate your recent stay with us at our such-and-such location” sort of email you tend to get after any interaction these days. Not only had he not stayed at that location — he’d never stayed at any hotel in the chain.

Now, I’m assuming this was a glitch of some sort — an actual customer with a similar email address had sloppy handwriting, or a friend of our reader has been giving out the reader’s information instead of his own, or the hotel company is a conglomerate and they actually meant to ask our reader to rate his experience buying a shirt at their clothing store  — but however it came about, it made me think — with all the ongoing discussion about probability samples versus non-probability samples, what about starting by just making sure you’re not interviewing people who are totally outside your frame?

2 Comments

Filed under Market Research, matrixes make me cry, web research, worrying about the wrong thing

Are You Polling a Lot of Miners?

Just clicked a link to take a survey about Twitter, hosted at SurveyMonkey, and was immediately struck by the odd choices in their “profession” question:

I’m frequently aggravated by this question on a personal level, because “market research” is almost never a choice. While you sometimes see a “marketing” choice, if there is a “research” option, it somehow always seems (either explicitly or via connotation in my mind) to be about white-coated goggle-wearing folks in chemistry labs. But I’m not really talking about my own experience here as much as I’m just commenting about how odd it seems that Twitter, of all things, expects to have so many of its users working in the mining, farming, and construction fields, none of which traditionally lend themselves to being able to sit in front of a computer and update the world about your latest moment-to-moment activities. While I’m sure there’s a guy in a combine somewhere right now using Foursquare to tell us exactly which corner of his field he’s in, I’m going out on a limb and guessing there aren’t more than a handful of guys like that. (And why on earth are farming and mining combined? Seems weird.) If I were designing this piece of research for Twitter, I think I’d have a lot more choices that might apply to people who are likely to be using a computer for 8 hours a day.

tl; dr: one size doesn’t fit all. Customize and update where needed.

3 Comments

Filed under answer choices, Market Research, social media, web research

CNET: At Least They Warn You It’s Going to be Horrible?

15-20 minutes? Even I probably won’t click to take this one.

Oh, all right, I did. Got about three minutes in (it’s about reward programs), hit my first matrix, and decided I wasn’t going any further:

Sorry, CNET, I’m out. Not only does this show every sign of making me miserable, it also doesn’t appear to have anything to do with your business, since I can’t imagine a product review site launching its own rewards program. If the research looked like it was directly connected to improving the site I actually was interested in viewing when I went to cnet.com this morning, I might have been willing to go further with this, but not for this.

Leave a comment

Filed under bad user experiences, Market Research, matrixes make me cry, web research

No Way Do Two Thirds of Americans Have HDTV. No Way.

Sorry, but I’m willing to bet this piece of research is completely wrong. I’d want to see the actual questionnaire, but here’s what I’ll assume until then:

Many, many, many people have no idea whether or not they have HDTV. Two main reasons:

  1. There is a serious lack of understanding among non-techie respondents about the terms “digital,” “high definition,” and “HDTV.” I’ll bet $20 that at least 20% of the population thinks they have HDTV because they bought a $40 conversion box for the digital transition.
  2. Because for years now, everything from network dramas to local newscasts has been opening with an onscreen logo that says something like “in HD where available,” or “presented in HD,” just like they used to do the exact same thing for stereo … only now they’ve also gone and incorporated it right into their station logos.

That’s right. Viewers with old 4×3 standard definition TV sets are constantly shown on-screen graphics that, in combination with the fact they bought conversion boxes, has them convinced they’re watching HDTV:

“Of course I have HDTV! It says HDTV right there on the screen!”

It’s difficult to research a topic when respondent confusion is this widespread. It’s not completely impossible, but it’s really, really hard. I can think of a couple of ways to try to do it, but they’re so cumbersome (as in, “look behind your TV and tell me the model number”) that they’re just not going to work.

Oh, and let’s not forget that there’s also God-only-knows how many people — this would include many of our parents, I’ll wager — who have HDTV sets but are watching standard definition broadcasts on them.

16 Comments

Filed under data quality, Market Research, TV