Yet Another Example of How Not To Use The Internet To Conduct Research

This is from a couple of weeks ago, and I’m just now getting a chance to post it.

88% of Americans live in a state with fewer than 53 US congressional districts in it. Only California has that many; Texas comes in second with 32.

And yet, here’s how the good folks at Zogby Interactive ask what congressional district you live in:

That’s right. Zogby asks what state you live in, and then asks you, regardless of how many districts your state contains, which of 53 districts you live in. This is terrible for a lot of reasons, beginning with what should be obvious to everyone: it’s really lazy.

Looking at this from a practical political standpoint, though, it’s a mess. Folks just don’t think about their congressional district that way. Many (certainly not all) will know the name of their representative — or at least be able to pick the name from a short list of possibilities — but the odds of them knowing the actual district number aren’t great.

That being said: it can be problematic to ask people who their representative is if you’re then going to ask them if they’re going to vote for that person — it creates a priming effect and reminds (or informs, if the respondent is less focused on politics) of incumbency and makes it difficult to do a clean “would you vote for x or y” question. While I didn’t get that question as a follow-up, it’s possible some respondents did, though I somewhat doubt it this far out.

A much better way to ask this question is to ask for zip code, which will let you look up the right district in most cases; a simpler method (for the respondent), and one that might feel less personally intrusive, is to remember that this is the internet and present a state map, on which the respondent can zoom in and actually CLICK WHERE HE LIVES.

And, saying what should be obvious, but maybe isn’t: if you structure your research in such a way that only the very very very top-most super-engaged respondents are qualified to answer a follow-up, your results are only going to reflect that tiny slice of the population.

Pathetic, and sadly, about what one would expect.

No Way Do Two Thirds of Americans Have HDTV. No Way.

Sorry, but I’m willing to bet this piece of research is completely wrong. I’d want to see the actual questionnaire, but here’s what I’ll assume until then:

Many, many, many people have no idea whether or not they have HDTV. Two main reasons:

1. There is a serious lack of understanding among non-techie respondents about the terms “digital,” “high definition,” and “HDTV.” I’ll bet $20 that at least 20% of the population thinks they have HDTV because they bought a $40 conversion box for the digital transition.
2. Because for years now, everything from network dramas to local newscasts has been opening with an onscreen logo that says something like “in HD where available,” or “presented in HD,” just like they used to do the exact same thing for stereo … only now they’ve also gone and incorporated it right into their station logos.

That’s right. Viewers with old 4×3 standard definition TV sets are constantly shown on-screen graphics that, in combination with the fact they bought conversion boxes, has them convinced they’re watching HDTV:

“Of course I have HDTV! It says HDTV right there on the screen!”

It’s difficult to research a topic when respondent confusion is this widespread. It’s not completely impossible, but it’s really, really hard. I can think of a couple of ways to try to do it, but they’re so cumbersome (as in, “look behind your TV and tell me the model number”) that they’re just not going to work.

Oh, and let’s not forget that there’s also God-only-knows how many people — this would include many of our parents, I’ll wager — who have HDTV sets but are watching standard definition broadcasts on them.

Gary Langer Hates Robopolls, Probably Because They’re More Accurate Than His Live Interviewer Polls And He’s Apparently Just Jealous, The Poor Thing, So He’s Decided That Accuracy Doesn’t Matter.

Gary Langer, Director of Polling at ABC News, has a problem with what he calls “robopolls” and what we’re going to call interactive voice response, or IVR, polls.

As best I can tell, his root problem is that they’re more accurate than his live interviewer polls, despite using what he considers inferior methodology. But here, let him tell you:

In our ABC News polling standards we don’t regard autodialed, pre-recorded polls as valid and reliable survey research. Our concerns include noncoverage, nonresponse, lack of respondent selection, nonvalidation, opaque weights and the absence of a persuasive, independent literature evaluating the approach. Some other news shops – the few that, in my view, take coverage of public opinion seriously – share these judgments.

There’s the germ of a good point or two in there, to be honest with you, at least on the surface, but let’s go through all of them and poke some holes:

• Noncoverage: It’s illegal to call a cell phone with an autodialer, so by default, a poll conducted by an IVR firm probably doesn’t include any cell phone interviews, so yes, cell phone only households wouldn’t be covered in the sample — but, first, there’s nothing keeping an IVR firm from hiring a data collection house to make cell phone calls with live interviewers to supplement their IVR polling — IVR is a method of conducting polls, not an ideology — and second, as it happens, there’s a bit of an intersect between the people most likely to live in a cell phone only household and the people least likely to vote. Not that Langer cares about results, but the studies I’ve seen have suggested that so far, this isn’t a factor.
• Nonresponse: I suppose response rates could be lower for recorded voice polls than live voice polls, but they could also be higher. I don’t know, and Langer doesn’t know either: there are a number of different ways to calculate response rates, and few pollsters release them at all, and when they DO release them, they all seem to pick a different measure. See page 21 (table 5) of this AAPOR report on 2008 primary polls — there isn’t a single contest there where you can compare an IVR and a live interview pollster’s response rate because none of the pollsters seem to agree on which measure to use!  (There’s also a body of literature suggesting there’s really not much correlation between response rate and accuracy, but Gary doesn’t care about accuracy, just methods, so I guess we won’t bother linking to any of that. Or not.)
• “Lack of respondent selection” is a major bullshit non-issue, brought up time and time again by Langer and his ilk to discredit research that threatens their world. According to them, your poll is unreliable if you don’t start every call by asking to speak to a specific random person — the youngest male, for instance, or the person in the household with the next birthday. There’s absolutely no reason why “robopolls” can’t do this — it’s just another question, for crying out loud — and as far as I know, there’s no evidence that there’s any reason for them to do it, or for anyone else to, other than because Langer says they should.
• “Nonvalidation.” I have no idea, but assume this means Langer thinks good polls employ extra live interviewers to call back selected respondents to make sure the original live interviewers actually did contact the respondents and that the original live interviewers didn’t just make up the data so they could collect their $25 or whatever and go home. Hmm. Gosh, Gary, why would it possibly be the case that IVR polls don’t bother doing this? Maybe because they don’t have a room full of telemarketers falsifying data so they can go home early? (And, once again: why couldn’t IVR pollsters do validation? If it mattered, is there some reason they couldn’t make those callbacks, either through an IVR or a contracted live interviewer methodology? Couldn’t the office manager make the calls the next day? You only “need” to validate a sampling of the calls, right?)
• “Opaque weights” is another term that doesn’t mean a lot to me. Apparently Gary thinks the IVR pollsters are more secretive about how they weight their data than the live interview pollsters. No idea why that would be the case, no idea what evidence there is suggesting it is.
• The absence of literature is one of my other favorites. Yeah, Gary, there’s not a lot of peer-reviewed scholarly articles about IVR polls, all of which happen to be run as businesses (as opposed to academic projects) because, unlike academics, business people don’t usually write peer-reviewed scholarly articles.

What else you got, Gary? Oh, right:

Some defenders of robopolls will point in another direction, at those that have accurately predicted election outcomes. From my perspective, accurate modeling is not an adequate stand-in for good polling. For reliability and validity alike, what matters are sound methods. And what matters next are substantive measurements, not horse-race bingo.

Wait, sorry, I couldn’t hear you, I was laughing too hard. What? Is he seriously saying that it just doesn’t matter how accurate a poll is, it only matters if the pollster follows the right rules? The Gary Langer rules? Oh, Gary.

Look: twenty years ago, if you wanted to take a public opinion poll, you needed, I don’t even know, a Ph.D., a university, and I’m going to guess something like $40,000.  Today, you can get a PPP or a SurveyUSA or a Rasmussen to do it for you for maybe 10% of that, or you can just go do it yourself for a few hundred dollars. I get that this is scary as hell if you came up through the academic ranks; I get that irrelevancy really  pisses people off — but I just don’t think you have a lot of credibility when you try to tell us that whether or not you get elections right isn’t a good measure of your skill as an election pollster. From where I sit, it’s the only thing.

(PS: PPP, I love you guys, but I think he’s right about one thing: an “or not” wording choice would probably be better.)

tl; dr: threatened much?

ETA: Forgot to add this, which is Nate Silver’s last (and horribly outdated, from May of 2008) pollster ranking chart. Still, it’s the last thing he gave us:

Look on the bright side, Gary: you’re better than Zogby.

Just Say No Already.

Annie Pettit this morning tweeted from the Net Gain 4.0 Conference in Toronto:

Clients still want 1 hour surveys and we can’t do anything about it : I say turn it down!!

I’ll go further than that: I say turn it down and make it clear to the client that they are the cancer that is killing market research. What in the world can you learn from a sixty minute survey that you can’t learn from a 5-minute one? (I’m not talking about an in-depth qualitative research project, or something where you hook someone up to an EEG and have them watch an episode of CSI: Miami to see what their brain has to say. I’m talking about asking questions, on the phone or on a screen. 60 minutes is 55 minutes too long!

Do we really think the respondents still on the phone (or on the web) at the one-minute mark, the ten-minute mark, and the 60-minute mark are identical?

Joel on Research and the Paradox of Choice

Joel Spolsky runs a software company and writes a blog called Joel on Software, but I often find what he writes is applicable to business in general, and occasionally to the research world as well. Recently, in his Inc.com magazine column, he wrote about the problems that develop when too many people are involved in a decision making process — in this particular case, he mentions how a former Microsoft developer tells how designing the Windows Vista shutdown menu took something like 43 people something like a full year and produced a menu with nine near-identical options. The developer calls it The Windows Shutdown Crapfest. The lesson there is obvious — too many cooks spoil the soup — and relevant, I think, to our work, with plain-as-day solutions — trim your meeting invite list and get extra hands out of the work — but dig a little deeper for a more important lesson.

Each of those links in the above paragraph is worth perusing, but the most worthwhile link I’ll have for you today is this, Joel’s original 2006 post on this topic, which does a great job of explaining the resulting mess in terms we all should be able to understand:

The fact that you have to choose between nine different ways of turning off your computer every time just on the start menu, not to mention the choice of hitting the physical on/off button or closing the laptop lid, produces just a little bit of unhappiness every time.

How do we expect our respondents feel when we ask them to tell us if they are Extremely Satisfied, Very Satisfied, Somewhat Satisfied, Neither Satisfied Nor Satisfied, Somewhat Unsatisfied, Very Unsatisfied, or Extremely Unsatisfied with something? What about when we ask them that about a whole page worth of somethings? And what about when some percentage of the questions — anywhere from 1/8 to 1/4 in my rough estimate — don’t apply to the respondent at all? I’d argue we create more than “just a little bit of unhappiness every time.”

The lesson is the same as it so often is here: keep it simple. Satisfied/Unsatisfied/Not Sure should be perfectly sufficient in mot cases, and has the advantage of making the results much more comprehensible at a glance. When comparing results across multiple questions, or across a wide time line of tracking data, it’s infinitely easier to comprehend a single number. The presidential approval number is generally reported as a single figure: 55% in this poll, 48% in this other poll, 53% a month ago, 58% today, etc. Instantly understandable by everyone, as opposed to something like this:

Today, 23% strongly approve of the President’s job performance; 27% say they somewhat approve. Two weeks ago, 29% strongly approved; 17% somewhat approved.

How do you parse that? Strong approval is up down 6 points at the same time that mild approval is up 10 points; overall, if you add the “strong” and “somewhat” numbers together, you can see that overall approval is up four points, but what do you do with those shifts in the gradated responses? Well, if you’re the nightly news — and I’m not suggesting that’s who we should be emulating, necessarily — but if you’re the nightly news, you ignore it and report it as a four point climb. (Well, depending on your viewpoint, you might say the nightly news would be most likely to point out the six point drop in “one measure of the President’s approval rating” and leave it at that, and I don’t think you’d necessarily be wrong about that observation, so.) If you’d only asked the question as approve/disapprove, though, you’d give respondents a simpler experience, and you’d give those interpreting the results both an easier time of it and less wiggle room for those with an agenda.

Let’s see what happens if we offer fewer choices. You really don’t need nine ways to turn off the computer, or seven ways to tell us how satisfied or unsatisfied you are. Honest.

“All this has happened before, and all of it will happen again.”

Not directly research-related, but worth noting to both sky-is-falling Democrats and gleeful Republicans alike:

Much is being made of the fact that since Barack Obama was elected, Democrats have lost races for governor in both Virginia and New Jersey, and of course, have now lost the Massachusetts US Senate seat held by Democrats since JFK won it in 1952. This is, depending on your personal beliefs, either a harbinger of doom or a reason to rejoice, but it seems to me it’s really just a repeat of recent electoral history. Newly elected Presidents of the previously out-of-power party lose their first off-year contests.

Check it out:

• November 1992: Bill Clinton (D) elected President after 12 years of Republican rule.
• November 1993: Christie Todd Whitman (R) elected NJ Governor, replacing Jim Florio (D)
• November 1993: George Allen (R) elected VA Governor, replacing Doug Wilder (D).
• June 1994: Kay Bailey Hutchison (R) elected US Senator from Texas, replacing Lloyd Bentsen (D).

Clinton, as you may recall, went on to serve two terms as President. Then, of course:

• November 2000: George W. Bush (R) elected President after 8 years of Democratic rule.
• November 2001: Jim McGreevey (D) elected NJ Governor, replacing Christie Todd Whitman (R). (technically, McGreevey was immediately preceded by a series of acting governors after Whitman resigned to join the Bush administration)
• November 2001: Mark Warner (D) elected VA Governor, replacing Jim Gilmore (R).

Bush, so soundly thrashed here in his first elections as titular head of his party, served two terms as well.

November 2008: Barack Obama (D) elected President after 8 years of Republican rule.

• November 2009: Chris Christie (R) elected NJ governor, replacing Jon Corzine (D).
• November 2009: Bob McDonnell (R) elected VA governor, replacing Tim Kaine (D).
• January 2010: Scott Brown (R) elected US Senator from Massachusetts, replacing Ted Kennedy (D).

I’m not saying progressives shouldn’t be concerned and that conservatives shouldn’t be happy — I’m saying this is in no way indicative of a game-changing turn of events. That’s all.

McGreevey

The Market Research Failure Behind NBC’s Jay Leno Debacle: Part II

Welcome back for the second part of this post.

NBC clearly  had a number of problems to solve. Lackluster ratings for the programs they were airing in prime time was one; the fact they’d promised the Tonight Show to Conan to keep him from leaving the network was another, as they’d have to pay him something like $40 million if they broke that contract; the fact Leno didn’t want to retire was a third. So, brilliant plan on NBC’s part: drop all the expensive 10 pm dramas that cost maybe $3 million per episode and replace them with the Leno show, for something like $2 million per week. Save a fortune in production costs, save $40 million in Conan payouts, and “manage for the margins,” as everyone reports NBC’s Jeff Zucker, pictured at right, planned to do.

Here’s where the research finally comes in: NBC apparently did the math, figured out what sort of ratings Leno would need at 10 pm in order to make the show profitable for the network via ad sales, and conducted viewer research to make sure enough people would watch the show to produce that rating. I’m sure that consisted of asking people variations of “what would you watch if all of the following programs were on at 10 pm,” and running down each night of the week, with Leno inserted in place of each NBC drama that was then airing. The results of that research seem to have been pretty accurate; the network was promising advertisers ratings that were pretty similar to what Leno was actually getting. They were embarrassingly low, I think, but they were around the base level that NBC had promised. The failure of imagination is that they just don’t appear to have asked much in the way of logical follow-ups of the people who said they wouldn’t be watching Jay at 10 — i.e., the vast majority of people. Questions like:

• Do you watch any NBC shows at 10 pm right now?
• After you watch one of them, what do you watch at 11? And then at 11:30?
• How often do you find out about one program by seeing a promo during another?
• If your favorite NBC show no longer existed, what do you think you would watch at 10 if these were your remaining choices? And then at 11? And 11:30?
• When you watch an 11:00 newscast, do you always watch the same station? Or do you watch different stations at different times?
• Do you ever decide to watch an 11:00 newscast because of a promo you saw earlier in the evening?
• Do you ever decide to watch an 11:30 or 12:30 late night show because of a promo you saw earlier in the evening?

(You can make an argument that they should have been asking a whole series of other questions as well,  like “Is Jay Leno even a little bit funny,” and “Would you rather watch Jay Leno or that guy singing ‘Pants on the Ground’,” or “What the hell were we thinking when we gave the Tonight Show to Leno and not Letterman in the first place all those years ago,” and “Hey what’s that over there oh my god look out it’s Johnny Carson’s ghost and he’s coming at you with a crowbar,” but that’s probably inappropriate here.)

I think if they’d thought to ask those questions, and had the imagination to work through what the answers met (and shared the data with the affiliates), they’d have foreseen the future: the folks who used to watch NBC’s 10 pm programming are now watching something else at 10, and they’re not turning the channel back at 11, or 11:30, or, in some cases, maybe not at all. I really think promos are critical, especially in a world with hundreds of channels and no one reading TV Guide. If I’m not watching your 10 pm show on Wednesday, I don’t know what you’ve got going on in your 8 pm sitcoms on Thursday. If I don’t watch your 10 pm show on Friday, I might not have any idea that you’re airing the wildcard NFL games on Saturday. (Seriously, who knew the Jets/Bengals and Dallas/Philadelphia games were on NBC last week? Complete surprise in my house.) When I’m watching less on your network in general, I’m not seeing you promote your new shows, or new episodes of existing ones — I’m getting nothing, and you, my soon-to-be-owned-by-Comcast friends: you are in a death spiral.

It’s worth noting this wasn’t solely Leno-related: local news itself has been suffering greatly in recent years, and losing its lead in was sure to kill it (at least temporarily) in some markets. I touched on this briefly near the end of Part I. Talk about your market research failures: each market has three or more stations all bringing you the same news you can’t use: the same overhyped “team coverage” of the same piddly winter storm, the same drug-related shooting  nowhere near your home or office, the same fire at the same house with the same interviews with the same neighbors, and all of them in a race to cut costs by dumping whatever talent they still have with a connection to the community or an ability to actually report. Who needs it?

So, how does this all end? Looks like it’ll be with Conan leaving the network and Leno back on at 11:30, unfortunately. In a fair world, it would end with Jeff Zucker and Jay Leno both out of jobs, but that strikes me as unlikely, especially considering how much NBC enjoys mediocrity these days. Could they have avoided all of this with some smartly designed research? Truly, probably not. I think they were hell-bent on this path and wouldn’t have paid much attention to any data contradicting what they already decided they “knew” — that America loves Leno and that he’d be a surefire hit at 10. Which, come to think of it, is another failure of imagination — the inability to imagine that you might just be wrong.

The Market Research Failure Behind NBC’s Jay Leno Debacle: Part I

We spend a lot of time here talking about the quality of the research we all see being conducted, and about the respondent experience, and about how it may be difficult to trust the data some of the panel vendors are getting back because of their problems in those areas — if only strange, unrepresentative people are willing to take your survey, how representative can it be of the population as a whole?

We don’t, however, spend very much time looking at the other problems market researchers can run into, and I think the NBC television network ran into a really big one of those when they ran whatever research they ran on moving Jay Leno’s show to 10 pm.  (Some may question if they ran any research at all, but they seem to have accurately predicted the ratings he’d receive, so let’s be nice and assume they did.) It’s really a simple problem — a failure of imagination, basically — but to understand what I mean, it’s going to be necessary to understand how the American television broadcast system operates. I’m not sure how well this is understood outside America, or, for that matter, within the country, either. That’s what I’m going to write about today: the underlying system and the reason the Leno experiment failed. The next post will explain the market research implications.

Some of the explanation that follows is going to be too simplistic for some of you: I’ve tried to write an explanation that would also explain the American TV business for some of my non-US readers, who may have a different system where they’re from. I think NBC’s failure is hard to understand if you don’t understand the network/affiliate relationship, though, so here it is. If you get bored, scroll down a bit.

The United States has been divided into something like 200 local TV markets. A market is essentially the area surrounding a particular city. Some are geographically huge, others geographically tiny, but in general, it’s the area that looks to a particular major city. Within each market, there are a number of local TV stations competing with each other. Some are affiliated with networks, others are independent; for this discussion, we can ignore the independent ones. Some stations are owned by the networks themselves; most are not. The three major broadcast networks are ABC, CBS, and NBC; Fox is the fourth network, and because it doesn’t program as many hours in the day as the others, we’re going to ignore it as well. In most of the markets, each network has an affiliated station. Each of them (like all TV stations) makes most of their money by selling advertisements on the programming they air. Now, some of that programming is stuff they produce themselves — the local newscast, for instance. Other programming is syndicated and purchased from distributors — the Oprah Winfrey show would be an example. Finally, the network itself provides a lot of programming, mixed in with the station’s own content throughout the day — network morning news might be followed by a locally produced or syndicated talk show and a local noon newscast; the network then might provide some afternoon soap operas before more local syndicated fare and local evening news. Then comes the national network newscast, followed by an hour of local local time often used for syndicated game shows and celebrity “news” shows, and finally, the important part of our story: evening “prime time” entertainment shows, followed by the local 11 pm newscast, and of course, starting at 11:35 pm, the late night shows.

During the programming the local station provides, they sell most of the commercial slots themselves; during the network shows, it’s the other way around, with the network selling the time to national advertisers and the station getting a small percentage of those commercial slots to sell to local advertisers. (This is why you’ll see a terrible commercial for a local used car dealer in the middle of your favorite network program.) In days gone by, the networks paid the affiliates to air their programs; these days, those payments are stopping, and in some cases, reversing, with the stations now paying the network for the privilege of airing their programs. The prices the stations get to charge for the advertisements they sell time for depend entirely on the number of people watching the show, as determined by the Nielsen ratings. If a show gets a lot of viewers, the commercials cost a lot of money. Important to know: some of these commercial breaks also include promotional spots for the local upcoming programming — during a 10 pm prime time network drama, for instance, you might hear something like “After CSI, stay tuned for Channel 3 Eyewitness News! Don’t miss tonight’s top story — we’ll tell you which local restaurant is killing its customers with tainted food!” You’ll also see network promos — “After your late local news, don’t miss David Letterman, with tonight’s guest former President Clinton!”

So: in a perfect world, the network provides very popular programming that gets people to watch their affiliated stations and see all those promos, which produces great ratings for everything, and as a result, everyone makes money. The 11 pm news has long been a cash cow for local affiliates, who benefited from a strong network prime time lineup and kept those viewers for their late news, where they sold all the ads and did very, very well. You’d think people would just turn the channel at 11:00 and watch whichever local newscast they liked best, but in reality, many viewers don’t seem to feel that strongly about the shows and are easily swayed to watch one or another based on the promos they see and by simple inertia. All three local affiliates are going to be covering the same upcoming storm and the same local shooting and the same warehouse fire at 11, so why change the channel? (Indeed, why watch any of them, but that’s another post for another blog.) In any case, this is the root of NBC’s problem: putting Leno on at 10 destroyed their prime time lineup, which destroyed 11 pm news ratings, which removed more ad revenue from the stations, which are already paying to air this junk in the first place. Angry affiliates do not make for a happy network.

(How angry? This study that I just saw now, right after this post initially went live, says affiliates lost 1/4 of their late news audience (“in a key advertising demo”) from November 2008 to November 2009, and another 3 months of Leno would have cost the affiliates another $22 million in lost ad revenues. Ouch.)

Next time: how NBC’s failure of imagination led to this mess, and a look at the questions I bet they didn’t ask their respondents.

Why Not Use Subway Turnstiles?

What does it end up costing every time they have to shut down an airport terminal, cancel all the flights, and re-screen all the passengers because some dumbass went in through the out door?

What do these subway exit turnstiles cost to install? They’re in every subway station in the free world, pretty much, so they can’t be that outrageous.

Just saying.

(Online research still sucks. How much more can I say about that?)

How Not To Link

This has nothing to do with research, and I’ve probably complained about this before, but it really aggravates me every time I see it:

THAT IS NOT HOW THE INTERNET WORKS, NEW YORK TIMES. CUT IT OUT.

Possibly not obvious from the screengrab, especially because the arrow gets left out: if you click the URL in the story, you go to a New York Times page that uselessly lists all their articles about Facebook. Just in case anyone from the world of old media is reading this and wants to know what they should have done: LINK TO MS. SALAHI’S FACEBOOK PAGE.

God, how is this still so difficult?