No Way Do Two Thirds of Americans Have HDTV. No Way.

Sorry, but I’m willing to bet this piece of research is completely wrong. I’d want to see the actual questionnaire, but here’s what I’ll assume until then:

Many, many, many people have no idea whether or not they have HDTV. Two main reasons:

1. There is a serious lack of understanding among non-techie respondents about the terms “digital,” “high definition,” and “HDTV.” I’ll bet $20 that at least 20% of the population thinks they have HDTV because they bought a $40 conversion box for the digital transition.
2. Because for years now, everything from network dramas to local newscasts has been opening with an onscreen logo that says something like “in HD where available,” or “presented in HD,” just like they used to do the exact same thing for stereo … only now they’ve also gone and incorporated it right into their station logos.

That’s right. Viewers with old 4×3 standard definition TV sets are constantly shown on-screen graphics that, in combination with the fact they bought conversion boxes, has them convinced they’re watching HDTV:

“Of course I have HDTV! It says HDTV right there on the screen!”

It’s difficult to research a topic when respondent confusion is this widespread. It’s not completely impossible, but it’s really, really hard. I can think of a couple of ways to try to do it, but they’re so cumbersome (as in, “look behind your TV and tell me the model number”) that they’re just not going to work.

Oh, and let’s not forget that there’s also God-only-knows how many people — this would include many of our parents, I’ll wager — who have HDTV sets but are watching standard definition broadcasts on them.

Just Say No Already.

Annie Pettit this morning tweeted from the Net Gain 4.0 Conference in Toronto:

Clients still want 1 hour surveys and we can’t do anything about it : I say turn it down!!

I’ll go further than that: I say turn it down and make it clear to the client that they are the cancer that is killing market research. What in the world can you learn from a sixty minute survey that you can’t learn from a 5-minute one? (I’m not talking about an in-depth qualitative research project, or something where you hook someone up to an EEG and have them watch an episode of CSI: Miami to see what their brain has to say. I’m talking about asking questions, on the phone or on a screen. 60 minutes is 55 minutes too long!

Do we really think the respondents still on the phone (or on the web) at the one-minute mark, the ten-minute mark, and the 60-minute mark are identical?

Joel on Research and the Paradox of Choice

Joel Spolsky runs a software company and writes a blog called Joel on Software, but I often find what he writes is applicable to business in general, and occasionally to the research world as well. Recently, in his Inc.com magazine column, he wrote about the problems that develop when too many people are involved in a decision making process — in this particular case, he mentions how a former Microsoft developer tells how designing the Windows Vista shutdown menu took something like 43 people something like a full year and produced a menu with nine near-identical options. The developer calls it The Windows Shutdown Crapfest. The lesson there is obvious — too many cooks spoil the soup — and relevant, I think, to our work, with plain-as-day solutions — trim your meeting invite list and get extra hands out of the work — but dig a little deeper for a more important lesson.

Each of those links in the above paragraph is worth perusing, but the most worthwhile link I’ll have for you today is this, Joel’s original 2006 post on this topic, which does a great job of explaining the resulting mess in terms we all should be able to understand:

The fact that you have to choose between nine different ways of turning off your computer every time just on the start menu, not to mention the choice of hitting the physical on/off button or closing the laptop lid, produces just a little bit of unhappiness every time.

How do we expect our respondents feel when we ask them to tell us if they are Extremely Satisfied, Very Satisfied, Somewhat Satisfied, Neither Satisfied Nor Satisfied, Somewhat Unsatisfied, Very Unsatisfied, or Extremely Unsatisfied with something? What about when we ask them that about a whole page worth of somethings? And what about when some percentage of the questions — anywhere from 1/8 to 1/4 in my rough estimate — don’t apply to the respondent at all? I’d argue we create more than “just a little bit of unhappiness every time.”

The lesson is the same as it so often is here: keep it simple. Satisfied/Unsatisfied/Not Sure should be perfectly sufficient in mot cases, and has the advantage of making the results much more comprehensible at a glance. When comparing results across multiple questions, or across a wide time line of tracking data, it’s infinitely easier to comprehend a single number. The presidential approval number is generally reported as a single figure: 55% in this poll, 48% in this other poll, 53% a month ago, 58% today, etc. Instantly understandable by everyone, as opposed to something like this:

Today, 23% strongly approve of the President’s job performance; 27% say they somewhat approve. Two weeks ago, 29% strongly approved; 17% somewhat approved.

How do you parse that? Strong approval is up down 6 points at the same time that mild approval is up 10 points; overall, if you add the “strong” and “somewhat” numbers together, you can see that overall approval is up four points, but what do you do with those shifts in the gradated responses? Well, if you’re the nightly news — and I’m not suggesting that’s who we should be emulating, necessarily — but if you’re the nightly news, you ignore it and report it as a four point climb. (Well, depending on your viewpoint, you might say the nightly news would be most likely to point out the six point drop in “one measure of the President’s approval rating” and leave it at that, and I don’t think you’d necessarily be wrong about that observation, so.) If you’d only asked the question as approve/disapprove, though, you’d give respondents a simpler experience, and you’d give those interpreting the results both an easier time of it and less wiggle room for those with an agenda.

Let’s see what happens if we offer fewer choices. You really don’t need nine ways to turn off the computer, or seven ways to tell us how satisfied or unsatisfied you are. Honest.

The Market Research Failure Behind NBC’s Jay Leno Debacle: Part II

Welcome back for the second part of this post.

NBC clearly  had a number of problems to solve. Lackluster ratings for the programs they were airing in prime time was one; the fact they’d promised the Tonight Show to Conan to keep him from leaving the network was another, as they’d have to pay him something like $40 million if they broke that contract; the fact Leno didn’t want to retire was a third. So, brilliant plan on NBC’s part: drop all the expensive 10 pm dramas that cost maybe $3 million per episode and replace them with the Leno show, for something like $2 million per week. Save a fortune in production costs, save $40 million in Conan payouts, and “manage for the margins,” as everyone reports NBC’s Jeff Zucker, pictured at right, planned to do.

Here’s where the research finally comes in: NBC apparently did the math, figured out what sort of ratings Leno would need at 10 pm in order to make the show profitable for the network via ad sales, and conducted viewer research to make sure enough people would watch the show to produce that rating. I’m sure that consisted of asking people variations of “what would you watch if all of the following programs were on at 10 pm,” and running down each night of the week, with Leno inserted in place of each NBC drama that was then airing. The results of that research seem to have been pretty accurate; the network was promising advertisers ratings that were pretty similar to what Leno was actually getting. They were embarrassingly low, I think, but they were around the base level that NBC had promised. The failure of imagination is that they just don’t appear to have asked much in the way of logical follow-ups of the people who said they wouldn’t be watching Jay at 10 — i.e., the vast majority of people. Questions like:

• Do you watch any NBC shows at 10 pm right now?
• After you watch one of them, what do you watch at 11? And then at 11:30?
• How often do you find out about one program by seeing a promo during another?
• If your favorite NBC show no longer existed, what do you think you would watch at 10 if these were your remaining choices? And then at 11? And 11:30?
• When you watch an 11:00 newscast, do you always watch the same station? Or do you watch different stations at different times?
• Do you ever decide to watch an 11:00 newscast because of a promo you saw earlier in the evening?
• Do you ever decide to watch an 11:30 or 12:30 late night show because of a promo you saw earlier in the evening?

(You can make an argument that they should have been asking a whole series of other questions as well,  like “Is Jay Leno even a little bit funny,” and “Would you rather watch Jay Leno or that guy singing ‘Pants on the Ground’,” or “What the hell were we thinking when we gave the Tonight Show to Leno and not Letterman in the first place all those years ago,” and “Hey what’s that over there oh my god look out it’s Johnny Carson’s ghost and he’s coming at you with a crowbar,” but that’s probably inappropriate here.)

I think if they’d thought to ask those questions, and had the imagination to work through what the answers met (and shared the data with the affiliates), they’d have foreseen the future: the folks who used to watch NBC’s 10 pm programming are now watching something else at 10, and they’re not turning the channel back at 11, or 11:30, or, in some cases, maybe not at all. I really think promos are critical, especially in a world with hundreds of channels and no one reading TV Guide. If I’m not watching your 10 pm show on Wednesday, I don’t know what you’ve got going on in your 8 pm sitcoms on Thursday. If I don’t watch your 10 pm show on Friday, I might not have any idea that you’re airing the wildcard NFL games on Saturday. (Seriously, who knew the Jets/Bengals and Dallas/Philadelphia games were on NBC last week? Complete surprise in my house.) When I’m watching less on your network in general, I’m not seeing you promote your new shows, or new episodes of existing ones — I’m getting nothing, and you, my soon-to-be-owned-by-Comcast friends: you are in a death spiral.

It’s worth noting this wasn’t solely Leno-related: local news itself has been suffering greatly in recent years, and losing its lead in was sure to kill it (at least temporarily) in some markets. I touched on this briefly near the end of Part I. Talk about your market research failures: each market has three or more stations all bringing you the same news you can’t use: the same overhyped “team coverage” of the same piddly winter storm, the same drug-related shooting  nowhere near your home or office, the same fire at the same house with the same interviews with the same neighbors, and all of them in a race to cut costs by dumping whatever talent they still have with a connection to the community or an ability to actually report. Who needs it?

So, how does this all end? Looks like it’ll be with Conan leaving the network and Leno back on at 11:30, unfortunately. In a fair world, it would end with Jeff Zucker and Jay Leno both out of jobs, but that strikes me as unlikely, especially considering how much NBC enjoys mediocrity these days. Could they have avoided all of this with some smartly designed research? Truly, probably not. I think they were hell-bent on this path and wouldn’t have paid much attention to any data contradicting what they already decided they “knew” — that America loves Leno and that he’d be a surefire hit at 10. Which, come to think of it, is another failure of imagination — the inability to imagine that you might just be wrong.

The Market Research Failure Behind NBC’s Jay Leno Debacle: Part I

We spend a lot of time here talking about the quality of the research we all see being conducted, and about the respondent experience, and about how it may be difficult to trust the data some of the panel vendors are getting back because of their problems in those areas — if only strange, unrepresentative people are willing to take your survey, how representative can it be of the population as a whole?

We don’t, however, spend very much time looking at the other problems market researchers can run into, and I think the NBC television network ran into a really big one of those when they ran whatever research they ran on moving Jay Leno’s show to 10 pm.  (Some may question if they ran any research at all, but they seem to have accurately predicted the ratings he’d receive, so let’s be nice and assume they did.) It’s really a simple problem — a failure of imagination, basically — but to understand what I mean, it’s going to be necessary to understand how the American television broadcast system operates. I’m not sure how well this is understood outside America, or, for that matter, within the country, either. That’s what I’m going to write about today: the underlying system and the reason the Leno experiment failed. The next post will explain the market research implications.

Some of the explanation that follows is going to be too simplistic for some of you: I’ve tried to write an explanation that would also explain the American TV business for some of my non-US readers, who may have a different system where they’re from. I think NBC’s failure is hard to understand if you don’t understand the network/affiliate relationship, though, so here it is. If you get bored, scroll down a bit.

The United States has been divided into something like 200 local TV markets. A market is essentially the area surrounding a particular city. Some are geographically huge, others geographically tiny, but in general, it’s the area that looks to a particular major city. Within each market, there are a number of local TV stations competing with each other. Some are affiliated with networks, others are independent; for this discussion, we can ignore the independent ones. Some stations are owned by the networks themselves; most are not. The three major broadcast networks are ABC, CBS, and NBC; Fox is the fourth network, and because it doesn’t program as many hours in the day as the others, we’re going to ignore it as well. In most of the markets, each network has an affiliated station. Each of them (like all TV stations) makes most of their money by selling advertisements on the programming they air. Now, some of that programming is stuff they produce themselves — the local newscast, for instance. Other programming is syndicated and purchased from distributors — the Oprah Winfrey show would be an example. Finally, the network itself provides a lot of programming, mixed in with the station’s own content throughout the day — network morning news might be followed by a locally produced or syndicated talk show and a local noon newscast; the network then might provide some afternoon soap operas before more local syndicated fare and local evening news. Then comes the national network newscast, followed by an hour of local local time often used for syndicated game shows and celebrity “news” shows, and finally, the important part of our story: evening “prime time” entertainment shows, followed by the local 11 pm newscast, and of course, starting at 11:35 pm, the late night shows.

During the programming the local station provides, they sell most of the commercial slots themselves; during the network shows, it’s the other way around, with the network selling the time to national advertisers and the station getting a small percentage of those commercial slots to sell to local advertisers. (This is why you’ll see a terrible commercial for a local used car dealer in the middle of your favorite network program.) In days gone by, the networks paid the affiliates to air their programs; these days, those payments are stopping, and in some cases, reversing, with the stations now paying the network for the privilege of airing their programs. The prices the stations get to charge for the advertisements they sell time for depend entirely on the number of people watching the show, as determined by the Nielsen ratings. If a show gets a lot of viewers, the commercials cost a lot of money. Important to know: some of these commercial breaks also include promotional spots for the local upcoming programming — during a 10 pm prime time network drama, for instance, you might hear something like “After CSI, stay tuned for Channel 3 Eyewitness News! Don’t miss tonight’s top story — we’ll tell you which local restaurant is killing its customers with tainted food!” You’ll also see network promos — “After your late local news, don’t miss David Letterman, with tonight’s guest former President Clinton!”

So: in a perfect world, the network provides very popular programming that gets people to watch their affiliated stations and see all those promos, which produces great ratings for everything, and as a result, everyone makes money. The 11 pm news has long been a cash cow for local affiliates, who benefited from a strong network prime time lineup and kept those viewers for their late news, where they sold all the ads and did very, very well. You’d think people would just turn the channel at 11:00 and watch whichever local newscast they liked best, but in reality, many viewers don’t seem to feel that strongly about the shows and are easily swayed to watch one or another based on the promos they see and by simple inertia. All three local affiliates are going to be covering the same upcoming storm and the same local shooting and the same warehouse fire at 11, so why change the channel? (Indeed, why watch any of them, but that’s another post for another blog.) In any case, this is the root of NBC’s problem: putting Leno on at 10 destroyed their prime time lineup, which destroyed 11 pm news ratings, which removed more ad revenue from the stations, which are already paying to air this junk in the first place. Angry affiliates do not make for a happy network.

(How angry? This study that I just saw now, right after this post initially went live, says affiliates lost 1/4 of their late news audience (“in a key advertising demo”) from November 2008 to November 2009, and another 3 months of Leno would have cost the affiliates another $22 million in lost ad revenues. Ouch.)

Next time: how NBC’s failure of imagination led to this mess, and a look at the questions I bet they didn’t ask their respondents.

How Many Yards Do You Commute To Work, And Other Badly-Measured Intervals.

I’m really sorry I’ve been so dormant lately.  I don’t really have an excuse, other than that I’ve been busy enough with other things that I haven’t been taking many online surveys, and as a result, I haven’t had anything to post.

Today, though, that changes. Hopefully for good? We’ll see.

So I watched an episode of How I Met Your Mother at cbs.com just now, and following it, they gave me a survey from Magid about my use of streaming video, peer-to-peer sharing, and so on. I’ve actually been getting a lot of TV via the internet lately — there’s just too much on at the same time on Thursdays, and I’ve been forced to torrent or use Hulu to watch at least some of it, since my DVR can only do two things at a time, and there seem to be THREE things on simultaneously from 8:00 to 10:00 those nights. Some weeks I grab torrents, others I use Hulu — it mostly depends on when I’ll be watching, because I have kids, and I find it much easier to watch TV with closed captions when they’re around, since they’re noisy little things. If I’ll be watching when they’re home, I often use Hulu; if they’re out or asleep, I’ll often get the torrents, which are usually better quality, and are usually able to be streamed to my TV, too.

Anyway, the point here is to share this incredibly ill-conceived question, which was the one really badly thought-out item in an otherwise pretty solid survey:

Really? You want me to think about how much TV I watch in 15-minute increments? Why on earth would you think this was the right way to ask this question? I had to do MATH to answer the question, counting up the number of hours of TV I watch and multiplying by 4, which might not even be an obvious option to every respondent. The strangest thing is, the 15-minute increment makes no sense in either context. Online versions of TV shows aren’t ever in 15 minute formats — half hour sitcoms run around 22 minutes, and hour dramas are around 44 — and the other things people watch online, like movie trailers and clips of people being idiots on YouTube are much shorter.

I don’t get it. Which I suppose isn’t unusual.

Straightlining vs. Answering Your Stupid Question Honestly

OK, this is something I hadn’t thought of before.

When I’m staring at a bad survey question — asking me to compare two absolutely identical companies in a matrix, for instance — my tendency is to do this:

They’re equal. There’s no difference between Visa and MasterCard in my mind. Discover and American Express, those are different, both from one another and from these two brands, but Visa and MasterCard might as well just merge, as far as I’m concerned. Of course, there’s no way to provide that answer in the framework provided here, so I decided to simply give each company a score of “5″ for each item. That seemed to get the message across, as far as I was concerned. Of course, as soon as I clicked the button, I got booted, with the same generic non-qualified message you get when you tell them you don’t have kids or haven’t seen a movie in the past two months or whatever it is. We all know the truth: they booted me for straightlining.

Which I wasn’t.

At the very least, wouldn’t it be smarter to keep me in and see what the rest of my answers looked like? With the amount of amply-documented badly designed questionnaires out there, shouldn’t we maybe consider that a respondent will occasionally need to do something to get around a poorly framed question, or an item that simply doesn’t apply to them?

Simply ending the survey as soon as someone gives all items on a page the same value seems both too simplistic and too drastic a solution to me.

Another Fine Matrix

First, look at this full-size. See how there are 14 brands of cat food going across the top? I already told it I’d never heard of five of them, and yet here they all are again. It’s one thing to ask me if I’ve ever heard of a brand and to then, even if I haven’t, show me an ad for the brand and ask if I’ve seen that ad — I very well could have forgotten about it, or misremembered what brand it was for.

This is just stupid…

Worse, though, it’s endless. Here’s a reduced-size capture so you can see how long it is:

This is what I’m referring to in the comments on Gary Langer’ post here — what the hell sort of non-representative person is going to sit through this? This kind of garbage really is the cancer that is killing market research. Stop pulling this crap, and then go and worry some more about probability samples.

Needs Moar Choices.

Seriously? Shouldn’t they also have broken out high school by year, or something? Maybe included a radio button for each individual year from kindergarten through law school? No, really, I just can’t imagine how such fine distinctions are useful to anyone. Is someone really looking at this and saying, “Wow, the 7 respondents with some advanced degree work are slightly more likely to say x than the 11 respondents who are currently in advanced degree work! Fascinating! Oh, wait, the margin of error is +/- 37.8%.”

I get that there’s value in collecting more, not less data; I’m a firm believer in asking respondents for their actual ages, actually, instead of for a range — because when you have the actual data, you can put it back together in whatever groupings you want, which may not be the groupings you think make sense before you see the results — but this here is just a mess.

Is the panel research business model creating a gold farming problem?

Greenfield must be having trouble getting panelists to complete research these days. Maybe it’s the summer blahs, with respondents too hot, too sweaty, or just too on vacation to be bothered.

Then again, maybe it’s something to do with people just getting sick of trying to imagine their orange juice has come to life and is displaying personality traits.

In any case, I’m sure this is the answer:

50 cents per survey! At 20 minutes per survey, that’s like, $1.50 an hour! This will totally solve all of Greenfield’s problems, and can only lead to amazing data quality.

Right?

But let’s turn this repetitive Greenfield mockery into a real question:  what are the odds that this sort of incentive (and incentives in general, really) has already led to or soon will lead to the market research version of “gold farming?”

Gold farming, if you don’t want to bother reading the Wikipedia entry, is an exploit carried out within massive online role-playing games, like World of Warcraft. I’m no expert in it, but as I understand it, people hire low-wage workers (this has apparently been an issue in China) to sit in front of multiple computer terminals logged into the online game. The workers don’t actually play the game as it’s intended to be used, but they instead perform repetitive actions, generally using automated scripts, to earn (or, colloquially, to farm for) in-game cash — virtual money, essentially, that can be spent on in-game items like better weapons and the like. The folks behind the operation then sell the virtual currency online, to actual players of the game who want to buy a really cool sword or whatever but who can’t be bothered spending weeks building up the in-game cash to buy it.

So, since Greenfield is paying 50 cents for 20 minutes worth of human labor here, it occurs to me that someone has probably already figured out that they write some scripts to blast through these things in (let’s say) five minutes each — 12 per hour, as opposed to 3 per hour. And that’s per computer. So you sit a guy in front of five screens, each logged in on a different Greenfield account, each earning $6 an hour — so $30 an hour across those five screens — you know, if your labor only needs to make around $3 an hour, that’s $216 a day in pure profit for the guy in charge. And that’s assuming he’s only got one guy doing this on only five accounts at once.

Now, I’m sure I can’t be the first person this sort of thing has occurred to, and I’m sure Greenfield and the other panel outfits are trying their hardest to make this impossible, limiting the number of surveys one respondent can complete in a day, maybe checking for a total elapsed time and invalidating surveys that move too quickly — but, I don’t know, that strikes me as sort of being similar to making the roads near the bars really wide and straight instead of outlawing 24-hour happy hours, or some similarly goofy comparison.

If we want honest answers from real people, maybe we should rethink this entire insulting “we’ll pay you fifty cents to answer 120 repetitive questions about the minute differences between four brands of orange juice” business model.