Fun With Crosstabs

A couple of thought’s on PPP’s bombshell last week about interracial marriage in Mississippi, which, if you missed it, said that 46% of “usual Republican Primary voters” there say interracial marriage should be against the law (full PDF of results here):

First, at the very outset, this: I like PPP quite a lot. They do solid work. They release more data on their polls than many of teir competitors. They are a Democratic pollster, but if you look at the data Nate Silver puts together for FiveThirtyEight, PPP’s results are both pretty accurate and pretty unbiased. Yes, one can argue that being partisan gives them an agenda, and while I’m sure they’re not sad to have uncovered this data point, at the end of the day, a pollster is only as good as his numbers, and PPP’s numbers are good. If anything, I’d say their partisan leaning gives them some “cover” when they want to ask something controversial, like they did here. Actual question wording: “Do you think interracial marriage should be legal or illegal?” I think a good number of pollsters would have difficulty asking that question; it seems, to me at least, to be too likely to cause offense. More on that in a minute, though.

Needless to say, the fact that 46% of their respondents said “illegal” made a lot of news last week. PPP says they asked the question of Democrats (and I assume independents) as well, and will be releasing that part of the data shortly, and I’m sure that will make some news, too. I’m not naive enough to believe that racism is dead, and it’s worth noting that while the US Supreme Court struck down state laws banning interracial marriages back in 1967, neighboring Alabama didn’t get around to removing the anti-miscegenation bit from its state constitution until the year 2000, but I still think these numbers are surprisingly high. These are findings that produce coverage because people talk about them; in this case, a lot of sad head-shaking mixed with gleeful potshots from one side, and a lot of unoriginal pollster attacks from the other, mostly of the usual tired “but it’s IVR, it could have been an 8-year-old taking the poll” sort. (Because it would be a better or easier-to-stomach finding if 46% of Mississippi 8-year-olds felt this way?)

Still, I think there are about three perfectly reasonable reasons to question these results, so let’s get to it:

First, there’s this: “legal” and “illegal” sound really similar on the phone, no matter how carefully the announcer (or the live interviewer) says the words. Everyone does it, because workaround wording is cumbersome and ends up being something like “should so-and-so be allowed by law? Or against the law,” which is problematic because people generally just don’t say “allowed by law,” so you end up causing some confusion there, as well. Instead, it might be better, somewhat illogically, to go with something even longer, like “should Mississippi allow interracial couples to get married? Or should interracial marriage be against the law?” It’s impossible to mis-hear that, except for the next thing:

There isn’t (or, there wasn’t until this poll was released) any real debate in this country right now about interracial marriage. There is, however, one about same-sex marriage. I think it’s very possible that some percentage of respondents simply misunderstood the question. This was raised, to an extent, in the comments on the original PPP post on this poll, and I think PPP’s response might have been a little too hasty, though I understand where they’re coming from. To be very, very clear: I do not think the respondents on this poll, or southerners, or Republicans, or southern Republicans, or any people in general, are stupid. I do not think people are hearing the word “interracial” and failing to understand what it means. I think they’re just not hearing the word at all. Look: people are not taking polls in ideal lab conditions; they’re taking them in real life, in rooms that contain television sets, and children, and the internet, and in some cases they’re taking the polls in their cars on their cell phones, despite the pollster’s best intentions to not accidentally break the law by calling any wireless numbers. No matter how clearly worded and how well-read the question is, the pollster is battling an infinite number of distractions for each respondent’s time, and can’t expect each respondent to be giving them his or her undivided attention. In 2011, in an election poll, when the 14th question you’re asked, coming immediately after several “how would you vote” questions, if you hear “Do you think interracial marriage should be legal or illegal,” I think you’re going to have one of three reactions, and only one of them is good.

You might hear the question correctly and answer it. Spoiler alert: that’s the good one.

You might hear the question incorrectly and answer it. Another spoiler: that’s bad.

You might get so offended that you hang up the phone. This is so bad that it’s the topic of the next paragraph.

If you ask a question that deeply offends some of your respondents, they’re going to hang up on you. That’s going to leave only the non-offended to answer that question. What does that do to your results? If you, out of the blue, put a question in the middle of your Mississippi Republican primary election poll, a question that carries the pretty clear subtext of “are you a racist,” I think there’s a reasonable chance that many non-racist respondents will decide they know where this is headed — to a bad place — and drop out of the poll. I actually think a similar thing may be driving some of the polls showing large numbers of Republicans in the “birther” camp — a lot of non-birthers are hearing that question as “Now I’d like to ask you something to try to make people who share some of your views on politics look really, really stupid. Where do you think Barack Obama was born?” Reasonable Republicans may be dropping out at this point, which not only is inflating the number of Republican birthers, but it’s also having the side effect of inflating the poll numbers for “candidates” like Donald Trump, because, in most cases, if you hang up during the course of a poll, none of your previous answers count. If Romney voters start dropping out of a poll near the tail end, their answers up at the top end of the poll are also going to go in the garbage, and that’s going to give a bump to the fringe respondents’ favorite candidates.

All this is testable, of course, with some a/b sample splitting. I think the hardest thing is making it absolutely clear we’re talking about interracial marriage (as opposed to same-sex marriage), which might require, ironically enough, asking about same-sex marriage first to make it more abundantly clear, in the subsequent interracial marriage questions, that we’re now talking about something else.

I look forward to seeing what data PPP shows for the rest of Mississippi respondents from this poll; offhand I’d expect it to be somewhat less of a large number, but still larger than many will be comfortable with — but I’d take it all with a grain of salt until someone is able to do some testing of my hypothesis here.

Yet Another Example of How Not To Use The Internet To Conduct Research

(edit, April 6, 2011: Over a year since I posted this, and I just took another Zogby poll (now an “Ibope Zogby poll,” by the way), and they’re still asking this question the same way. And I still, despite being pretty politically aware, knowing my congressman’s name, and having even written the guy and gotten a response on at least one occasion, have absolutely no idea what district number I live in. Everything below this parenthetical addition is old content, so if you have seen it before, sorry.)

This is from a couple of weeks ago, and I’m just now getting a chance to post it.

88% of Americans live in a state with fewer than 53 US congressional districts in it. Only California has that many; Texas comes in second with 32.

And yet, here’s how the good folks at Zogby Interactive ask what congressional district you live in:

That’s right. Zogby asks what state you live in, and then asks you, regardless of how many districts your state contains, which of 53 districts you live in. This is terrible for a lot of reasons, beginning with what should be obvious to everyone: it’s really lazy.

Looking at this from a practical political standpoint, though, it’s a mess. Folks just don’t think about their congressional district that way. Many (certainly not all) will know the name of their representative — or at least be able to pick the name from a short list of possibilities — but the odds of them knowing the actual district number aren’t great.

That being said: it can be problematic to ask people who their representative is if you’re then going to ask them if they’re going to vote for that person — it creates a priming effect and reminds (or informs, if the respondent is less focused on politics) of incumbency and makes it difficult to do a clean “would you vote for x or y” question. While I didn’t get that question as a follow-up, it’s possible some respondents did, though I somewhat doubt it this far out.

A much better way to ask this question is to ask for zip code, which will let you look up the right district in most cases; a simpler method (for the respondent), and one that might feel less personally intrusive, is to remember that this is the internet and present a state map, on which the respondent can zoom in and actually CLICK WHERE HE LIVES.

And, saying what should be obvious, but maybe isn’t: if you structure your research in such a way that only the very very very top-most super-engaged respondents are qualified to answer a follow-up, your results are only going to reflect that tiny slice of the population.

Pathetic, and sadly, about what one would expect.

Gary Langer Hates Robopolls, Probably Because They’re More Accurate Than His Live Interviewer Polls And He’s Apparently Just Jealous, The Poor Thing, So He’s Decided That Accuracy Doesn’t Matter.

Gary Langer, Director of Polling at ABC News, has a problem with what he calls “robopolls” and what we’re going to call interactive voice response, or IVR, polls.

As best I can tell, his root problem is that they’re more accurate than his live interviewer polls, despite using what he considers inferior methodology. But here, let him tell you:

In our ABC News polling standards we don’t regard autodialed, pre-recorded polls as valid and reliable survey research. Our concerns include noncoverage, nonresponse, lack of respondent selection, nonvalidation, opaque weights and the absence of a persuasive, independent literature evaluating the approach. Some other news shops – the few that, in my view, take coverage of public opinion seriously – share these judgments.

There’s the germ of a good point or two in there, to be honest with you, at least on the surface, but let’s go through all of them and poke some holes:

• Noncoverage: It’s illegal to call a cell phone with an autodialer, so by default, a poll conducted by an IVR firm probably doesn’t include any cell phone interviews, so yes, cell phone only households wouldn’t be covered in the sample — but, first, there’s nothing keeping an IVR firm from hiring a data collection house to make cell phone calls with live interviewers to supplement their IVR polling — IVR is a method of conducting polls, not an ideology — and second, as it happens, there’s a bit of an intersect between the people most likely to live in a cell phone only household and the people least likely to vote. Not that Langer cares about results, but the studies I’ve seen have suggested that so far, this isn’t a factor.
• Nonresponse: I suppose response rates could be lower for recorded voice polls than live voice polls, but they could also be higher. I don’t know, and Langer doesn’t know either: there are a number of different ways to calculate response rates, and few pollsters release them at all, and when they DO release them, they all seem to pick a different measure. See page 21 (table 5) of this AAPOR report on 2008 primary polls — there isn’t a single contest there where you can compare an IVR and a live interview pollster’s response rate because none of the pollsters seem to agree on which measure to use!  (There’s also a body of literature suggesting there’s really not much correlation between response rate and accuracy, but Gary doesn’t care about accuracy, just methods, so I guess we won’t bother linking to any of that. Or not.)
• “Lack of respondent selection” is a major bullshit non-issue, brought up time and time again by Langer and his ilk to discredit research that threatens their world. According to them, your poll is unreliable if you don’t start every call by asking to speak to a specific random person — the youngest male, for instance, or the person in the household with the next birthday. There’s absolutely no reason why “robopolls” can’t do this — it’s just another question, for crying out loud — and as far as I know, there’s no evidence that there’s any reason for them to do it, or for anyone else to, other than because Langer says they should.
• “Nonvalidation.” I have no idea, but assume this means Langer thinks good polls employ extra live interviewers to call back selected respondents to make sure the original live interviewers actually did contact the respondents and that the original live interviewers didn’t just make up the data so they could collect their $25 or whatever and go home. Hmm. Gosh, Gary, why would it possibly be the case that IVR polls don’t bother doing this? Maybe because they don’t have a room full of telemarketers falsifying data so they can go home early? (And, once again: why couldn’t IVR pollsters do validation? If it mattered, is there some reason they couldn’t make those callbacks, either through an IVR or a contracted live interviewer methodology? Couldn’t the office manager make the calls the next day? You only “need” to validate a sampling of the calls, right?)
• “Opaque weights” is another term that doesn’t mean a lot to me. Apparently Gary thinks the IVR pollsters are more secretive about how they weight their data than the live interview pollsters. No idea why that would be the case, no idea what evidence there is suggesting it is.
• The absence of literature is one of my other favorites. Yeah, Gary, there’s not a lot of peer-reviewed scholarly articles about IVR polls, all of which happen to be run as businesses (as opposed to academic projects) because, unlike academics, business people don’t usually write peer-reviewed scholarly articles.

What else you got, Gary? Oh, right:

Some defenders of robopolls will point in another direction, at those that have accurately predicted election outcomes. From my perspective, accurate modeling is not an adequate stand-in for good polling. For reliability and validity alike, what matters are sound methods. And what matters next are substantive measurements, not horse-race bingo.

Wait, sorry, I couldn’t hear you, I was laughing too hard. What? Is he seriously saying that it just doesn’t matter how accurate a poll is, it only matters if the pollster follows the right rules? The Gary Langer rules? Oh, Gary.

Look: twenty years ago, if you wanted to take a public opinion poll, you needed, I don’t even know, a Ph.D., a university, and I’m going to guess something like $40,000.  Today, you can get a PPP or a SurveyUSA or a Rasmussen to do it for you for maybe 10% of that, or you can just go do it yourself for a few hundred dollars. I get that this is scary as hell if you came up through the academic ranks; I get that irrelevancy really  pisses people off — but I just don’t think you have a lot of credibility when you try to tell us that whether or not you get elections right isn’t a good measure of your skill as an election pollster. From where I sit, it’s the only thing.

(PS: PPP, I love you guys, but I think he’s right about one thing: an “or not” wording choice would probably be better.)

tl; dr: threatened much?

ETA: Forgot to add this, which is Nate Silver’s last (and horribly outdated, from May of 2008) pollster ranking chart. Still, it’s the last thing he gave us:

Look on the bright side, Gary: you’re better than Zogby.

What are the Republicans Thinking?

For reasons completely beyond me, I’m on the Republican National Committee’s junk mail list. This morning, I received the strangely bad survey they’ve been taking to try to figue out what they’re doing wrong and what they can do to fix it.

Partisan feelings aside (for the moment), this isn’t a horrible idea; there’s nothing wrong with trying to survey your membership, or failing that, survey people you’ve misidentified as your membership (my hunch is that this dates back to my signing up for McCain’s email list back in 2004 when we all thought he was going to switch parties, but who knows). The problem, of course, is simply that the execution is possibly among the worst I’ve seen. You’d think they’d have asked a Republican pollster to come up with a professionally-written instrument, but thankfully, they didn’t: let’s make fun of it.

Random capitalization aside (more on that shortly), what the heck is up with the “Need to Lead in Congress” choice? If they’re trying to list perceived weaknesses, wouldn’t it need to be constructed differently, like “failure to lead in Congress,” or “lack of leadership in Congress,” or other words to that effect? “Standing up for Principles” isn’t much better — as worded, it’s impossible to tell if the poll is suggesting that Republicans are weak as a party because they stand up for their principles (which aren’t popular enough to help them win elextions) or if they’re saying Republicans are weak as a party because they don’t stand up for their principles enough. Either interepetation could make sense, but the pollster apparently wants us to figure it out for ourselves.

Also, take a look at the preceeding question, which asked what the party’s biggest strengths were:

I’m certain this would be an unpopular view at GOP Headquarters, but couldn’t the case be made that every single one of those items should also be in the weakness box? Isn’t it possible there are some actual Republicans who believe the party’s focus on “values” issues like gay marriage turned off moderate voters who might have been receptive to a message focused solely on fiscal responsibility?

I don’t even know where to begin with this mess, so let’s just leave it here as an example of what a biased poll question looks like.

Why are Those Things Capitalized In That Way?

Two separate problems here, and they’re both fairly common. First, obviously, the “scare quotes,” the use of “wasteful” to modify “government spending,” and the hilarious “Pelosi-Reid Democrats” label — is that supposed to be like “Reagan Democrats” or something? Who identifies with a Speaker of the House/Senate Majority Leader enough that it works as a label? Whatever. The other problem here is very common, and I see it all the time, and I think most people miss it: they’re setting up a question where the respondent has to choose a positive answer choice — “yes,” in this example — if they want to vote for a negative answer choice, “oppose.” This creates just enough of a moment of confusion in the respondent’s mind that it should be avoided, especially because there’s nearly ever a need for it. “Should Republicans in Congress support or oppose the new blah blah blah,” with “support” and “oppose” as the two answer choices — that’s much simpler and avoids that dangerous construction entirely.

This just amuses me — shouldn’t it be “instead of you and your doctor and the non-doctors who make the decisions at your insurance company?”

This was my favorite part, though. (It’s actually the first item in the survey, but I saved it for last.) The email address field was pre filled-in, by the way, and my email address was also in the survey’s URL. That’s one way to make sure no one fills in the “other” blanks with anything you don’t want to hear, I guess.

Pathetic, really.

A Quick Thought: Where Are We Headed?

Just a quick thought:

Where is the industry headed, not ten or even five years down the road, but three years down the road?

When we start taking serious, state-level polls on the 2012 presidential election, will we be doing it mostly still on the telephone? Will political research have moved to the web? Will market research move even more to the web than it already has?

I have a lot of thoughts on this, which I’ll post in due course; I’m curious as to whether anyone else would like to weigh in before I do.

Why We Fight

I think I need the occasional reminder that the point of this isn’t so much to point and laugh at Greenfield and other worthy targets, but instead that bad research needs to be eliminated, because it actually hurts us all.

What’s the long-term effect of Greenfield getting respondents to take survey after survey, in a never-ending chain of sweepstakes entries?

What’s the long-term effect of Polling Point asking respondents to mark screen after screen of company names as green or red?

What’s the long-term effect of untrained telemarketers asking respondents who they’ll vote for — and mispronouncing the name of each candidate?

What’s the long-term effect of Zogby’s willingness to get in bed with and take research on just about any topic for just about any client?

As the pool of willing cooperators dwindles, how much longer can we rely on the remaining respondents to fill in the gaps?

Short-term, there are ways to compensate, but at some point, we’re going to regret that we let this go on as long as we did.

What the Hell is This About?

I can’t figure out why Zogby is asking this; I’m not aware of either candidate having anything specific to say about either steroids or tort reform, and I’m really pretty sure that if they had spoken about steroids, they’ve have been on the same side of the issue, no?

Weird.

Been Busy…

Pretty sure I mentioned this before:

There is no such thing as a Bradley Effect. There certainly isn’t one now, and there wasn’t one in the Bradley race, either, though there MAY have been one in the Wilder race for governor in VA.

Thank you, Barack Obama, for proving it.

Dear Correspondents:

Before e-mailing the pollster to complain that the pollster is “cooking the books” for candidate so-and-so because the poll includes 15 points more members of Party A than members of Party B, please check the official party registration statistics for the state in question, because it’s probably going to show, if the state in question registers voters by party, that the state has about 15 points more members of party A than members of Party B.

tl; dr: Bite me, whiner.

So, What’s All This About Party Identification in Election Polls?

Can of worms, I hereby declare you open.

OK. If you’re paying attention to the Presidential race, you probably have noticed a few controversies over election polling in the past few weeks. Here’s a short summary:

• RNC holds convention, picks Sarah Palin as VP.
• Various polls show McCain/Palin with pretty large surge.
• Various blogger types complain the polls are “rigged” because they have more Republicans in them then they should, and certainly have more Republicans in them then the previous round(s) of polls.

So, what’s going on? Are the polls being rigged?

No.

Of course not. Look, polls may be a lot of things, including shoddy, sloppy, and/or hopelessly incompetent, but with very rare exceptions — I think there is a pollster or two out there who will put his thumb on the scale if the final poll before the election looks “off” — no one is going to risk their reputation to do something of such dubious value eight weeks out. Yes, the media loves that the race is so close right now, but I think the media would be having just as much fun if Obama was 20 points ahead, and really, probably the same if McCain was. A 20 point lead would be astounding, for either party, and would be representative of a campaign that really caught fire with the public in a way that just doesn’t happen anymore. In short, I think the McCain surge in the polls is largely “real,” and, yes, the polls do have “too many” Republicans in them, but that’s OK, because the polls are reflecting reality at this particular time, and the current reality is that Sarah Palin has invigorated a lot of conservatives who previously were not very happy with John McCain.

Different pollsters ask the party identification question in different ways. On one end of the scale, you can treat party ID as a largely immutable characteristic like gender, age, or race — if you were born in 1950, you’re 58 years old, and that answer isn’t going to change until your next birthday comes around. These pollsters, if polling a state that registers voters by party ID, may be weighting their data or managing their sample in such a way that the respondents in the poll closely match those party registration figures, or, possibly, closely match a different set of figures the pollster has decided will more accurately reflect the electorate this year. These pollsters (again, in the states where you register by party) are asking, essentially, “Are you registered to vote as a Republican, as a Democrat, or as a member of another party?”

On the completely opposite end of the scale, you have those who believe that party ID is a very changeable characteristic, and that it can be envisioned as a 10-point scale, with Democrats all the way at the left, Republicans all the way at the right, and independents dead center. A “1” might be mad at Barack Obama one day (FISA, perhaps?) and say she’s a “3.” A “5” might have liked Palin’s convention speech so much that he’s an “8” today, but maybe he’s a “6” a few days later, a “4.5” when you poll him a week later, and during the course of your too-long poll, where you randomly branch him down a path that asks critical questions about Palin, he decides you’ve gone too far and by the time you ask him his party affiliation, he’s on the other side of the hump, a “5.5” Pollsters who treat party ID as a changeable characteristic, as the way a respondent feels on a given day, are more likely to ask a question along the lines of “Do you generally consider yourself a Democrat, a Republican, an independent, or something else?”

There are polling outfits in the middle, too, who apply some weighting based on their past experience, past polling, whatever. There are a million ways to do it, and no one can say for sure which is right. If your personal worldview is that party affiliation is what your voter registration card says on it, and that the percentage of Democrats and Republicans from poll to poll should rarely or never shift, I’d suggest you look at West Virginia, where voter registration statatistics, as of April (PDF), are 56% Democratic and 29% Republican. Now, both US Senators from West Virginia are Democrats, as are two of its three US House members, and the governor, but I guarantee you this:

Take a random sample of West Virginia registered voters and split it in half. For half, ask them how they are registered, followed by four non-political questions — ask about movies, or college football, or something. The other half, reverse it — questions first, then registration. The results will generally line up with the registration data you’re making calls from, and the two halves of the sample will have pretty similar numbers — about 2:1 Democrats over Republicans, just like in the real registration data.

Now take another random sample, again split in half, and ask the same non-political questions; this time, though, ask the party ID question differently — try asking what respondents generally consider themselves to be. I’m betting that you’re going to get 10 to 15% more Republicans this time, and that the numbers will be similar for the two halves — the fact we’re asking “consider” instead of “registered” will help the Republicans, but there won’t be a difference between those asked the question before and those asked the question after the topical questions of the poll.

Next: ask questions about the 2008 race for Governor. Again, ask what respondents consider themselves. This time, I’m betting the question asked up front will be 10% to 15% more Republican than actual registration, and the question asked after the questions on the guberbatorial election wil be closer to the actual 2:1 figure. Joe Manchin is pretty popular, and after people say they’re going to vote for him, they’re going to consider themselves Democrats, which they technically are.

Finally, swap out the Governor race questions and ask about McCain/Obama. This time, I predict 10 to 15% more Republicans in the affiliation question asked up front, and 20 to 30% more Republicans in the question asked after the horserace question.

If party identification is able to swing so much based simply on where it’s asked in what type of poll, and I’m pretty sure it is, why isn’t it just as likely that political happenings — the introduction of a new face to the campaign, one who is hugely popular with the exact same conservative voters who previously were unenthusiastic about McCain — also change party ID numbers the same way?