Gary Langer, Director of Polling at ABC News, has a problem with what he calls “robopolls” and what we’re going to call interactive voice response, or IVR, polls.
As best I can tell, his root problem is that they’re more accurate than his live interviewer polls, despite using what he considers inferior methodology. But here, let him tell you:
In our ABC News polling standards we don’t regard autodialed, pre-recorded polls as valid and reliable survey research. Our concerns include noncoverage, nonresponse, lack of respondent selection, nonvalidation, opaque weights and the absence of a persuasive, independent literature evaluating the approach. Some other news shops – the few that, in my view, take coverage of public opinion seriously – share these judgments.
There’s the germ of a good point or two in there, to be honest with you, at least on the surface, but let’s go through all of them and poke some holes:
- Noncoverage: It’s illegal to call a cell phone with an autodialer, so by default, a poll conducted by an IVR firm probably doesn’t include any cell phone interviews, so yes, cell phone only households wouldn’t be covered in the sample — but, first, there’s nothing keeping an IVR firm from hiring a data collection house to make cell phone calls with live interviewers to supplement their IVR polling — IVR is a method of conducting polls, not an ideology — and second, as it happens, there’s a bit of an intersect between the people most likely to live in a cell phone only household and the people least likely to vote. Not that Langer cares about results, but the studies I’ve seen have suggested that so far, this isn’t a factor.
- Nonresponse: I suppose response rates could be lower for recorded voice polls than live voice polls, but they could also be higher. I don’t know, and Langer doesn’t know either: there are a number of different ways to calculate response rates, and few pollsters release them at all, and when they DO release them, they all seem to pick a different measure. See page 21 (table 5) of this AAPOR report on 2008 primary polls — there isn’t a single contest there where you can compare an IVR and a live interview pollster’s response rate because none of the pollsters seem to agree on which measure to use! (There’s also a body of literature suggesting there’s really not much correlation between response rate and accuracy, but Gary doesn’t care about accuracy, just methods, so I guess we won’t bother linking to any of that. Or not.)
- “Lack of respondent selection” is a major bullshit non-issue, brought up time and time again by Langer and his ilk to discredit research that threatens their world. According to them, your poll is unreliable if you don’t start every call by asking to speak to a specific random person — the youngest male, for instance, or the person in the household with the next birthday. There’s absolutely no reason why “robopolls” can’t do this — it’s just another question, for crying out loud — and as far as I know, there’s no evidence that there’s any reason for them to do it, or for anyone else to, other than because Langer says they should.
- “Nonvalidation.” I have no idea, but assume this means Langer thinks good polls employ extra live interviewers to call back selected respondents to make sure the original live interviewers actually did contact the respondents and that the original live interviewers didn’t just make up the data so they could collect their $25 or whatever and go home. Hmm. Gosh, Gary, why would it possibly be the case that IVR polls don’t bother doing this? Maybe because they don’t have a room full of telemarketers falsifying data so they can go home early? (And, once again: why couldn’t IVR pollsters do validation? If it mattered, is there some reason they couldn’t make those callbacks, either through an IVR or a contracted live interviewer methodology? Couldn’t the office manager make the calls the next day? You only “need” to validate a sampling of the calls, right?)
- “Opaque weights” is another term that doesn’t mean a lot to me. Apparently Gary thinks the IVR pollsters are more secretive about how they weight their data than the live interview pollsters. No idea why that would be the case, no idea what evidence there is suggesting it is.
- The absence of literature is one of my other favorites. Yeah, Gary, there’s not a lot of peer-reviewed scholarly articles about IVR polls, all of which happen to be run as businesses (as opposed to academic projects) because, unlike academics, business people don’t usually write peer-reviewed scholarly articles.
What else you got, Gary? Oh, right:
Some defenders of robopolls will point in another direction, at those that have accurately predicted election outcomes. From my perspective, accurate modeling is not an adequate stand-in for good polling. For reliability and validity alike, what matters are sound methods. And what matters next are substantive measurements, not horse-race bingo.
Wait, sorry, I couldn’t hear you, I was laughing too hard. What? Is he seriously saying that it just doesn’t matter how accurate a poll is, it only matters if the pollster follows the right rules? The Gary Langer rules? Oh, Gary.
Look: twenty years ago, if you wanted to take a public opinion poll, you needed, I don’t even know, a Ph.D., a university, and I’m going to guess something like $40,000. Today, you can get a PPP or a SurveyUSA or a Rasmussen to do it for you for maybe 10% of that, or you can just go do it yourself for a few hundred dollars. I get that this is scary as hell if you came up through the academic ranks; I get that irrelevancy really pisses people off — but I just don’t think you have a lot of credibility when you try to tell us that whether or not you get elections right isn’t a good measure of your skill as an election pollster. From where I sit, it’s the only thing.
(PS: PPP, I love you guys, but I think he’s right about one thing: an “or not” wording choice would probably be better.)
tl; dr: threatened much?
ETA: Forgot to add this, which is Nate Silver’s last (and horribly outdated, from May of 2008) pollster ranking chart. Still, it’s the last thing he gave us:
Look on the bright side, Gary: you’re better than Zogby.