As Polls Go, I Really Prefer the Partisan Stuff

by Stuart Rothenberg September 15, 2010 · 9:00 AM EDT

Like most of you, I’m a huge consumer of polls. I rely on survey data, because they represent one of the few quantitative measures of candidate strength. Or do they?

For years, I have complained about national and local news anchors who don’t seem to understand what polls mean and what a margin of error is. But increasingly, for me at least, it’s the proliferation of polls — and polls of questionable value — that are the problem.

Some observers assume the worst offenders are campaign pollsters, who presumably cook their numbers to show whatever they want to show. Not true. When I’m presented with two polls, one conducted by a well-regarded Republican or Democratic pollster and the other by a “nonpartisan” pollster or state media outlet, I often place greater weight on the partisan pollster’s numbers, especially if I regard the pollster highly.

Obviously, not all “nonpartisan” pollsters are equal (any more than all partisan pollsters are), and some of them regularly deliver what appear to be reasonable data. But I’m increasingly suspicious of many polls.

Gallup is a first-class polling organization that has been around for years and seeks to maintain a certain level of methodological rigor. I have worked with Gallup’s editor in chief, Frank Newport, who has a doctorate in sociology from the University of Michigan, and I can testify that he knows more about quantitative methods than I could ever know.

And yet, despite my high regard for the people at Gallup, I have to shake my head in disbelief at how Gallup’s generic ballot has jumped around recently. And it isn’t just the most recent wild bounce.

Gallup’s weekly track of July 19-25 showed Democrats ahead by 4 points on the generic ballot, but the next week, July 26-Aug. 1, Gallup had the GOP up by 5 points. That’s a huge swing that isn’t easy to explain.

In late August, of course, Gallup’s weekly tracking data showed Republicans opening up a 10-point lead in the generic Congressional ballot, causing more than a small overreaction from veteran political observers. But a week later, Gallup’s weekly tracking data had the Congressional generic ballot even at 46 percent.

Then, if you compare Gallup’s generic ballot of June 28-July 3, when Republicans held a 2-point advantage, to its July 12-18 weekly generic ballot, when Democrats held a 6-point edge, you get another uncomfortably large swing without some dramatic news event.

One outlier over two months is understandable. Two is a stretch. Three is something to worry about.

It’s worth noting that the NBC News/Wall Street Journal poll, conducted jointly by a Democratic and a Republican pollster, doesn’t have dramatic swings in its version of the generic ballot.

The last nine NBC News/Wall Street Journal polls, conducted from December 2009 to the end of August, haven’t shown Democrats with more than a 9-point advantage on the generic or Republicans with more than a 2-point advantage. That’s a narrow range, and it strikes me as much more believable than Gallup’s wildly swinging numbers.

But Gallup isn’t the problem. If you want to see a problem, just look at the recent Columbus Dispatch poll of the Ohio gubernatorial race.

That survey, which showed Republican John Kasich with a 12-point lead over incumbent Gov. Ted Strickland (D), had an inexplicable sample that was 51.9 percent male and 87.3 percent white. In the 2006 midterms, exit polling showed the Ohio electorate to be 49 percent male and 79 percent white. That’s a problem.

Even more significant, while voters aged 65 and older constituted 34.4 percent of the Dispatch’s sample, they accounted for only 19 percent of ’06 midterm voters in Ohio. Not surprisingly given the overabundance of older voters in the Dispatch poll, younger voters are dramatically underrepresented in the recent survey.

Kasich would have been leading Strickland even if the sample looked better, but the margin would have been much closer, and almost certainly much closer to reality.

Or take the bizarre August SurveyUSA poll in Virginia’s 5th district, which showed Democratic Rep. Tom Perriello trailing his Republican challenger, Robert Hurt, by an astounding 26 points, 61 percent to 35 percent.

Democrats criticized the poll’s methodology, but I didn’t even need that to know the numbers were a joke. Both Democratic and Republican pollsters show Hurt ahead narrowly (and he is favored to defeat Perriello in November), but nothing like 26 points.

So why should anyone place more faith in partisan pollsters than in “neutral” polls?

Everyone agrees polling gets harder each election cycle. “It’s not just response bias, which favors the party that is winning in a particular cycle,” one veteran partisan pollster told me. “It’s a lot of things. Young people have cell phones, everyone has caller ID, so many people won’t take the call.”

Nonpartisan pollsters simply have less at stake in getting the right numbers in their polls. Sure, there is personal pride involved, but usually those pollsters are conducting surveys for local media outlets that simply want numbers to put in their newspapers or on local TV news. Often, these nonpartisan pollsters were established to promote an educational institution’s visibility.

Campaign pollsters have much more at stake. Their numbers drive campaign strategy, with victory or defeat of their candidate hanging in the balance. Their numbers have to be correct.

Partisan pollsters also argue that they spend more time making certain their samples reflect the actual electorate, even if it means incurring additional costs.

“Survey research depends on the dedication of the pollster to get it right. There are procedures when hiring the lowest bidder makes sense. Polling isn’t one of them,” said one partisan campaign pollster who isn’t a fan of many nonpartisan district and statewide polls.

It’s hard to disagree.