Survey Says: No Consensus On Best Political Polling Method

Voters increasingly sense that political polls often have more to do with influencing public opinion rather than measuring it.

But some still might not know enough about the survey process to judge a poll's credibility.

"There are just so many polls coming out now that it's difficult to really keep track of them," said Inside Michigan Politics Editor Bill Ballenger. "Nobody should get overly excited about the results of just a single poll. I tell people that about the only thing they can do is look at all of the polls, average them out, and use their own judgment."

Predicting who will actually participate in an election is a key factor in polling. To do this pollsters typically use data from previous elections to build an electoral model. This model is a prediction of what percentage of voters in an election will be Democrats, what percent will be Republicans, Independents, etc. The models also attempt to break down the prospective voters by categories such as gender and age.

Using their electoral models pollsters set benchmarks for the various voting groups. After a voter survey is taken, the pollsters adjust (or weight) the survey results to compensate for groups that either surpassed or failed to reach the pre-set benchmarks. These adjustments bring the poll results in line with the electoral model that each particular pollster uses.

If the initial electoral model is flawed, chances are that the poll results will be wrong. There are strong disagreements over which models come closest to predicting who will participate in an election. There also are disagreements over various aspects of polling methods.

For instance, (FMW)B, a relatively new Michigan-based polling firm, refuses to use exit polling data from past elections when it assembles its model.

"We don't use exit-polling for our electoral modeling because we believe that it under represents some voting groups and those who vote by absentee ballot," said Eric Foster of Foster, McCollum White & Associates, a Troy-based political consulting firm. "We believe the way we're doing it is more accurate. But not everyone agrees with us."

In mid-August, (FMW)B raised eyebrows among poll watchers with a poll showing Republican presidential candidate Mitt Romney leading Obama in Michigan 47.68 percent to 43.88 percent. If the poll was accurate, the Obama campaign could be fighting an uphill battle in what is considered a must-win state for his re-election chances

Nate Silver, an East Lansing native who works for The New York Times, cast aspersions on the (FMW)B poll by pointing to another poll that (FMW)B did earlier in Florida. Silver said the (FMW)B Florida poll estimated the turnout for voters aged 18 to 30 at just 3 percent.

Foster took issue with the criticism.

"I've explained this again and again," Foster said. "The 3 percent represented the number of respondents, not the percentage of voters aged 18 to 30 our model predicted. We weren't saying there will be only a 3 percent turnout among that age group.

"I'd also like to know what our Florida poll had to do with our Michigan poll," Foster said. "They criticize our poll but say nothing about some of the other polls that are based on exit polling from 2008 and are over-weighted on the Democratic side. Look, everyone who knows me knows I'm a Democrat, I vote for Democrats. It's no fun being criticized by other Democrats, but I just happen to believe that personal political views shouldn't affect the integrity of a poll."

In spite of the flap, many considered Silver's article informative. It pointed out a clear disparity between polls done in Michigan by polling firms that frequently poll nationally and in multiple states and those that usually focus primarily on Michigan. At the time the article was written the average of the national polls had Obama leading in Michigan by 7.3 points. Meanwhile, the average of the polls done by Michigan-focused firms had Obama up just 1.8 points.

"There is a gap between what the Michigan pollsters are getting and what the big national polls are getting," said Paul King, pollster for Marketing Resource Group (MRG) of Lansing. “When I look at all of these national polls. . . .  almost all of them are done by automated phone calls. One recent poll by a national firm was just of registered voters. That in itself makes the poll results suspect.”

Capitol Confidential has reported previously on the value of doing polls of "registered voters" instead of "likely voters."

"Another thing about all of these polls is that, amazingly, very few of them even try to get at the growing number of people who only use cell phones," King said. "I'm working very hard to try to get as many cell phone numbers as I can."

Foster says the method used by (FMW)B is unique.

"We have an analytical system that takes statistical data from the clerks (county, municipal, etc.) all around the state," Foster said. "Our benchmark is 1990 and we get data from that year and every election year since then. We don't use exit polling data, which by the way, isn't even available for pre-2002 elections.

"I'm not about to bad mouth other polls and pollsters," he said. "We are all out here trying to do our jobs. The approach we use is unusual because it's so time consuming. In fact, it's tedious. But we believe that doing that work on the front end allows us to be more accurate on the back end."

Steve Mitchell, of Mitchell Research in East Lansing, said that unlike (FMW)B, he uses exit polling data, but that doesn't mean he thinks (FMW)B model is necessarily flawed.

"Some pollsters use exit polling data and some don't," Mitchell said. "We look at it to help determine turnout . . . things like what percentage of voters are likely to be women, what percentage will likely be union members and so on.

“Every pollster has their own way of making their model," Mitchell said. "I'm not about to criticize any other pollster. What it really all comes down to is whether they get it right in the end. In my view, Foster has got a pretty good record. He called this year's Michigan presidential primary to within three percentage points."