Diary

Lies, Damn Lies and Polls

Answer: They don’t

Ok, for the ADD folks out there you can go back to reflexively hitting refresh, hoping to get past the 500 errors now (I kid, I kid!). For those of you who need this education again, lets talk about what polling is and what it isn’t.

We’ve had a spate of polling data today that is causing interesting reactions among members (Obama is leading in Virginia and Colorado!, McCain is leading in Colorado and Virginia!, Everyone blames McCain for the financial mess!). Everyone take a deep breath.breathe

There, feel better?

Let’s reestablish a few facts:

1) Polling firms are a business. They conduct polls for clients and expect to get paid for the results of their work. In some cases (e.g. Rasmussen) they also publicly release a large amount of their results publicly because their clients want them too (news organizations generating news) or because it helps them generate business. In the case of Rasmussen, he wants you to pay to be a premium member so you can get results earlier and see the poll internals.

2) Polls all use a methodology. These methodologies are all different, but they have some similarities. All of them need a mechanism to normalize their data. Random chance dictates that at some point the poll will select twice as many Republicans as Democrats, how do you normalize the data to reflect an accurate picture of the full body of voters? Also, how do you distinguish between adults, voters and likely voters?

3) Some polls have better track records than others. This is shown by comparing the final poll results against the actual votes. The flip side to this is that in general there is no way to judge the accuracy of a poll at any point prior to the final week before the election.

4) Pollsters have agendas. This is a fact of life, and doesn’t mean anything necessarily sinister. Pollsters work for clients. If clients want a question asked a certain way, then it will be asked that way. The client may well then report the results because it fits the template of their agenda. This is especially true for partisan pollsters (those working directly for campaigns) that publish results. Campaign internal polls are never shared, and usually are the most accurate of all.

So what to make of the differences in all the polling results? Is Obama really up 5? Or is it tied? Or is McCain up 2? Well the raw results from the polls are certainly accurate, the differces have to do with the methodology applied to normalize the data and assess likeliness of voting. This is where we get into the dreaded “over sampled Democrat” controversy. Is this real or not? I personally believe that it is, but then in 2006 I thought the polls were flawed and I was shown to be dead wrong.

Rasmussen has an interesting new analysis on party affiliation. This is a poll he has been conducting since 2002 which has a solid methodology. It uses a massive sample size of 15,000 to ask a single question regarding party affiliation. During July, this poll gave a 39/31/30 Democrat/Republican/Independent split. Note that this breakdown is much lower than other affliation models you will see from other pollsters, who often sample Democrats in the low 40s (and CBS even released one with a Democrat sample of 48% once).

Well this breakdown shifted in August. It is now 39/33/28 D/R/I. This shift of two points from the Independents to the Republicans has to be factored into Rasmussen’s polling methodology, which tends to now produce a slightly better result for Republicans. As for how Rasmussen actually factors this into his methology, I really don’t know.

What is all means is take current polls with a grain of salt. Pollsters have no inherent incentive to give accurate results at this stage. There is nothing to stop them from “pulling a Zogby” and claiming a late breaking shift to one of the candidates that accounts for the wild swings in their polls up to election day. Personally, I like Rasmussen, I think he has played it straight over the years, and I don’t believe that the electorate really fluctuates much at all late in the season. I also respect the final results of SUSA for state polls, but I don’t know their track record reporting 6 weeks out. But who really knows? Zogby was dead on in 1996, so anything is possible.