Bradley Effect--Fact or Fiction?

I was struck, looking at recent state polling on the Real Clear Politics site, how inconsistent the Quinnipiac polls are with those of other organizations.

  • In Ohio, they show an Obama lead 6-9 points greater than does any other poll.

  • In Florida, they show an Obama lead of 5-9 points greater than does any other poll.

  • In Pennsylvania, they show an Obama lead of 6-14 points greater than does any other poll.

These Quinnipiac polls show, at the very least, a 5 percent differential from ANY and ALL polls done during similar time-frames, and this differential is always to Obama’s advantage.

Note also, that in most cases, if one were to subtract the margin of error from each of Quinnipiac’s numbers, and add the stated margin of errors of the other polls, Quinnipiac would still usually be showing numbers that are either moderately or wildly at variance with all other polling. And again, always to the advantage of Barack Obama.

How is this even possible?

It goes without saying that either Quinnipiac is right or everybody else is wrong. In either case, it means that at least some people are doing questionable polling out there (not everyone can be right within their margins of error with these results) and that there must be some factor or factors being brought into play in polling that ought to be isolated and considered for what it might teach us about polling in general.

In investigating Quinnipiac’s methodology, one thing that jumped out at me immediately was their use of Quinnipiac student interviewers to conduct the polls.

Weighting by party ID, as all polls do, is always something to account for–but why would Quinnipiac be weighting differently in three separate states from all other organizations?

What stands out to me above all else–in regard to Quinnipiac’s differences with other polling organizations–is that their results are based on telephone conversations between voters and 18-22 year olds. The very same demographic, as virtually everyone has come to know, which supports Barack Obama over John McCain by overwhelming margins. This theory I am putting forth assumes that significant numbers of interviewees can identify–over the phone–the age of their interviewers. And that they will respond accordingly.

The so-called “Bradley Effect,” which is not, as is sometimes suggested, so much about racism as the phenomenon of interviewees attempting to conform to the assumed biases of their interviewers, begins to look like a very plausible explanation in the case of Quinnipiac. Again, since the use of those belonging to a very specific and narrow demographic group to conduct interviews is the most obvious way in which Quinnipiac differs from all other polling outfits.

Many pollsters (like Rasmussen) attempt to avoid the potential for this kind of statistical bias through the use of robo-calling, in which interviewees don’t interact with live human beings.

Why lie to a machine?

But to broaden the argument beyond Quinnipiac: if those polled are ever willing to lie (and I believe that the odd results put forth by the Quinnipiac polls plausibly suggest this to be the case) that means that the so-called “Bradley Effect” is real, and under the right conditions it will manifest itself in polling.

So where does this potential for lying stop and where does it end? Does robo-calling eliminate this potential form of statistical bias completely or merely diminish it? To answer that question, we’d have to know just strong the social pressures to identify oneself as an Obama voter are, and how far into the closet some percentage of McCain voters have gone. The cynical response, I suppose, is to say that perhaps they’re so deep in the closet that they’ll pull the lever for Obama on election day!

I have no doubt, for the record, that McCain is behind–in some important places significantly. But is polling really telling the whole story?

I believe that we can take for granted that the candidacy of Obama, the first African-American candidate to head a national presidential ticket, is a unique phenomenon that’s injected any number of factors into the political scene that might skew traditional polling. Some of these factors could benefit Obama. Perhaps some intending to vote Obama lie–because they live among McCain supporters–by claiming they’ll vote for McCain. And more likely than that, perhaps the pollsters have failed to account for a massive and unprecedented surge of first-time voters.

In any event, there is some solid evidence from this current presidential election cycle that suggests that even robo-calling can result in a significant pro-Obama bias.

Here’s an exellent example. Rasmussen predicted, in his final poll in the race, that Barack Obama would defeat Hillary Clinton in New Hampshire by 7% points.

The level of political activity in New Hampshire at the time was extremely high–like it is now in the entire country. New Hampshire is a state which is something like 96% white. The historical nature of Obama’s historical candidacy was trumpeted from the mountaintops. Polled Independents claimed they’d be voting for Obama by a 2-1 margin. Though presumably the Bradley Effect would be a less salient factor in a Democratic primary than in a general election, Barack Obama went on to lose by 2.6% to Hillary Clinton. The robo-calling was off by 9.6%. Currently, as a point of reference, Rasmussen says that McCain is down 10 points in New Hampshire to Barack Obama.