As promised, a mea culpa on my pre-election poll analysis: why I was wrong, why the state poll averages were right – and why I’d say most of the same things if I had to do it over. I suppose I have lost a good deal of credibility with a number of people by making the kind of out-on-a-limb prediction I don’t usually make, and being wrong. But my assumptions have always been out in the open. Let’s examine why they led to the wrong answer, and which of those assumptions should be re-evaluated in the future.

I. More Evidence Is Better Than Less

Discussions of polling often lend themselves to more heat than light. A lot of the post-election poll commentary is even dumber than the pre-election poll commentary, as victorious liberals spin a narrative that conservatives were all “poll deniers” or “poll truthers” ignoring the polls. Now, it’s true that there were more than a few people on the Right who made intelligent discussion of the polls harder rather than easier. It’s also true that some of the efforts at “unskewing” the polls were unhelpfully ham-handed; in this pre-election essay I explained why Dean Chambers’ unskewedpolls.com was more alchemy than science. Chambers was projecting a strong Romney lead in the polls back in September, when neither I nor almost anybody I knew believed that Romney was actually ahead. My mantra on Twitter in August and September, like that of many conservatives I respect, was simply that it was a close race, that Obama had problems with independent voters and hadn’t closed the deal yet, and that there was still time for Romney to catch up to him. While a lot of Romney’s problems were baked-in by mid-summer, I don’t think that was an unreasonable view to take at the time; as it turned out, there would be twists in the race throughout October. But those of us who attempted to take the polls seriously, and drew conclusions from the polling evidence itself as well as external evidence, were not denying anything; we were just looking under the hood.

A. Believe The Polls, But Don’t Believe Only The Polls

Let me start by restating my philosophy of polling, and indeed my philosophy of examining most any question. Polls are not reality. They are a tool for measuring reality. They are traditionally the best single tool, and polling averages make them a better tool by evening out the outliers. But they are not perfect tools: public polls have called the outcome of races wrong before, or been off on the margins of victory by a significant amount. There are reasons why campaigns spend a lot of money on their own polls, which – other than for purposes like push-polling or testing messages – they would not do if an average of public polls was as flawless a guide to the electorate as a thermometer. If you think public poll averages are an infallible predictor, then the uniform practice of actual campaigns is totally irrational. Indeed, listen to Jim Messina, the nuts-and-bolts guru of the Obama campaign, on his view of public polling:

Every night, Obama’s analytics team would run the campaign 66,000 times on a computer simulation. “And every morning we would come in and spend our money based on those simulations,” said Messina.

Their models ultimately predicted Florida results within 0.2%, and 0.4% in Ohio. The only state they got wrong, noted Messina, was Colorado, “where we got one more point than we thought we would.”

The Obama campaign was able to do that, he said, because they turned away from mainstream polling from shops like Gallup, which he called “wrong the entire election,” in their prediction that fewer minorities and fewer young people would turn out to vote.

“We spent a whole bunch of time figuring out that American polling is broken,” said Messina. “We never did a national poll. We only did local and state polls.”

Is Messina a “poll truther” or “poll denier” for saying that “American polling is broken” or rejecting “traditional” polls? Was he wasting Obama’s money by running his own state and local polls instead of just reading 538? Or is he reflecting the fact that public polls involve a certain amount of guesswork about voter turnout that can only be definitively tested each new election cycle by the final vote tallies?

The most recent set of publicly available polls are also not the only tool for measuring reality. There are all sorts of metrics – some more hard and quantifiable than others – that have traditionally been useful in assessing the state of play: voter-registration numbers, early voting data and absentee ballot data, trendlines in the polls, what the pollsters themselves are reporting about voter enthusiasm, and hazier indicators like small-dollar donations and the size of crowds on the campaign trail. It’s hardly anti-empirical to examine these additional facts, and allow them to affect your conclusions. The people who correctly predicted that Harry Reid would hang on to beat Sharron Angle when the final polls said otherwise were looking at this kind of information. If all you did was read the RCP average or 538, you would have missed that.

None of these are reasons to disregard the polls; they should still be the primary item of evidence. But neither does it make sense to treat them with blind reverence and ignore all the external evidence – especially when (1) different sets of polls are in conflict with each other and (2) the internal breakdowns of the polls, which can act as a sanity check on the plausibility of the polls’ assumptions, are telling you that the polls in question are predicting something that is historically unlikely to happen.

To go back to one more example from baseball analysis (sabermetrics), this is sort of like the “statheads vs scouts” debate: whether you can better project a player’s future accomplishments by looking at his statistics or by listening to experts analyze his “tools.” I’m in the stathead camp: I believe strongly in quantifiable data, and I place it at the center of just about any baseball analysis I do. In fact, I probably err more often than not on overemphasizing the numbers. But as Bill James, the father of sabermetrics, has often said, sabermetrics (or any other field of hard science, soft science or would-be science) is not the search for statistics, it’s the search for truth, and in searching for the truth, you don’t blind yourself to some of the available evidence simply because it doesn’t fit in a column of your model. Most statheads today will tell you that no matter how good your statistical measurements and how highly you prioritize them, you can learn things from also listening to the scouts that you might not learn anywhere else. The scouts can’t tell you a guy with a terrible stat line is actually a major star – but in a close case, their view can make the difference.

B. Regrets, I Have A Few

If I had it to do over, maybe I’d be less definitive in pronouncing Obama “toast,” but for the most part I’d look at the same evidence, see the same things and draw the same conclusions I drew at the time. I do, however, have two main regrets.

First, which is an unfortunate side effect of doing punditry as a part-time unpaid hobby rather than a full time job, is that I should probably have spent more time breaking down the internals of all the battleground-state polls to see exactly how geographically consistent the national trends were: I had seen those trends confirmed in Ohio and a couple of the other key battleground states, and didn’t do the deep, time-consuming dive into the internals of every single battleground state. As it turned out, Romney did very well with independent voters in a number of the battleground states, but critically underachieved with them in others and lost those states. Romney beat Obama with independents by 5 points or more in Ohio (+10), Virginia (+10), North Carolina (+15), Nevada (+8), and Pennsylvania (+5). In more lightly-contested states, he won them in Missouri (+26), Indiana (+11), New Mexico (+8) and Arizona (+6). But in two other battleground states, his margin with independents was narrow: Colorado (+4), and Michigan (+1). And he actually lost independents in Iowa (-14), New Hampshire (-7), Florida (-3), Wisconsin (-1), and Minnesota (-4). (It says something about Romney that his biggest failure with independent voters, especially in states with lily-white electorates that didn’t present the demographic challenges apparent elsewhere, came in the places where the voters had been exposed at length to his scorched-earth 2008 and 2012 primary campaigns). In non-battleground states where exit polls were taken, independents mostly trended pro-Romney in red states – Alabama (+52), Mississippi (+34), Montana (+7) – and pro-Obama in blue states – California (-12), New York (-4), Illinois (-5), Vermont (-40), Maine (-28), Maryland (-12), Massachusetts (-7), Oregon (-7), Washington (-3), Connecticut (-3). In just two totally non-competitive states did independents buck the state’s partisan tilt: Romney won independents in New Jersey (+1), and Obama tied him in Kansas. If Romney had won all the states where he carried independents, he would have won, but not in all the states I thought he would win:

I might have caught some of this with more time to commit to each state, although in some cases, that weakness came out only in the very late polls. Or not; for example, PPP had Romney up by 2 with independents in New Hampshire on October 28, and up by 7 with independents in Florida the same day.

Second, and far more importantly, I didn’t do enough to re-evaluate my conclusion after declaring it on Twitter on October 19 and laying it out in detail on October 26, eleven days before Election Day – and that week turned out to be Romney’s high water-mark in the national and state polls. Nationally, Romney led in the RCP average all but two days between October 9 and October 31, and Obama was below 48% in the average every day from October 8 to November 5. In four key states, Romney had seized the lead in the RCP average: in Colorado, he led from October 9-29; in New Hampshire, he led from October 19-21; in Virginia, from October 19 to November 2; in Florida, from October 8 to the end of the race. Those leads gave me confidence that Romney’s momentum in the national polls was real and would leave him needing to pick off just one more state to win. Instead, he lost all four, and the state poll averages were predicting three of those four losses by Election Day. When I wrote my initial post, I assumed (given the lateness in the cycle and the evident deterioration of Obama’s position in the national polls and with independent voters by October 26), that there was not time for any new game-changing events. I was wrong.

I try hard to avoid confirmation bias in evaluating the evidence before reaching a conclusion – but it can be a lot harder to avoid it after publicly committing to a conclusion, which of course is the exact same “painted in a corner” problem I suspected Nate Silver of having. I spent the week after my “toast” post without electricity, internet, heat or hot water and only sporadically able to get to my office (I had written most of this essay> when the lights went out on October 29), and probably did not spend enough of my remaining time going back over the subsequent polling to reconsider whether the conditions that seemed to show Obama in an untenable position were abating. (By Election Day, for example, the national poll averages were running in Obama’s favor as well). I remain doubtful that Hurricane Sandy had enough impact to swing the election and particularly doubtful that it did much to affect turnout, which decided the election; but its interruption of the dynamic of the endgame probably did do what the final week state and national polls showed happening: cut into Romney’s lead with independents and possibly cost him his chance to win New Hampshire, Virginia, and/or Florida. Exit polls showed that late-deciding voters broke in Obama’s favor, which I had not expected to happen for an incumbent whose approval ratings and favorability with independent voters had been underwater for so much of his term.

II. Poll vs. Poll

The polls, particularly the national and swing-state polls in the third and fourth weeks of October, were telling consistent stories about the opinions of different groups of voters, but contradictory stories about the numbers in which those groups would show up to vote. I looked at internal evidence, and saw that Romney was winning independents handily and Obama was drawing nearly no crossover support from Republicans, which meant Obama needed to win entirely by having enough extra Democrats vote to overcome Romney’s independent advantage. The first premise was borne out by Romney’s win with independents and slight edge with crossovers, although his 5-point margin with independents was at least 3 points smaller than the margin most polls had been showing when I first made my “toast” call. Had Romney carried independents by 8 or 9, as the polls were showing at the time, and not lost them in a few key states, Obama would have needed at minimum a D+4 electorate to win nationally. As it turned out, Obama won by about 2 points with a D+6 electorate compared to 2008’s D+7; Democratic turnout was down a point from 2008, but Republican turnout matched the 32% of 2008. So, my analysis correctly judged what Obama needed to do. The historic turnout conditions of 2008, which I believed were necessary for Obama to win, were effectively repeated.

There were all sorts of reasons, based in history and observable fact, to believe that those turnout conditions were unlikely to repeat themselves. Republican turnout had been unusually low in 2008 – the lowest since before Ronald Reagan started converting a lot of people to the party – following the historic financial crisis and 8 years of Bush, and had bounced back in 2010. While 2010 was a midterm election and involved a distinct electorate, its results underscored that there seemed no obvious reason to believe it would approach that nadir again. History tells us to treat with caution the assumption that elections held in the aftermath of a catastrophic event like the 2008 financial crisis are representative. The harder metrics, which I will discuss in Part III, also suggested GOP turnout was still doing well, if not as well as in the off-year elections of 2010. And the vaunted OFA turnout machine could only count for so much: certainly, no matter how sophisticated Obama’s turnout operation, low GOP turnout was something he could not manufacture through operational efficiency. Obama had polled poorly in areas like job approval for most of his tenure, and campaigns that lose independent voters tend to be losing at least some of the enthusiasm of their base as well. As I will discuss below, virtually all the features of Obama’s profile in the polls – mirrored in the post-election exit polls – were traditionally characteristic of losing candidates, as were the nature of the “this time it’s different, really!” arguments made by his boosters. It had to be different for him to win. The customary laws of political gravity had to be defied. And they were.

As I’ll get to more in Part II, with regard to turnout, what the exit polls show is that Obama really did do something that was very, very historically unusual – and on top of that, and perhaps even more importantly, Romney’s GOTV operation (both the mechanics and his ability to inspire marginal Right-leaning voters to show up) turned out to be far less than projected. This time, it really was different. And the challenging question going forward is whether it will stay that way.

III. State vs. Nation

For the third consecutive presidential election, the RealClearPolitics averages of state polls correctly called nearly all the states correctly – 49 of 50 states (all but Wisconsin) in 2004, 48 of 50 (all but Indiana and North Carolina) in 2008, 49 of 50 (all but Florida) in 2012. (The four missed calls all involved states that broke late and narrowly to the Democrats). There was a lot of hoopla about Nate Silver’s 538 model, and in my poll-reading pieces I noted – as have others – some quibbles with his model, with which I still have issues. But it’s not clear that the methodological quirks of the 538 model matter much anyway; they’ve yet to deliver a different answer than the RCP average in a presidential contest or in any significant number of statewide elections. Ditto for other state poll averages, all of which were basically on the mark. The state poll averages implicitly projected an electorate with a large Democratic turnout advantage, and that’s what happened.

That does not mean “the polls” were all correct. The national poll averages at both RCP and TPM had Romney ahead for much of October and ahead at the time I wrote my original “toast” post. This race was often compared to 2004 and ended up much the same, but in 2004, John Kerry never led in the RCP average in October. There were a variety of similar precedential indicators that Obama was in worse shape in the national polls than anybody who had gotten re-elected in the past. And traditionally, we have tended to see movements in the national polls followed by movements in the swing state polls – exactly what was reflected in the RCP average in Colorado, New Hampshire, Virginia and Florida. By October 29, Rasmussen’s state-by-state polls were showing Romney leading in enough states to win the Electoral College. Of course, Rasmussen was wrong.

It is somewhat Orwellian for Obama supporters to paint as “poll deniers” those of us who looked at a conflict between two sets of polls containing two divergent sets of assumptions, and conclude that one of those sets of polls was more likely to be correct because external evidence rendered those assumptions more reasonable.

There was great debate over whether it mattered at all to look at the national polls, since presidential elections are determined on a state-by-state basis. This was really two questions in one: how likely was it that there would be a popular vote/electoral college split, and was it more likely that you could predict the national popular vote from state polls, or predict the battleground state vote from national polls. It is worth noting, on the first score, that the defenders of the state poll averages were only half right. On the one hand, they spent months arguing that Ohio in particular was a firewall that would remain favorable to Obama even if the national vote went slightly for Romney – and if Romney had hung onto his leads in Colorado, New Hampshire, Virginia and Florida, Ohio would have been enough to put him over the top. I thought the theory that Ohio would vote more Democratic than the nation was unlikely: Ohio has had an exceptionally consistent record from 1860 through 2008 of being just barely more Republican than the country as a whole nearly every election. I was right about that: Obama won Ohio by less than his national margin. As The New Republic’s Nate Cohn notes:

[T]he exit polls show that Obama did worse among Ohio’s white voters than John Kerry…. if there was anywhere that the president should have excelled due to the auto bailout, it would have been northeast Ohio. But the president lost northeastern Ohio’s two classic white middle class bellwethers: Lake County, home to the overwhelmingly white suburbs and exurbs east of Cleveland, and Stark County, home to Canton. The president also lost additional ground in traditionally Democratic stretches of eastern Ohio, where Obama performed worse than any Democrat since McGovern in a stretch of “coal country” along the Ohio River. And Obama’s problems weren’t limited to eastern Ohio. The president performed poorly in southwestern Ohio, including one deeply conservative and culturally southern county where Obama’s performance was the worst by a Democrat since at least 1868.

But score one as well for the firewall theory as a whole: Obama could have lost Ohio and Florida (both of which voted more Republican than the nation) and Virginia (which voted more Democratic than the nation) and still won 272 electoral votes.

In theory, you should always look only at the state polls, since that’s where the election is; if the state pollsters are as good at what they do and poll as regularly as the national pollsters, there would never be a reason to look at national polling. While we all look at national polls, especially early in the race when states are being polled less regularly, I mainly looked at state poll averages in 2004 and 2008. Which gets to the second point: how likely state polls were to offer a better picture of the national environment than national polls. I agreed with Sean Trende that the potential flaw in this theory was that many of the state pollsters at issue (other than pollsters like Rasmussen or PPP that poll both nationally and in the states) were relatively new to the game or had individually sketchy track records that couldn’t be automatically substituted for the record of “state polls” as a whole.

As it turned out, though, the state pollsters were closer to the mark, and should be the main focus of our attention in future races, at least in the closing months. But why were they right when the national pollsters were wrong? For the answer to that, you need to look at how the pollsters decided what the electorate would look like. I will cover that topic tomorrow in Part II.