Yesterday, the American Association for Public Opinion Research released a report on polling for the 2020 election. It was pretty ugly. This is the key, in my view, finding.

• The 2020 polls featured polling error of an unusual magnitude: It was the highest in 40 years for the national popular vote and the highest in at least 20 years for state-level estimates of the vote in presidential, senatorial, and gubernatorial contests.5 Among polls conducted in the final two weeks, the average error on the margin in either direction was 4.5 points for national popular vote polls and 5.1 points for state-level presidential polls.
• The polling error was much more likely to favor Biden over Trump. Among polls conducted in the last two weeks before the election, the average signed error on the vote margin was too favorable for Biden by 3.9 percentage points in the national polls and by 4.3 percentage points in statewide presidential polls.

 

In short, as a group, the polls were very wrong, and they were wrong in the same direction. When I was an Inspector General investigator, this is what we referred to as a clue.

Via Politico’s Pollsters: ‘Impossible’ to say why 2020 polls were wrong:

But unlike 2016, when pollsters could pinpoint factors like the education divide as reasons they underestimated Donald Trump and offer specific recommendations to fix the problem, the authors of the new American Association for Public Opinion Research report couldn’t put their finger on the exact problem they face now. Instead, they’ve stuck to rejecting the idea that they made the same mistakes as before while pointing to possible new reasons for inaccuracy.

“We could rule some things out, but it’s hard to prove beyond a certainty what happened,” said Josh Clinton, a professor at Vanderbilt University and the chair of the association’s 2020 election task force. “Based on what we know about polling, what we know about politics, we have some good prime suspects as to what may be going on.”

Those “prime suspects” will hardly be comforting to pollsters and those who depend on them, from political campaigns to the news media. The most likely — if far from certain — culprit for off-kilter polling results is that key groups of people don’t answer polls in the first place.

Decreasing response rates have been a major source of concern for pollsters for more than a decade. But the politicization of polling during the Trump era — including the feedback loop from the former president, who has falsely decried poll results he doesn’t like as “fake” or deliberately aimed at suppressing enthusiasm for answering polls among GOP voters — appears to be skewing the results, with some segment of Republicans refusing to participate in surveys.

A couple of observations. Response rates to polls have been declining for about 30 years. No evidence is presented in the study that indicates Republicans refuse to participate in surveys to a greater degree than Democrats. But blaming Trump is still the go-to solution.

That makes the problems with polling a lot harder to fix than the diagnosis four years ago, which mostly focused on adjusting surveys to account for Trump’s popularity with voters who haven’t earned college degrees and his corresponding weakness with college degree-holders.

“It seems plausible to the task force that, perhaps, the Republicans who are participating in our polls are different from those who are supporting Republican candidates who aren’t participating in our polls,” Clinton said. “But how do you prove that?”

This is some great analytical jujitsu. What this guy is saying is, “We were wrong four years ago because we underestimated the number of uneducated rubes who would vote for Trump and we fixed that problem. As a result our 2020 results were worse than our 2016. But we’re really sure than its Republicans screwing around with the hard science of polling.”

According to the report, national polls of the presidential race conducted in the final two weeks of the election were off by an average of 4.5 percentage points, while the state polls were off by just over 5 points. Most of the error was in one direction: Looking at the vote margin, the national polls were too favorable to now-President Joe Biden by 3.9 points, and the state polls were 4.3 points too favorable for Biden.

Most of the error came from underestimating Trump’s support, as opposed to overestimating Biden’s. Comparing the final election results to the poll numbers for each candidate, Trump’s support was understated by a whopping 3.3 points on average, while Biden’s was overstated by a point — turning what looked like a solid Biden lead into a closer, if still decisive, race.

It wasn’t just a Trump effect, either. The polls of Senate and governor’s races were off by an even greater margin: 6 points on average.

“Within the same state, polling error was often larger in senatorial contests than the presidential contest,” the AAPOR report reads. “Whether the candidates were running for president, senator, or governor, poll margins overall suggested that Democratic candidates would do better and Republican candidates would do worse relative to the final certified vote.”

No one methodology performed head-and-shoulders above the others. According to the report, there were only “minor differences” whether polls were conducted on the phone, over the internet or using a mixed methodology, including texts and smartphone apps — or whether they contacted voters randomly versus off a list of registered voters. “Every mode of interviewing and every mode of sampling overstated the Democratic-Republican margin relative to the final certified vote margin,” the report said.

Over at the website of polling guru Nate Silver, fivethirtyeight.com, they conducted an analysis last month (The Death Of Polling Is Greatly Exaggerated) and found polling was pretty damned good because of course, they did.

But while polling accuracy was mediocre in 2020, it also wasn’t any sort of historical outlier. The overall average error of 6.3 points in 2019-20 is only slightly worse than the average error across all polls since 1998, which is 6.0 points. There were also presidential years before the period our pollster ratings cover, such as in 1948 and 1980, when the polls exhibited notably larger errors than in 2020.

So, let’s recap. Polls favored Democrats regardless of the method used. Polls favored Democrats across most of the polling industry, no matter the modeling used by the analysts. The polls went the same direction in very similar amounts. The best reasons they can come up with is that Republicans aren’t participating because President Trump said mean things about them OR they are interviewing in great numbers Republicans who are not representative of the Republicans who turn out to vote…perhaps they have the staff of The Bulwark, the North American Man-Boy Lincoln Association, and The Dispatch on speed dial?

I think there is an easier explanation for what happened that doesn’t involve the entire polling industry’s failure to discover a key segment of GOP/conservative voters. The polls were conducted to drive a media narrative that America was appalled with Trump and with the GOP, and the Democrats were going to win big up and down the ticket. The polls provided the hook for “news” stories and fodder for talk shows. The idea was to create an air of inevitability about a Biden win. What they did not factor in was the anger of the conservative base and the degree to which millions of Americans felt that President Trump stood up for them and repaid that perceived loyalty by voting for him.

The real question looming in 2022 is will it happen again? Biden has a tenuous hold on the House and the Senate. The Democrats need a crap-ton of things to go their way to hold onto Congress. Will the pollsters revert to form and begin showering us with polls showing the Democrats marching inexorably to victory? Or will they try to produce accurate forecasts? I know what I’m betting on.