The Pathetic Pundit Playbook

by Andrew McAfee on November 9, 2012

In 2008, Nate Silver correctly predicted the winner of 49 of the 50 states in the US presidential election. Based on this impressive track record the New York Times hosted his fivethirtyeight blog this year. All throughout the election season Silver’s statistical models, which make heavy use of results from national and state polls as well as data on economic trends, constantly updated their predictions about, among other things, the percentage of the popular vote Obama and Romney would receive, the number of electoral votes each would get, and the probability that each would win the election, expressed in percentage terms.

Silver started posting his predictions on May 31. At that time, he gave Obama a 63.7% chance of winning. That number dropped to 59% on June 2, and that was as low as it got. Throughout the entire summer and fall, fivethirtyeight projected that Obama would win, and made the projection with a lot of confidence. The model always predicted that the popular vote would be quite close (on October 12, for example, it had Obama taking home 49.8%, Romney 49.1% on November 6), but that Obama would capture enough electoral votes to win.

For over five months, in short, Silver consistently stated that although the election would be close, it wouldn’t really be close. The selection of Paul Ryan as Romney’s running mate, the conventions, the debates, the outside money, the ads… none of them changed the fundamental fact that Obama was going to win, according to the model (Romney’s prospects did improve significantly after the first debate but worsened again after the second and the model wound up, just before the election, right about where it was before the first debate). Obama consistently held big enough leads in enough states to get him above the 270 electoral votes required to win.

For this numerical slant and certainty, Silver was pilloried and ridiculed — as far as I can tell without many exceptions — by political ‘experts’ on the right (If he’d been predicting a Romney victory, pundits on the left might not have been much kinder to him). Their responses are super instructive, because they’re a microcosm of how pundits react to geeks.

By pundits here, I mean people who are perceived as being knowledgeable about a topic because of some combination of accumulated experience, insider access, intuitive ability, glittering resume, and bully pulpit. By geeks, I mean people who form their opinions after spending a lot of time analyzing the relevant data. These are two very separate classes, and they don’t think much of each other.

Pundits react to the work of geeks by:

Misunderstanding basic concepts. Former congressman and Morning Joe host Joe Scarborough mocked Silver on his show, stating that “Nate Silver says this is a 73.6 percent chance that the president is going to win? Nobody in that campaign thinks they have a 73 percent chance — they think they have a 50.1 percent chance of winning. And you talk to the Romney people, it’s the same thing… Both sides understand that it is close, and it could go either way. And anybody that thinks that this race is anything but a tossup right now is such an ideologue, they should be kept away from typewriters, computers, laptops and microphones for the next 10 days, because they’re jokes.”

Scarborough appears here to be confusing the percentage of the popular vote with Silver’s probability estimate, expressed in percentage terms, that Obama would get at least 270 electoral votes. As Silver said,  “I’m sorry that Joe is math-challenged.”

Columnist-pundit David Brooks appeared to make an even more fundamental error: that in an uncertain world, predictions of any kind are a fool’s errand. He said that “…if you tell me you think you can quantify an event that is about to happen that you don’t expect, like the 47 percent comment or a debate performance, I think you think you are a wizard. That’s not possible. The pollsters tell us what’s happening now. When they start projecting, they’re getting into silly land.”

No, projections are not always silly, even though uncertainty exists. When a team is leading 6-0 after 8 innings, it’s not silly at all to predict that they’re going to win. It would be dumb to make that prediction with 100% confidence, but that’s not what Silver was doing. His probability estimate of an Obama victory never got higher than 91%.

Changing the topic. Michael Gerson wrote in the Washington Post that “The main problem with [Silver's] approach to politics is not that it is pseudo-scientific but that it is trivial. An election is not a mathematical equation; it is a nation making a decision. People are weighing the priorities of their society and the quality of their leaders. Those views, at any given moment, can be roughly measured. But spreadsheets don’t add up to a political community. In a democracy, the convictions of the public ultimately depend on persuasion, which resists quantification.”

I never heard Silver claiming that his models added up to ‘a political community;’ they instead yielded a prediction about who was going to win the election. This might not be what Gerson values or wants to do, but it’s far from trivial.

Doubling down on old-school punditry. One of my favorite examples of this was Michael Barone’s November 2 column in which he forecasted the outcomes in swing states using, as far as I can tell, a little data and a lot of gut feel. An excerpt:

Ohio (18). The anti-Romney auto bailout ads have Obama running well enough among blue-collar voters for him to lead most polls. But many polls anticipate a more Democratic electorate than in 2008. Early voting tells another story, and so does the registration decline in Cleveland’s Cuyahoga County. In 2004, intensity among rural, small -town and evangelical voters, undetected by political reporters who don’t mix in such circles, produced a narrow Bush victory. I see that happening again. Romney.

Virginia (13). Post-debate polling mildly favors Romney, and early voting is way down in heavily Democratic Arlington, Alexandria, Richmond and Norfolk. Northern Virginia Asians may trend Romney. Romney.

Colorado (9). Unlike 2008, registered Republicans outnumber registered Democrats, and more Republicans than Democrats have voted early. The Republican trend in 2010 was squandered by weak candidates for governor and senator. Not this time. Romney.

Iowa (6). The unexpected Romney endorsements by the Des Moines Register and three other newspapers gave voice to buyer’s remorse in a state Obama carried by 10 points. Democrats’ traditional margin in early voting has declined. Romney.

Obama won all four of these states.

Claiming better information. Karl Rove, who in his own November 2 column used a ‘methodology’ close to Barone’s to predict at least 279 electoral votes for Romney, was part of Fox’s election night coverage. When the network called Ohio for Obama, Rove disagreed, saying that his sources had the two candidates much closer than did Fox.

Trying to overturn reality. Rove actually tried to get Fox to take back its Ohio call for Obama. “Karl Rove said that we should figure out what the deal is” said an election night anchor. A conversation then ensued among Rove, the anchors, and leaders of the team responsible for deciding when to declare that each state had gone to one of the candidates. Fox stayed with its call, and Ohio stayed with Obama.

If you keep your eyes open for these tactics, pundits become easy to recognize. When you spot one, move calmly and purposefully toward the nearest exit.

As I look at the work of pundits here (and elsewhere), I have two main questions. The first one is whether they know how inferior their methods are compared to those of geeks. Do they honestly think that Silver and his kind are misguided or beatable? Or are they hypocritically disparaging the geeks in public while believing them in private? It’s a tough question to answer; knaves and fools often look very similar from the outside.

My second question is why does anyone still listen to pundits after the geeks move in? The great Web comic xkcd got it exactly right after the election: “BREAKING: To Surprise of Pundits, Numbers Continue to be Best System for Determining Which of Two Things is Larger.” As this graph from the Simply Statistics blog shows, Silver’s 2012 predictions were amazingly accurate. In fact, he called all 50 states correctly this time. He forecast that Obama would take 50.8% of the popular vote, and Romney 48.3%; Google tells me that the actual totals were 50.5% and 48.1%, respectively. Sam Wang’s Princeton Election Consortium and other geek prediction sites were similarly spot-on. So they should be the go-to sources, full stop.

I realize that TV stations, newspapers, websites, and magazines have space to fill, and that pundits are a great way to do this. So we probably can’t stop hearing from pundits when it comes to elections, markets, trends, and so on. But we can certainly stop listening to them. As soon as we get data and geeks, that’s exactly what we should do.

  • Anonymous

    In fairness, there is some sober minded criticism of Nate. Questioning whether his inputs (state polls) were valid isn’t a conspiracy theory. Pointing out where Nate has been wrong in the past (because the state polls were wrong) is hardly an assault on math. His model is built on assumptions and some people didn’t buy those assumptions. Turns out Nate’s assumptions were correct, but that doesnt mean it was outrageous to question them.

  • http://community.ca.com/blogs/theitgovernanceevangelist/ Steve Romero

    I don’t think Andrew ever said it was “outrageous to question” Nate Silver. I do believe Andrew implied it was outrageous to ridicule or dismiss Nate Silver – which most pundits did. In this matter the pundits were indeed quite pathetic.

  • Anonymous

    Perhaps I overstated, but my point was that the geeks can be wrong. I can’t believe I am actually defending pundits, but there were some good ones (and good statisticians) it had honest disagreements with Silver’s methodology. Silver certainly turned out to be right, but I think we can all recall sometimes where they geeks got it wrong. Many of those could have been potentially prevented by willing to listen to some people who have experience (beyond geekery).

Previous post:

Next post: