Making Sense of Probabilistic Forecasts -- 2016 Election Edition

Posted in civics on Friday, November 11 2016

The U.S. Presidential election was on Tuesday and the result was not what many people were expecting, to put it mildly. Let's take a voyage of discovery to find out how we ended up here, and what to make of the apparent failure of election forecasters to predict...this...

This was my timeline all week.

The day of the election, the election forecasts looked like this:

Election forecasts

If we look only at the numerical scores, most forecasters were predicting a Hillary Clinton victory with >85% probability with the exception of FiveThirtyEight's which was more circumspect with a prediction of 71% (which was higher than it was the week before at about 65%). So all of the forecasters failed, right? Well yes, in that they all picked the wrong candidate, but there's more to it than that. Probabilities, in this context, represent our level of certainty in our prediction. Or to put it another way, probabilities tell us how surprised we should be if we are wrong.

I would like to take a moment and rewind to a few days before the election when everyone was piling on Nate Silver claiming his model was too uncertain and that reality was a 99% chance of a Hillary Clinton victory, anointing a new king of election forecasting. This is amazing reading in the cold hard light of what happened, and there is some spectacular shade thrown between stats nerds. More interesting is what everyone was fighting about.

Everyone agreed on what the polls were showing (broadly, I'm simplifying the heck out of everything) and in the pre-election models, there was broad agreement on the predicted vote share. However, different models treated the uncertainty in the polling differently. Nate Silver was more cautious than the rest and left a lot of uncertainty in his model, whereas (for example) the PEC used some different modeling techniques that they believed could account for this uncertainty and ended up with a more certain prediction. They were fighting over how surprised you should be if the election results turned out differently from the polling results.

I think there are a lot of lessons to learn in this about modeling in general, and I'm patiently waiting for those think-pieces to come out of the ovens, all hot and cinnamony. In the meantime, I would like to point out that Nate Silver's book The Signal and the Noise does a very good job discussing the difficulties and shortcomings in forecasting in a variety of contexts. The thread that ties the whole book together is how easy it is to fool yourself into being overly certain of an outcome, it is especially easy to be blindsided by an event that is not accounted for in your model.

Anyways, back to the election results: As I've been saying, the different election forecasts presented us with different levels of surprise, but what the hell does that mean? Let's take a silly example: coin flipping!

The FiveThirtyEight model said that the odds that Trump would win (0.41) were a little better than the odds of you flipping a coin twice and having it come up heads both times (0.33). If you don't think getting two heads in a row when flipping a coin is that shocking then, if you believe the FiveThirtyEight model, you shouldn't be that shocked that Trump won the election. Sure Hillary was favored, but not that heavily.

The PEC model, on the other hand, gave Trump a <1% chance of winning (odds <0.01). This about equivalent to flipping a coin seven times in a row and getting heads every time (odds = 0.0079). I would be very surprised by this coin flipping. Sure it's not quite Rozencrantz and Guildenstern are Dead territory, but it would be pretty surprising.

I was listening to the Weeds podcast post-election and they mentioned that the Vox election model, which was based on political science models and "fundamentals" instead of on polling, had predicted a Trump victory from the beginning. If you had been spending the last few years living in a cave in Tibet and didn't know anything about Clinton or Trump you would have predicted a Republican win, since historically the field is tilted against "3rd term" presidencies. In retrospect, our prior belief going into this election should have been that it was the Republican candidate who was favored, however the majority of forecasters had the exact opposite prior: the Democratic candidate was heavily favored right out of the gate. Not just that, but reporters wrote many a think piece about how crazy this election was because you would expect the Republican candidate to have the advantage, but we all know that Trump doesn't have a chance. That's something to reflect upon. It is easy to come up with posthoc rationalizations to filter data before it gets to your model: the fundamentals favor Trump? well, the fundamentals are wrong/inappropriate, the polling favors Hillary? great add that to the model.