2008 State by State Electoral Predictions Compared

The 2008 election is over and Obama solidly trounced McCain. Several sites kept a running total of polling results by state, including pollster.com and RealClearPolitics. Nate Silver at fivethirtyeight.com went a step further, analyzing the polling data and explicitly making predictions of the voting totals by state. I also found RJ Elliot who gives a state by state prediction of the voting percentages for Obama and McCain. There are no doubt other sites with predictions. I found one, but they wanted money to view their predictions, so they are omitted from this comparison.

I took the final estimates from these four sites to compare to the actual vote totals. A caveat should be kept in mind: Silver was making an explicit prediction about the election, while RealClearPolitics and pollster.com are aggregating and summarizing polling data, though presumably with the purpose of predicting the election. RJ Elliot also was making predictions, though I didn't see any explanation of his methodology. For actual results, I took data from Dave Leip's Atlas of U.S. Presidential Elections. I didn't try other sources, this was the first that I found.

Final cavil: I collected fivethirtyeight.com and RealClearPolitics' predictions on the day after the election. The pollster.com and RJ Elliot predictions were collected later, on Thanksgiving or the day after. While I don't think have any suggestion that any of these predictions were changed after the election, there certainly is the possibility of that happening.

Results
Herewith the results. I took the predictions for Obama and McCain's state by state percentage, subtracted from the actual percentage, squared the difference, averaged across states, and took the square root. This gives a root mean square error (RMSE) of prediction difference from truth for both the Obama side and the McCain side of the estimates. The results are in the image. Hopefully the image shows up in the right place.


For each site there are two results, for Obama and for McCain. For each row there are two numbers. It turns out that RealClearPolitics (RCP) only predicted results for 38 states out of the 50 states plus DC. The others predicted for all 51. The first column (starts 3.0, 2.9) includes all 51 states plus DC plus the national vote treated as a 52nd state. The second column is the predictions only on the 38 states where RCP made predictions. So the first column you can compare 538, pollster and Elliot. In the second column you can compare all 4 sites.

The states where RCP did not make predictions were the very partisan and small states where the winner was very clear throughout the entire election (think Utah and DC) and where there were very few polls. Both 538 and pollster do much better in the accuracy of their predictions for the subset where RCP was making predictions. On the 38 states where RCP made predictions, we see that 538 had the smallest RMSE followed by pollster then Elliot with RCP having the worst predictions.

Across the 52 predictions (50 states + DC + national), Elliot edges 538 perhaps barely, both of which definitely beats pollster. The RCP result in the first column is not comparable to the other numbers. I inspected the individual residuals (differences prediction minus reality) in the states where RCP did not make predictions. These residuals were often very large. The largest residuals for 538 were from the states (Alaska, Arkansas, DC, Hawaii, Louisiana, Nevada, Vermont, Wyoming), defined as being in error by more than 4% on either Obama's or McCain's vote total. DC, Hawaii, Louisiana Vermont and Wyoming were not predicted by RCP.

How do we know if the average pollster is unbiased?

There is a discussion going on at fivethirtyeight.com about whether Realclearpolitics is cherry picking the polls going into their averages.

Now, a comparison site is pollster.com, which uses all polls.

There is a problem with any of these models:

How do we know if the average pollster is unbiased?

A key parameter in 538's (or anyone's) election modeling has to be the grand mean across all polls.

Speaking in rough terms, the grand mean is estimated as the average across all pollsters, and the grand mean is our best estimate of who is winning. But we don't have that large a universe of pollsters. It's entirely possible the actual sample of pollsters is biased on average. How would we know if they were or were not?

For example, Daily Kos (DK) (Research 2000) recently started a tracking poll. DK appears to have the strongest pro-Obama lean of all tracking polls at the moment. They may be right, they may be wrong, I can't tell. BUT, suppose that prior to DK starting up, we actually had an unbiased set of currently active pollsters. Then, when DK starts up, suddenly the average pollster will lean towards Obama.

One thought: 538 does have some background data on pollster quality from the primaries. Under the assumption that pollster quality carries over to the general election, then it could be possible to estimate the average bias of a set of pollsters. However, the incentives, (really, the utilities) of the many various pollsters are different in the primaries and the general election. Whether you have a democratic or republican bias, you really don't have much preference between primary candidates, (unless you are employed by a particular candidate). In contrast, in the general election, you will have a preference between the two candidates and thus may be inclined to tilt your polling.

If Sean's/Nate's discussions of the Obama ground game are correct, DK could come closest to estimating the final result correctly. And if public predictions in other areas of policy (oil prices for example) are symptomatic, its entirely possible that all pollsters could end up on the same side of the actual result. This usually doesn't happen in the political sphere, but do remember New Hampshire.


I posted a version of this over at fivethirtyeight.com as well.