• Background
  • Presidential
  • State
    • State Win Probabilities
    • State win vote-shares
  • Senate
    • Senate Win Probabilities
    • Senate win vote-shares
    • Log scores of win predictions
  • Summary tables
    • RMSEs
    • Brier scores
    • Log scores
  • See also
  • External links

Statistically analyzing in R hundreds of predictions compiled for ~10 forecasters of the 2012 American Presidential election, and ranking them by Brier, RMSE, & log scores; the best overall performance seems to be by Drew Linzer and Wang & Holbrook, while Nate Silver appears as somewhat over-rated and the famous Intrade prediction market turning in a disappointing overall performance.

In November 2012, I was hired by CFAR to compile an extensive dataset of pundits, modelers, hobbyists, and academics who had attempted to statistically forecast the 2012 American presidential race and other minor races; the results were interesting in that they contradicted the lionization of Nate Silver’s forecasts in The New York Times. This page is a full listing of the R source code I used to produce my analysis for the CFAR essay; notes on the derivation of each dataset are stored at 2012-election-gwern-notes.txt.

The essay itself lives at Was Nate Silver the Most Accurate 2012 Election Pundit?.

Background

This election prediction judgment divided up into several sections dealing with different categories of predictions:

  1. the overall Presidential race predictions: probability of Obama victory, final electoral vote count, and percentage of popular vote
  2. the Presidential state-by-state predictions: the percentage Obama will take (vote share/margin/edge), as well as the probability he will win that state at all
  3. the Senate state-by-state predictions: similar, but normalized for the Democratic candidate

Few forecasters made predictions in all categories, the ones who did make predictions did not always make their full predictions public, etc. Note that all percentages are normalized in terms of that going to Obama, Democrats, or in some cases, Independents/Greens. The Reality forecaster is the ground truth; these were all updated 23 November in what is hopefully a final update.

The point of these calculations is to extract Brier scores (for categorical predictions like percentage of Obama victory) and RMSE sums (for continuous/quantitative predictions like vote share). Intrade prices were interpreted as straightforward probabilities without any correction for Intrade’s long-shot bias1

Presidential

presidential <- read.csv("www.gwern.net/docs/2012-election-presidential.csv", row.names=1)
# Reality=2012 result; 2008=2008 results
presidential
                probability electoral popular
Reality              1.0000       332   50.79
2008                 1.0000       365   53.00
Nate Silver          0.9090       313   50.80
Drew Linzer          0.9900       332      NA
Simon Jackman        0.9140       332   50.80
DeSart               0.8862       303   51.37
Margin of Error      0.6800       303   51.50
Wang & Ferguson      1.0000       303   51.10
Intrade              0.6580       291   50.75
Josh Putnam              NA       332      NA
Unskewed Polls           NA       263   48.88

# probability can be scored as a Brier score; available in 'verification' library
install.packages("verification")
library(verification)
# handle lists & vectors for later
br <- function(obs, pred) brier(unlist(obs),
                                unlist(pred),
                                bins=FALSE)$bs # bins=FALSE avoids rounding
# convenience function
brp <- function(p) brier(presidential["Reality",]$probability,
                         presidential[p,]$probability,
                         bins=FALSE)$bs
lapply(rownames(presidential)[1:9], brp)
  • Reality: 0
  • 2008: 0
  • Wang: 0
  • Linzer: 0.0001
  • Jackman: 0.007396
  • Silver: 0.008281
  • DeSart: 0.01295044
  • Margin: 0.1024
  • Intrade: 0.116964
  • Random: 0.25 (50% guess is always 0.25)
# To score electorals and populars, we use RMSE
rmse <- function(obs, pred) sqrt(mean((obs-pred)^2,na.rm=TRUE))
rpe <- function(p) rmse(presidential["Reality",]$electoral, presidential[p,]$electoral)
lapply(rownames(presidential), rpe)
  • Reality: 0
  • Linzer: 0
  • Jackman: 0
  • Putnam: 0
  • Silver: 19
  • DeSart: 29
  • Margin: 29
  • Wang: 29
  • 2008: 33
  • Intrade: 41
  • Unskewed: 69
rpp <- function(p) rmse(presidential["Reality",]$popular, presidential[p,]$popular)
lapply(rownames(presidential)[c(1:9,11)], rpp)
  • Reality: 0
  • Wang: 0.31
  • DeSart: 0.58
  • Jackman: 0.01
  • Silver: 0.01
  • Intrade: 0.04
  • Margin: 0.71
  • 2008: 2.21
  • Unskewed: 1.91

State

State Win Probabilities

# Reality=final 2012 result - 0 for Romney states, 100 for Obama
# 2008=2008 state results (=Reality, negated for Obama loss of Indiana & North Carolina)
statewin <- read.csv("www.gwern.net/docs/2012-election-statewin.csv", row.names=1)
statewin
                    al       ak     az     ar     ca      co     ct      de
Reality         0.0000 0.000000 0.0000 0.0000 1.0000 1.00000 1.0000 1.00000
2008            0.0000 0.000000 0.0000 0.0000 1.0000 1.00000 1.0000 1.00000
Nate Silver     0.0000 0.000000 0.0200 0.0000 1.0000 0.80000 1.0000 1.00000
Drew Linzer     0.0000 0.000086 0.0000 0.0000 1.0000 0.98333 1.0000 0.98333
Margin of Error 0.0269 0.099800 0.4388 0.0451 0.9443 0.64710 0.9125 0.93770
Intrade         0.0000 0.000000 0.0600 0.0000 0.9500 0.55600 0.9900 0.96000
DeSart          0.0000 0.090000 0.0390 0.0000 1.0000 0.52300 0.9990 1.00000
Simon Jackman   0.0052 0.000000 0.0050 0.0000 1.0000 0.76520 1.0000 1.00000
Wang & Ferguson 0.0000 0.000000 0.0000 0.0000 1.0000 0.84000 1.0000 1.00000
Josh Putnam         NA       NA     NA     NA     NA      NA     NA      NA
Unskewed Polls      NA       NA     NA     NA     NA      NA     NA      NA
                   dc     fl     ga     hi     id     il indiana     ia      ks
Reality         1.000 1.0000 0.0000 1.0000 0.0000 1.0000  0.0000 1.0000 0.00000
2008            1.000 1.0000 0.0000 1.0000 0.0000 1.0000  1.0000 1.0000 0.00000
Nate Silver     1.000 0.5000 0.0000 1.0000 0.0000 1.0000  0.0000 0.8400 0.00000
Drew Linzer        NA 0.6040 0.0000 1.0000 0.0000 1.0000  0.0000 0.9966 0.03866
Margin of Error 1.000 0.4575 0.1972 0.9987 0.0086 0.9569  0.3273 0.6467 0.09710
Intrade         0.975 0.3300 0.0300 0.9750 0.0000 0.9890  0.0200 0.6630 0.00000
DeSart          1.000 0.4910 0.0300 1.0000 0.0000 1.0000  0.0020 0.7700 0.00000
Simon Jackman   1.000 0.5216 0.0014 1.0000 0.0000 1.0000  0.0000 0.8376 0.00000
Wang & Ferguson 1.000 0.5000 0.0000 1.0000 0.0000 1.0000  0.0000 0.8400 0.00000
Josh Putnam        NA     NA     NA     NA     NA     NA      NA     NA      NA
Unskewed Polls     NA     NA     NA     NA     NA     NA      NA     NA      NA
                      ky     la     me      md     ma     mi     mn         ms
Reality         0.000000 0.0000 1.0000 1.00000 1.0000 1.0000 1.0000 0.00000000
2008            0.000000 0.0000 1.0000 1.00000 1.0000 1.0000 1.0000 0.00000000
Nate Silver     0.000000 0.0000 1.0000 1.00000 1.0000 0.9900 1.0000 0.00000000
Drew Linzer     0.000000 0.0000 1.0000 1.00000 1.0000 1.0000 1.0000 0.07866667
Margin of Error 0.019100 0.0442 0.8403 0.96837 0.8988 0.6837 0.7149 0.13470000
Intrade         0.000000 0.0000 0.9300 0.94000 0.9950 0.8840 0.8490 0.00000000
DeSart          0.000000 0.0000 0.9930 1.00000 1.0000 0.9350 0.9610 0.00000000
Simon Jackman   0.000004 0.0000 1.0000 1.00000 1.0000 0.9998 0.9992 0.00000000
Wang & Ferguson 0.000000 0.0200 1.0000 1.00000 1.0000 1.0000 1.0000 0.00000000
Josh Putnam           NA     NA     NA      NA     NA     NA     NA         NA
Unskewed Polls        NA     NA     NA      NA     NA     NA     NA         NA
                      mo     mt     ne        nv     nh     nj     nm     ny
Reality         0.000000 0.0000 0.0000 1.0000000 1.0000 1.0000 1.0000 1.0000
2008            0.000000 0.0000 0.0000 1.0000000 1.0000 1.0000 1.0000 1.0000
Nate Silver     0.000000 0.0200 0.0000 0.9300000 0.8500 1.0000 0.9900 1.0000
Drew Linzer     0.000000 0.0000 0.0000 0.9993333 0.9980 1.0000 1.0000 1.0000
Margin of Error 0.447300 0.2436 0.0562 0.7710000 0.6886 0.8647 0.8579 0.9697
Intrade         0.050000 0.0500 0.0000 0.8370000 0.6490 0.9790 0.9390 0.9500
DeSart          0.052000 0.0080 0.0000 0.7680000 0.7560 0.9980 0.9740 1.0000
Simon Jackman   0.000004 0.0032 0.0000 0.9120000 0.8324 0.9998 0.9968 1.0000
Wang & Ferguson 0.000000 0.0000 0.0000 0.9900000 0.8400 1.0000 1.0000 1.0000
Josh Putnam           NA     NA     NA        NA     NA     NA     NA     NA
Unskewed Polls        NA     NA     NA        NA     NA     NA     NA     NA
                        nc     nd        oh     ok        or     pa     ri
Reality         0.00000000 0.0000 1.0000000 0.0000 1.0000000 1.0000 1.0000
2008            1.00000000 0.0000 1.0000000 0.0000 1.0000000 1.0000 1.0000
Nate Silver     0.26000000 0.0000 0.9100000 0.0000 1.0000000 0.9900 1.0000
Drew Linzer     0.08533333 0.0000 0.9986667 0.0000 0.9986667 1.0000 1.0000
Margin of Error 0.50030000 0.1284 0.6038000 0.0029 0.7886000 0.7562 0.9684
Intrade         0.23000000 0.0030 0.6550000 0.0010 0.9590000 0.8200 0.9500
DeSart          0.06600000 0.0000 0.7040000 0.0000 0.9430000 0.8810 1.0000
Simon Jackman   0.28120000 0.0000 0.9298000 0.0000 0.9726000 0.9910 1.0000
Wang & Ferguson 0.16000000 0.0000 0.9300000 0.0000 1.0000000 0.9300 1.0000
Josh Putnam             NA     NA        NA     NA        NA     NA     NA
Unskewed Polls          NA     NA        NA     NA        NA     NA     NA
                       sc     sd       tn     tx     ut     vt     va     wa
Reality         0.0000000 0.0000 0.000000 0.0000 0.0000 1.0000 1.0000 1.0000
2008            0.0000000 0.0000 0.000000 0.0000 0.0000 1.0000 1.0000 1.0000
Nate Silver     0.0000000 0.0000 0.000000 0.0000 0.0000 1.0000 0.7900 1.0000
Drew Linzer     0.1386667 0.0000 0.000000 0.0000 0.0000 1.0000 0.9760 1.0000
Margin of Error 0.1345000 0.1665 0.053100 0.0545 0.0035 0.9846 0.5046 0.8473
Intrade         0.0400000 0.0500 0.020000 0.0200 0.0450 0.9800 0.5800 0.9750
DeSart          0.0030000 0.0010 0.000000 0.0000 0.0000 1.0000 1.0000 0.9980
Simon Jackman   0.1290000 0.0068 0.000004 0.0000 0.0000 1.0000 0.7840 1.0000
Wang & Ferguson 0.0000000 0.0000 0.000000 0.0000 0.0000 1.0000 0.8400 1.0000
Josh Putnam            NA     NA       NA     NA     NA     NA     NA     NA
Unskewed Polls         NA     NA       NA     NA     NA     NA     NA     NA
                         wv     wi          wy
Reality         0.000000000 1.0000 0.000000000
2008            0.000000000 1.0000 0.000000000
Nate Silver     0.000000000 0.9700 0.000000000
Drew Linzer     0.001333333 1.0000 0.000666667
Margin of Error 0.042700000 0.6448 0.006900000
Intrade         0.020000000 0.7460 0.000000000
DeSart          0.000000000 0.8560 0.000000000
Simon Jackman   0.005400000 0.9698 0.000000000
Wang & Ferguson 0.000000000 0.9900 0.000000000
Josh Putnam              NA     NA          NA
Unskewed Polls           NA     NA          NA

brstate <- function(p) br(statewin["Reality",], statewin[p,])
lapply(rownames(statewin)[1:9], brstate)
  • Reality: 0
  • Drew Linzer: 0.00384326
  • Wang/Ferguson: 0.007615686
  • Nate Silver: 0.00911372
  • Simon Jackman: 0.00971369
  • DeSart/Holbrook: 0.01605542
  • Intrade: 0.02811906
  • 2008: 0.03921569
  • Margin of Error: 0.05075311
  • random (50%) guesser 0.25000000

State win vote-shares

Sources:

  • 2008 vote-share
  • 2012 vote-share
statemargin <- read.csv("www.gwern.net/docs/2012-election-statemargin.csv", row.names=1)
statemargin
                      al       ak       az       ar       ca       co       ct
Reality         38.42829 40.79253 44.44855 36.87899 59.69455 51.56536 58.38274
2008            38.80000 37.70000 45.00000 38.80000 60.90000 53.50000 60.50000
Nate Silver     36.70000 38.60000 46.20000 38.60000 58.10000 50.80000 56.60000
Drew Linzer     40.30000 37.50000 46.20000 37.10000 59.80000 51.20000 56.80000
Margin of Error 37.00000 41.00000 49.00000 39.00000 61.00000 53.00000 59.00000
Josh Putnam           NA       NA 46.59500       NA 58.39500 50.87500 55.92000
Unskewed Polls  37.78000 36.40000 43.95000 44.68000 57.65000 49.48000 54.55000
Intrade               NA       NA       NA       NA       NA       NA       NA
Simon Jackman   38.70000       NA 46.10000 36.40000 58.60000 51.00000 56.80000
DeSart          35.20000 32.20000 46.40000 38.70000 59.20000 50.10000 57.70000
Wang & Ferguson 42.50000 39.00000 46.00000 38.00000 57.50000 51.00000 56.50000
                      de       dc       fl       ga       hi       id       il
Reality         58.61074 90.91402 50.00787 45.48216 70.54523 32.62233 57.53322
2008            61.90000 92.90000 50.90000 47.00000 71.80000 36.10000 61.80000
Nate Silver     59.60000 93.00000 49.80000 45.50000 66.50000 32.10000 59.80000
Drew Linzer     61.00000       NA 50.20000 46.00000 65.60000 31.20000 60.20000
Margin of Error 60.00000 91.00000 49.00000 44.00000 70.00000 35.00000 61.00000
Josh Putnam           NA       NA 50.08000 45.38000       NA       NA 59.58000
Unskewed Polls  86.88000 57.40000 47.60000 43.20000 58.55000 30.95000 55.25000
Intrade               NA       NA       NA       NA       NA       NA       NA
Simon Jackman         NA 91.60000 50.10000 45.50000 65.00000 32.00000 59.60000
DeSart          60.50000 95.80000 49.90000 45.50000 66.60000 29.10000 60.80000
Wang & Ferguson 62.50000 90.00000 50.00000 46.00000 63.50000 32.00000 59.50000
                 indiana      ia       ks       ky       la       me       md
Reality         44.08345 51.9882 37.82721 37.80994 40.57746 55.96352 61.97419
2008            49.90000 54.0000 41.40000 41.10000 39.90000 57.60000 61.90000
Nate Silver     45.30000 51.1000 37.90000 40.30000 39.30000 55.90000 60.90000
Drew Linzer     44.30000 51.6000 41.10000 45.10000 39.70000 56.40000 61.30000
Margin of Error 47.00000 52.0000 41.00000 36.00000 39.00000 57.00000 62.00000
Josh Putnam     44.36500 51.2750       NA       NA 43.06500 56.17000 60.64500
Unskewed Polls  41.90000 49.8800 36.60000 40.80000 43.63000 51.90000 55.83000
Intrade               NA      NA       NA       NA       NA       NA       NA
Simon Jackman   44.80000 51.4000       NA 40.90000 38.90000 56.00000 61.00000
DeSart          43.10000 51.8000 39.40000 41.90000 39.30000 55.90000 61.90000
Wang & Ferguson 43.50000 51.0000 41.50000 44.50000 43.50000 55.50000 61.00000
                      ma       mi      mn       ms       mo       mt       ne
Reality         60.74886 54.30391 52.6497 43.54862 44.34962 41.70813 37.86805
2008            62.00000 57.40000 54.2000 42.80000 49.30000 47.20000 41.50000
Nate Silver     59.00000 53.00000 53.7000 45.60000 45.60000 45.20000 40.40000
Drew Linzer     60.00000 52.70000 54.2000 41.80000 45.30000 45.30000 42.50000
Margin of Error 58.00000 53.00000 54.0000 43.00000 49.00000 45.00000 40.00000
Josh Putnam     56.17000 52.78500 53.7650       NA 45.92500 45.46500       NA
Unskewed Polls  60.10000 51.75000 51.0300 39.83000 46.20000 38.80000 34.40000
Intrade               NA       NA      NA       NA       NA       NA       NA
Simon Jackman   59.70000 53.60000 54.0000       NA 45.30000 46.00000 42.80000
DeSart          62.80000 53.70000 54.3000 40.00000 46.10000 44.20000 39.80000
Wang & Ferguson 59.50000 52.75000 53.7500 44.00000 45.25000 45.75000 43.00000
                      nv       nh       nj       nm       ny       nc       nd
Reality         52.35625 51.98268 57.85939 52.99547 62.62461 48.35097 38.69731
2008            55.10000 54.30000 56.80000 56.70000 62.20000 49.90000 44.70000
Nate Silver     51.80000 51.40000 55.50000 54.10000 62.40000 48.90000 42.00000
Drew Linzer     52.20000 51.60000 56.60000 54.40000 63.20000 49.10000 41.70000
Margin of Error 55.00000 53.00000 57.00000 57.00000 62.00000 50.00000 43.00000
Josh Putnam     52.02500 51.51500 56.18000 54.56500 62.51000 49.22000       NA
Unskewed Polls  52.15000 50.03000 53.80000 53.53000 58.75000 44.98000 37.15000
Intrade               NA       NA       NA       NA       NA       NA       NA
Simon Jackman   51.90000 51.30000 56.00000 54.40000 62.70000 49.20000 43.00000
DeSart          51.80000 51.70000 57.10000 54.70000 64.60000 47.70000 40.10000
Wang & Ferguson 52.50000 51.00000 56.00000 53.00000 62.00000 49.00000 43.00000
                      oh       ok       or       pa       ri       sc       sd
Reality         50.14323 33.22768 54.30016 51.75834 62.70096 44.08803 39.86614
2008            51.20000 34.40000 57.10000 54.70000 63.10000 44.90000 44.70000
Nate Silver     51.30000 33.80000 53.60000 52.50000 61.80000 43.20000 42.50000
Drew Linzer     51.60000 33.50000 53.60000 52.70000 63.10000 44.30000 44.80000
Margin of Error 52.00000 31.00000 55.00000 55.00000 62.00000 43.00000 44.00000
Josh Putnam     51.47000       NA       NA 52.84500       NA       NA 44.79000
Unskewed
Related searches:
reality silver intrade margin linzer
gipoco.com is neither affiliated with the authors of this page nor responsible for its contents. This is a safe-cache copy of the original web site.