Guardian blogger Enten answers questions about politics and statistics
Andrew Pham: A lot of the hype on the Internet these past 48 hours has been about Nate Silver and his perfect prediction of the Electoral College results, but Sam Wang of Princeton nailed the popular vote and predicted just as well as Silver did. Do you have any thoughts on either of them?
Harry Enten: All of these smart men are only as good as the polling data. It all comes back to who did a good job for the pollsters: Jay Leve at Survey USA, Doug Rivers and his team at YouGov and Jon Cohen at Washington Post. All of those guys had excellent data and nailed every single state that they polled. In reality, the aggregators had an excellent time. That to me is the real story. If you averaged Sam Wang, Pollster, Simon Jackman and so forth, their combined results are actually better than what any one of them predicted individually.
AP: Silver's algorithm seems a little hacky, relying on statistical massaging with poll data and inference from econometric variables.
HE: There has been a question in the community about Nate. He’s not open, and that was well known after he stopped with the baseball predictions. It was found that his complex system wasn’t really necessary. There’s certainly something to be said about a model that’s needlessly complex. An elegant model, something like a Sam Wang or a Simon Jackman, would be very ideal. Jackman was able to produce an estimate that was very accurate. Then again, Nate came up with a popular vote estimate that was very accurate. Nate took into account effects that weren’t related to polling, such as certain demographics of each state, but by throwing in the extra complexity as opposed to just simply referring to the poll data, he might have cost himself a couple of Senate seats.
AP: Do you think that, as respect for electoral statistics becomes more pronounced, that there will be a warping effect on model predictions, like reducing turnout for the predicted winner?
HE: I think one of the things we know to be true is that people like to vote for a winner, and there’s always a fear that there’s going to be a bandwagon effect. But I really do think that at the end of the day, most of the people who decide on the elections — the undecided voters — don’t read into the numbers that much when they go to the polls. So I don’t think statistics will affect the final numbers as much as one would believe.
AP: What processes do you think will be revolutionized by data?
HE: We’ll have greater voter turnout because of more data. Targeting swing voters has been becoming not just an art but also a science. That’s because if you know the profile of what an Obama voter looks like, you can call him up and make sure he gets every opportunity to go out and support his choice. The Democrats did that this time. They spent millions reaching out to the voters. The Republicans weren’t as good about it, but they will get better. Companies like Bitly collect a lot of data to figure out people’s preferences. Really, what it will come down to in terms of data mining is that we’ll get really good at profiling consumers and figuring out how we can market around that. It’s a scary thought, but we won’t be telling people what they ought to think, just bringing them more of what they want.