Category: Data

Nate Silver's Election Map

Nate Silver's Election Map

President Obama’s reelection last night happened amidst a confluence of trends in our nation’s demographics, economics, even some-would-say our meteorology. The most important of these is not a trend, but a meta-trend: the true onset of the Data Era. Should Florida go to the President (as it is leaning), Nate Silver, long-time author of the fivethirtyeight blog, will have correctly predicted all 50 states’ presidential vote outcomes, and all but two of the senate races. This accuracy marks a triumph of Statistics and Data Science, and signals the future for our elections, and many other things.

Data Data everywhere
It is important to note that we have long honed our ability to analyze data, with required statistics classes for many different majors on a College campus (hello #RITEMBAStats!). In fact, Silver’s method includes analysis of error using sum of squares just the same as introductory stats students learn in their regression unit. Over the years, “Database Marketing”, “Data Science”, “Big Data”, “Business Intelligence”, “Data Analytics” and others terms have become catch phrases to refer to the trend of analyzing data to discover useful patterns.

While our attention to data has existed for decades, our ability to make such visible results on a grand scale is relatively recent. The biggest reason for this is the new availability of data to large audiences. The most obvious source of this new information is social media. Twitter, for example, hit a new peak of 66,019 tweets per minute around 9:30pm last night. This massive data set is continuously broadcast for anyone who is interested to read, or analyze it. Many commercial ventures study it closely. Dell Computer’s 1st Chief Listening Officer Susan Beebe uses a wide variety of analysis tools to track the vast array of social media data relevant to her company.

Another major source of this new information are the companies that actively collect, and then share, their data. In 2009, the $1 Million NetFlix prize was awarded to team BellKor’s Pragmatic Chaos for their algorithm predicting how well a viewer will enjoy a movie based on their movie preferences. Since then, crowd-sourced data analysis has become mainstream with websites like Kaggleoffering new competitions as well as educational materials to enhance our ability to predict the future based on the past. A quick read of Kaggle’s forums provides a list of new analysis tools that are available (and inexpensive) like the programming language R, Wolfram Alpha or Orange.

Moneyball Poster

Statistics with a relatable face

The signs
But it is none of this that signals the new era of data. While some have argued how boring and uninteresting the details are, it is hard to miss how prevalent all of this has become. The opposite is in fact true: the advent of the Data Era is this moment when the data, the tools and people have created a user interface that everyone can see, understand and embrace. At its most mundane, we see this in an exploding number of “infographics” explaining all manner of information in a visual format that is appropriate for audiences with a short amount of time. We know we could present tables full of numbers but people won’t consume them. An election map is known to be the way to go.

But it goes much further than infographics. A 2009 NY Times article quotes Hal Varian, chief economist at Google as saying “I keep saying that the sexy job in the next 10 years will be statisticians…and I am not kidding.” Statisticians, or at least their output, have become increasingly visual and widely embraced. The recent hit movie, Moneyball, shows the impact of statistical thinking on baseball while providing Brad Pitt on the posters.

Nate Silver has effectively become a rock-star among the data-connoisseurs and others. He appeared this year on The Daily Show, the Colbert Report, and other television programs all the while authoring a bestselling book. His name was trending on Twitter during the election results coverage, and a search on the #natesilverfacts hashtag will show you how heroic he has become.

Naters gonna Nate

Nate Silver meme

In the wake of the 2012 election, we enter a new era. Nate Silver has become a modern hero by predicting politics and baseball with remarkable accuracy, but it is the accessibility of his thinking that we should pay attention to. Having passed through this moment, there is no way back. The availability of data, tools and new people to work with them is only increasing going forward. All elections will have simulation-powered visual, interpretable results. Statisticians will be much more commonplace and seen as they explain sophisticated computer models.

It is our ability to make the esoteric details of an analysis clear through our charts, words and actions that has crossed through this threshhold.

Sources:
For Today’s Graduate, Just One Word: Statistics By STEVE LOHR

Pundit Forecasts All Wrong, Silver Perfectly Right. Is Punditry Dead?

Meme from: @riondotnu