The gutless wonder of big data


Burson-Marsteller said in 2006 that 62 percent of CEOs rely heavily on gut feelings or intuition when making business decisions, especially when there is no clear direction to choose. If big data has its way, the gut feeling, women's intuition, hunches and other touchy-feely ways of decision making will go the way of the divining rod.

With a bounty of data and the right analytic tools, who needs guess work? As we saw in this week's elections in the U.S., even small data is moving us in that direction, if you consider polling data to be small. Most pundits who felt there was "something in the wind" or had a gut feeling that this factor or that factor was going to turn the tide in one direction or another woke up yesterday morning looking a lot less smart than they did the day before.

Nate Silver didn't wake up that way. The projections on his New York Times blog fivethirtyeight, were 100 percent correct across the 50 states in the presidential contest. Even his detractors up in Romney headquarters in Boston had to admit: that boy's wicked smaht.

His accuracy, as well as that of some other analysts, did not make a statement about big data, but they did make a statement about the power of facts and algorithms over hunches. If Silver's hot streak continues he may become the poster boy for all budding data scientists. He's been so spot-on that people are now starting to run the numbers on him. And it turns out that knowing how to "do the numbers" pays off. Silver's blog accounts for 10 to 20 percent of the New York Times' political traffic and just prior to the election accounted for 71 percent.

Ezra Klein, another prognosticator, as well as a columnist for The Washington Post, said pundits were the big losers in the election because they lost credibility as the weakness of their gut feelings was exposed. The numbers guys doing the real analysis gained new respect. "Anything can happen, but in 2016 readers would best put their confidence in hard poll numbers as opposed to the 'gut feeling' of someone on a cable news network," Klein wrote.

2016 is right about the time the pioneering cohorts from all the newly launched data science programs around the world will be graduating. For those who went into their chosen field because they fell for the hype about being big data rock stars when they came out, the 2012 election can provide a valuable lesson. Yes, the insights they will extract from the growing mountains of data can be invaluable. But if they--and big data itself--are to be successful, if they are going to instill the confidence required for the world to accept the conclusions they draw, these data scientists will need more than wicked math skills. They will need an ethical foundation strong enough to resist biases. They must shun popularity and let the numbers do the talking. Data scientists cannot be like biased pollsters who go out and find the proof its clients are looking for; they have to find the best truth available based on the best data available. Their careers will be short lived if they do otherwise.

The Nate Silvers of the big data world will be hard pressed to resist the corporate and institutional cash that urges them to find specific proof points while disregarding others. If big data turns into just another tool for the spin doctors, then big data fails. And we all lose. - Tim