The question big data can't answer: why?


Make no mistake, big data renders more reliable insights and delivers them faster than any traditional means of analysis. At least it does when it's done right. But that doesn't mean that big data analysis can tell you everything you need to know. Indeed, it most often fails at delivering the one thing you most need to know: why? From consumer sales to public uprisings, big data can only tell you what is happening and not why it's happening. Without knowing the why behind the what, the actions you take on big data insights can easily and woefully be off course.

To correct this shortcoming, many companies and governments are broadening their intelligence efforts to include the human sciences, i.e. anthropology, sociology, political science and philosophy.

"This new approach is finding its way into the labs of technology companies such as Intel, IBM and Samsung; the marketing departments of large consumer-product companies such as Adidas, Lego and Procter & Gamble; global health care companies such as Novo Nordisk and Pfizer; and the thinking and writing of business leaders and new breeds of consultancy that, like our own, merge hard and soft sciences," writes Christian Madsbjerg, director of client relations at the innovation and strategy consultancy ReD Associates, and Mikkel B. Rasmussen, director of ReD's European division, in their post in Harvard Business Review. Madsbjerg and Rasmussen are also coauthors of The Moment of Clarity: Using the Human Sciences to Solve Your Toughest Business Problems (Harvard Business Review Press, 2014). Their HBR post was developed from their book.

You may wonder why finding the motivation behind human actions cannot easily be done with the hard sciences. The answer is that the motivation is often soft and hidden and even sometimes illogical, e.g. emotion based or socially pressured. Sometimes the motivation is so complex that we give up and call it luck

Whereas some human emotion can be detected in data, primarily through efforts such as Semantic Search, Semantic Web and semantic ranking algorithms, these are largely limited to context and intent rather than to motivation. And while sentiment analysis can reveal the public mood in a moment in time, it does not necessarily reflect the cause of the mood nor predict future mood swings.

In short, the level of complexity in human action is beyond the abilities of the hard sciences alone--at least it is at this moment.

"According to a recent global study of 1,500 CEOs conducted by IBM, the biggest challenge those CEOs face is the so-called complexity gap," writes Madsbjerg and Rasmussen. "Eight out of 10 expect the business environment to grow in complexity, but fewer than half feel prepared for the change. The research also reveals that CEOs see a lack of customer insight as their biggest deficit in managing complexity. They prioritize gaining customer insight far above other decision-related tasks and rank 'customer obsession' as the most critical leadership trait."

"Accordingly, many companies are turning to customer research that is powered by big data and analytics," they continue. "Although that approach can provide astonishingly detailed pictures of some aspects of their markets, the pictures are far from complete and are often misleading. It may be possible to predict a customer's next mouse click or purchase, but no amount of quantitative data can tell you why she made that click or purchase. Without that insight, companies cannot close the complexity gap."

And that's precisely so. Certainly data scientists will one day routinely add emotion and motivation inputs to algorithms in order to get a better view of customers and the business environment. But that day isn't here yet.

First we have to learn how to capture and analyze the why behind human actions, then we can better use this information to fine-tune our analysis. It's time to bring in the human scientists. - Pam