Big data: The next 10 years


In one of many discussion groups on LinkedIn about big data and analytics, the question was posed: Where do you see big data in 10 years?

There was a time in, say, the world of telecom, when the lifecycle of transformational new networking technology was approximately 25 years. Asking where the industry and the society it served would be in ten years was not a difficult question. It's different now.  

For example, it was just seven years ago as we rung in the New Year in 2006 that the computing, networking and communications industries were all agog over this new thing called virtualization. Email was still hot. Hosted services were making another comeback with no mention of cloud computing. And Jack Dorsey had not yet created Twitter. Today, Twitter has over 500 million active users and generates 340 million tweets per day. It is one of the most visited websites on the Internet. The world of communications is profoundly different than it was seven years ago.

So where will big data be in 10 years? The discussion group had some rather high expectations.They ranged from simple, but continuous improvements like getting bigger, faster and more intelligent, to the most ambitious (and scary), like living memory embedded in liquid crystal by smart nanomachines.

There were some detractors who said big data will be ancient history in 10 years or that by then the industry will at least have settled on a consistent definition. Some said it would turn us into idiots (hopefully happy idiots), but most visions were very optimistic, saying the technology would be mature and woven into the fabric of our lives. It will pick presidents and make stock and investment decisions for us, and may even be imbedded within us and guide our daily decisions.

For the most part, all the high expectations can be tempered by knowing the comments all come from people with either vested business interests or elevated personal interest in the success of big data. Those of us in that position tend to focus on the capabilities of the technology, as if it were to proceed unimpeded by resistance from the market. And there is and will be plenty of resistance to big data, and plenty of other factors that could deter its meteoric rise.

The most significant potential impediment is the market success of the technology itself. As one commenter in the group said, "If the value from big data is not extracted in a timely manner, it becomes irrelevant with time."

It is unclear how much time big data will be given, but it is safe to say it has less than 10 years to prove itself in the marketplace. At the same time, it will have to mount a protracted defense of the power it has unleashed for companies, governments and fellow citizens to violate our privacy. Somewhere along the line, uncertainty around the rules and regulations could begin to adversely affect investment in big data analytics.

Perhaps the biggest unforeseen barrier to big data realizing its potential comes from a caveat expressed in the first line of a 1969 pop-rock classic by Zager & Evans: In the year 2525, if man is still alive.

Maybe big data can help us survive until then. Maybe it can help us survive the next 10. Maybe it can survive the next ten itself. But nothing survives unchanged. So in 10 years, big data may indeed be far from where we envision it today, but whatever it becomes, it will have a profound impact on how we relate to things. Most other network-based technologies have altered the way we communicate. Big data is aiming much higher. It aims to change the way we see the world. That's big. - Tim

Filed Under