Strata/Hadoop World speakers set high expectations for big data


After two days at the Strata Conference & Hadoop World all I can say is, Whew! Big data sure has some high expectations to meet.

I get it. Big data can be transformational. And the same innovative geniuses who built the infrastructures that have already transformed society into an electronically tethered and social community have spread throughout the big data industry to do it again. And they are aiming higher, much higher.

The enthusiasm at this event was electric to say the least. On one hand--or in one ear--you hear the excitement bubbling over in the voices of big data developers and entrepreneurs such as Eldad Farkash, co-founder and CTO of SiSense, as he describes the beauty of the columnar algebra he used to develop ElastiCube, the technology at the heart of his company's business intelligence software; or when Todd Papaioannou and Jonathan Gray, co-founders of Continuuity (which launched this week) tell of their backgrounds at Yahoo and Facebook (NASDAQ: FB) and how they brought their experience forward to create a big data application fabric for developers. Truth be told, Papaioannou's delivery was more subdued than ebullient, but nonetheless compelling.

In the other ear, you could hear industry leaders, such as Cloudera CEO Mike Olson, trumpeting the big problem solving capabilities of big data and the big questions it will answer: particle physic and the secrets of the universe, cancer, neurological disease, tax and other governmental policy, the education crisis, energy, world hunger, the genetic code and terrorism.

Whew! It is at this point one has to step back and look at where we are now, take a deep breath and proceed step by step, with the occasional giant leap, which Cloudera may have done with its new Impala real-time engine for Hadoop. We'll see.

Don't get me wrong. It is great to think big. The big ideas are what attracted me to big data and what feeds the hype about it, which in turn feeds the investment. But as I kept turning my ears to more and more of the players at the conference, it was clear most of them are focused on solving smaller issues directly for the business community and enabling those who might come up with the big cure or the big discovery. It sounds like a good first step.

There is one other barrier I see in front of those who want to change the world with the empirical evidence uncovered by big data. I see no evidence that we, in the U.S. anyway, are a society that accepts empirical evidence, especially if it conflicts with our closely held beliefs. We reject biological evolution, we reject the idea of climate change, we allow special interest groups to tell us that cigarettes don't cause cancer or that coal can be clean. We accept Jenny McCarthy's conclusions about vaccinations over the National Institute of Health's. We doubt the age of the planet and the universe. So good luck with that.

Businesses on the other hand, have no choice but to accept the evidence, and are clamoring for it. - Tim

Filed Under