Big data's big year


Welcome to 2013, the year big data begins to show the world its true value, the year headlines questioning the terminology and the efficacy of big data disappear, the year big data technology companies begin to show they are not the valueless dot-com hopefuls of the 1990s, nor even the overblown social stars of the new millennium.

This is the year big data begins to pay its way by enabling smarter business decisions, discovering previously unseen or misperceived realities, identifying dangerous trends in medicine and helping to discover new treatments, maybe even cures. It is when data scientists will begin finding solutions to intractable problems and do all sorts of statistical magic that can do just about anything short of compelling people to act.

This also is the year big data escapes its geekdom bubble and gets noticed by the masses, which is both good and bad. It is good in that it could inspire a dramatic shift in thinking that turns society from a population of wishful thinkers to one that relies on evidence-based reasoning. However, the down side is that they will begin to realize that they themselves are the source of most of the data creating these insights and making these new discoveries, and they will have to come to grips with the implications.

In this week's newsletter are articles showing both how the industry is progressing and how new realities about privacy are coming to light. One article shows how early work at the Cleveland Clinic led to the spinoff of a platform company (Explorys) that serves the broader health care community, while another shows the potential roadblocks ahead regarding privacy, particularly Fourth Amendment search and seizure violations, and new methods for shoring up the fallible practice of anonymising data.

This privacy issue is going to be a stickler. It pays to be a little paranoid in this regard and I was made even more paranoid over the holiday as I read a book Santa left in my stocking. It was David Brin's latest science-fiction novel, "Existence." (You may also care to read his 1999 non-fiction book on privacy called "The Transparent Society: Will Technology Force Us To Choose Between Privacy And Freedom?").

In "Existence," Brin paints a future picture of the connected world than shows how even the best efforts at privacy can be corrupted. One character explains how:

"All that's needed is to break reciprocity. By controlling information, making sure it flows one way. Take over the databases. Trump up panic situations, so the public will support paternalistic 'protections.' Make sure lots of privacy laws get passed , then bribe open some back door so elites can see it all anyway, and 'privacy' only protects them."

That sounds an awful lot like what is being described in the Atlantic article referenced here.

Supporting strong mutual privacy laws is not anti-big data. It is quite the opposite. Only by earning the trust of the public and the government will big data practitioners be left to work their magic. - Tim