The day big data kills the anecdote


Pick any given society and study them awhile and you'll likely find within them an emotional lot. They might be the majority or minority, but in every case they emotionally make decisions based on anecdotal evidence consisting of some correlation that is as far from true causation as Jupiter is from Earth.

Then this "evidence" and ignorance spreads like a disease and before you can say "Wait! Here are the facts!" people are suffering and dying needlessly. One example of this is the Anti-Vaccination Movement that increasingly makes mankind unnecessarily vulnerable to illnesses we could otherwise vanquish.

We often squeal in alarm at the dangers of big data, particularly in regards to the loss of privacy, all the while overlooking the major benefits to be found in it. One of those benefits is public access to hard information that can finally kill the anecdotal lie or factual omission for the betterment of all.

"As a single example, I wrote about the release of one such dataset on hospital pricing releases last year by the Government here," writes Dan Munro in his post in Forbes. By making hospital pricing available to the public for the first time EVER, the public became empowered to choose the costs they'll pay for a particular health service. No longer will they be at the mercy of hospitals and potentially bankruptcy courts soon after. The point is that people no longer have to pick a hospital based on Aunt Betsy's experience there or Uncle Joe's review of the food--all while blindly accepting that the pricing is fair.

But getting back to the Anti-Vaccination Movement example: Munro points to another recent dataset release, this one by the Council on Foreign Relations, wherein "a [interactive] chart shows vaccine preventable outbreaks around the world from 2006 to present day." Aaron Carroll writes an excellent post with charts and video at the Incidental Economist on precisely how valuable this information is to the public.

Now, one must remember at this point how closely people tend to embrace and guard their own biases and beliefs even in the face of overwhelming evidence to the contrary. So, there will always be some people who insist the data is wrong or there is some conspiracy afoot manipulating the results rather than change their emotionally fueled belief. In the end, truth does not always "will out."

Those people will remain a danger to their community, particularly those who befriend disease by so stubbornly presenting their own bodies and that of their children as hotbed incubators for microscopic people killers.

However, the presence of overwhelming scientific evidence that can now be made available in real time to the public at large will likely diminish the power of the anecdote and curtail the emotional spread of ignorance. That is the underlying difference between the Information Age and the Knowledge Era that I so often preach about.

In other words, it's about to become exceedingly difficult to point to science as "opinion" and to publicly spout nonsense and get away with it. One day soon, big data will finally kill the anecdote as the "evidence of choice" for the ignorant and leave the bearers as the public laughing stocks they deserve to be.

Meanwhile, we'll have a healthier and better informed society. That alone is worth praising the advent of big data. - Pam