Big data and the disputation arena


Even though I was told by a famous and favorite writer of mine (may he rest in peace) to never allow myself to be a fan of him or anyone else--I don't always heed his advice. I am weak that way. I like being a fan. So, I remain a fan of a different writer, a man for whom I justify my fandom by fostering the untruth that his work is, after all, "just fiction" and worse still is "science fiction," and therefore entertainment, which makes being a fan acceptable.

But readers of David Brin and some other futurist fiction writers in his league--there aren't many--know that there is more to learn about ourselves in these novels than in most works of so-called literature. But I digress. Learning more about us and our environment is the real point to reading books. That they may be entertaining is a bonus. It is not a stretch to say big data has that same potential.

Beyond its business and marketing applications, big data promises deep insights into us and our environment. Expectations are high for it to solve a myriad of problems in health care, education, business, space exploration, disease prevention, genomics, climate and more. But as we have seen over the last few years, something else needs fixing: our politics. Can big data help? And what does Brin have to do with this?

Big data has only just begun. As machine learning advances, as we put the capabilities of DNA or other more traditional technologies into play for storage, retrieval and processing, their capabilities will outpace our ability to come up with meaningful and helpful ways to use it. Hopefully, not for long. When it comes to our politics there was an idea floated by Brin in 2000 that seemed innovative and reasonable but far-fetched, idealistic and unwieldy from a technological perspective and destined to exist only in his novels.

His idea was to leverage the Internet to maximize its power to liberate the public, bypass older, dysfunctional systems of social authority, resolve disputes (locally, nationally, even globally) and perhaps even create law based on the efficacy of ideas proposed in four accountability arenas: science, democracy, courts and markets.

I won't do it justice in this short space so take a look at the full article here. In short, we can imagine how chaotic it would be for seven billion people to weigh in electronically on every disputed topic or need for law. But Brin's disputation arenas would rely on the emergent properties within chaos theory that allow order to emerge and popular opinions and arguments to rise to the top. Even Brin said, in 2000, concerning the Internet's ability to extend this self-organizing marvel to new heights: "Alas, despite the glad cries of cyber-utopians, today's Net just doesn't look ready. Not yet."

Well it's getting damn close, close enough--thanks to mass storage, new tagging techniques, the ability to manage unstructured data and the cloud--to begin contemplating the new arenas today.

Here's more: Disputation arenas would be online versions of the four major man-made social structures of our time: science, democracy, the justice system and fair markets. None of these were established as is. They bubbled up through years of competing ideas, challenges, persuasion and ultimately agreed upon rules for self-regulation. No, none are perfect, but they are accountable. The Internet will soon be ready to support this fifth arena: the disputation arena, which allows for the "potential for every fallible idea to face relentless scrutiny ... and where champions [of these ideas] can have it out, where ideas may be tested and useful notions get absorbed into an amorphous-but-growing general wisdom."

In other words, no more Fox News, no more MSNBC, no more dysfunctional congress who lacks comprehension on many of the important issues. The public would debate the merits of an idea and big data could track and rank the winning arguments. Today, these arguments take place in a vacuum without being pitted against each other, without being challenged and most importantly, without accountability.

Brin calls the disputation arena a "partner to democracy--society's messy process for floundering toward consensus and policy." He said it "won't be to award 'victory' to any one side, but to create an atmosphere of practical problem-solving, helping moderates understand each others' concerns and reach for some mutually beneficial consensus, leaving fanatics isolated and impotent at the wings." That's a good enough reason for the purveyors of big data to give him a call and draw up a plan. - Tim