The secret side of big data

Tools

For every step forward in big data, it seems there is a corresponding step if not backwards, then certainly sideways, in the public's perception of the technology and its bipolar ability to generate anguish even as it solves seemingly intractable problems like this.

According to the Wall Street Journal, the United States government has granted the National Counterterrorism Center sweeping authority to store and monitor massive amounts of data about law-abiding Americans. The NCC was established in 2004 based on recommendations from the 9/11 Commission. Its mission is to "Lead our nation's effort to combat terrorism at home and abroad by analyzing the threat, sharing that information with our partners, and integrating all instruments of national power to ensure unity of effort."

The center analyzes all terrorism-related intelligence and coordinates the sharing of information between agencies and other governments. It also establishes the IT architecture for sharing that information. A Biometric Update article this week called it a clearinghouse for big data and said new guidelines issued by the White House earlier this year "allow the agency to collate information about citizens from diverse sources such as state and local government databases, flight records, casino-employee lists, and immigration records, which incorporate personal and sensitive biometric data ... and develop predictive pattern-matching techniques that will be able to determine suspicious patterns of behavior." Under the new guidelines, these data sources can be provided to trusted foreign governments such as Canada for analysis.

Biometric anticipated an acceleration of big data initiatives in North America that "link government databases with more innocuous information, collected from public sources, such as the Internet. With an acceleration of such efforts, residents in both countries can expect a loss of anonymity, along with being tracked at unprecedented rates by law enforcement and intelligence agencies, without warrant, and with lack of effective external legal, independent oversight."

In a recent speech to the American Bar Association in May, Matthew Olsen, director of the National Counterterrorism Center, said, "We face adaptive adversaries, so we must be adaptive ourselves as we seek to confront and defeat them.  And that requires the right tools--including legal tools--that are up-to-date, relevant to the fight, and flexible enough to keep pace with an agile, determined, and unpredictable adversary."

He said al-Qaida in Pakistan has made a concerted effort to encourage extremists in the United States to carry out attacks. "Our mandate is to integrate and analyze all terrorism information collected across the government--that includes information collected overseas and information collected within the United States. Our analysts examine this information to make connections between pieces of data that at first blush may have no apparent link to a terrorist plot."

That's big data.

The NCTC replicates the data so that all necessary information can be aggregated and searched in one place by NCTC analysts and those they can share the data with. As the rest of the big data world is realizing, information that is not initially believed to relevant may later become critical based on subsequent events or information. For that reason, Olsen said, it made no sense to promptly delete information that could be important later on. Certain data sets needed to be retained for a longer period of time in order to ensure that terrorism information was not deleted simply because its significance was not immediately apparent.

The new guidelines allow the agency to keep such information for up to five years. While Olsen assures that the data is limited to people with a demonstrated need to know, and that the databases are subject to monitoring, recording and auditing requirements that prevent the dissemination of sensitive data, that's where the concern starts for some.

Matt Sledge, of the Huffington Post, described the agency this way last week: "An obscure counterterrorism agency now has access to an array of data on ordinary Americans so vast and with so few restrictions that critics are likening it to the "pre-crime" squad in the movie "Minority Report."

Even people from Homeland Security and the Justice Department were opposed to it, according to Chris Calabrese, the legislative counsel for privacy issues in the American Civil Liberties Union's Washington office, who was quoted in the Post article. "It's really telling that people who were very close to the program in a number of agencies were troubled by it. They're seeing how it's going to operate, they know what's going on with all this information and they're ringing the alarm bell," Calabrese said.

This won't be helpful for businesses and non-governmental institutions hoping to leverage big data technology to advance their own agendas.

Related Articles:
Strata/Hadoop World speakers set high expectations for big data
When big data becomes big brother: New York City's system for crime prevention
The PATRIOT Act: Shouldn't we be stunned and angry already?