Data scientists' new role: Big data app developers


Source: Pixabay

Big data is growing up. The technical struggles from its infancy are falling by the wayside as needs shift to those befitting a tech toddler. The shift has moved from a focus on Hadoop cluster configuration to big data apps. Accordingly, the work of data scientist is shifting to big data app developer.

"The Hadoop cluster has become a commodity," writes Tom Phelan, co-founder and Chief Architect at BlueData, in his post. "Let me put this another way: compare a Hadoop cluster to your phone. In the beginning, getting a phone to work at all was considered a technology marvel. Now phones are ubiquitous and mobile. They come in all shapes and sizes and colors. No one really cares about the inner workings of the phone."

"Today, they care about the apps that run on the phone; the App Store is what drove the phenomenal success of the iPhone. Data scientists are the application developers of the Big Data world; what really matters are the applications that they use and develop. And that's the future of Big Data: it's all about the apps."

Where once the focus was on the underlying Hadoop infrastructure, now a flexible infrastructure is considered a given. It's what data scientists do on top of that foundation that matters most now. And, yes that would be analytics and apps.

So, while Hadoop skill gaps stubbornly remain this year, according to a Gartner survey, and thus traditional data scientist skills remain in high demand, those who gain skills in big data analytics and big data app development will be in most demand. Those versatile data scientists will also be best positioned for job security and career advancement.

"It's no longer just about how to spin up a Hadoop cluster," writes Phelan. "Now they ask how to deploy the latest version of Platfora, or how to run a version of Hive built from source code customized to their needs, or how to run three different versions of Spark on three different clusters simultaneously. Oh, one more thing: this needs to be cookie-cutter simple."

Yes, Phelan is pitching BlueData's product for simplifying big data infrastructure. But he's nonetheless correct in his 'future of big data' call on big data apps.

What will come after that? Why, integrating those apps of course. Look for big data apps to follow a path similar to mobile and enterprise app progressions.

For more:
- see this BlueData post
- see this Gartner survey

Related articles:
The rise of algorithms for your algorithms
On the money trail: Data scientist skills in hottest demand this year