Welcome!

@DevOpsSummit Authors: Liz McMillan, Yeshim Deniz, Zakia Bouachraoui, Pat Romanski, Elizabeth White

Related Topics: @DevOpsSummit, Java IoT, Linux Containers, Open Source Cloud, Containers Expo Blog, Agile Computing, @CloudExpo, Apache, @DXWorldExpo, @ThingsExpo

@DevOpsSummit: Article

Apache #Hadoop and #BigData Standards | @CloudExpo #IoT #M2M #BI #ML

The platform’s penetration into enterprises has not lived up to Hadoop’s game-changing business potential

Making Apache Hadoop Less Retro: Bringing Standards to Big Data

Ten short years ago, Apache Hadoop was just a small project deployed on a few machines at Yahoo and within a few years, it had truly become the backbone of Yahoo's data infrastructure. Additionally, the current Apache Hadoop market is forecasted to surpass $16 billion by 2020.

This might lead you to believe that Apache Hadoop is currently the backbone of data infrastructures for all enterprises; however, widespread enterprise adoption has been shockingly low.

While the platform is a key technology for gaining business insights from organizational Big Data, its penetration into enterprises has not lived up to Hadoop's game-changing business potential. In fact, according to Gartner, "Despite considerable hype and reported successes for early adopters, 54 percent of survey respondents report no plans to invest [in Hadoop] at this time, while only 18 percent have plans to invest in Hadoop over the next two years," said Nick Heudecker, research director at Gartner.

These findings demonstrate that although the open source platform may be proven and popular among seasoned developers who require a technology that can power large, complex applications, its fragmented ecosystem has caused enterprises difficulty extracting value from Apache Hadoop investments.

Another glaring barrier to adoption is the rapid and fragmented growth happening with Apache Hadoop components and its platform distribution, ultimately slowing Big Data ecosystem development and stunting enterprise implementation.

For legacy companies, platforms like Apache Hadoop seem daunting and risky. If these enterprises aren't able to initially identify the baseline business value they stand to gain from a technology, they are unlikely to invest - and this is where the value of industry standards comes into play.

Increasing adoption of Apache Hadoop, in my opinion, will require platform distributions to stop asking legacy corporations to technologically resemble Amazon, Twitter or Netflix. Through compatibility across platform distribution and application offerings for management and integration, widespread industry interoperability standards would allow Big Data application and solution providers to offer enterprises a guaranteed and official bare-minimum functionality and interoperability for their Apache Hadoop investments.

Additionally, this baseline of technological expectation will also benefit companies looking to differentiate their offerings. Similarly, standards within this open source-based Big Data technology will enable application developers and enterprises to more easily build data-driven applications - including standardizing the commodity work of the components of an Apache Hadoop platform distribution to spur the creation of more applications, which boosts the entire ecosystem.

A real world illustration of standardization in practice occurs within the container shipping industry, which was able to grow significantly once universal guidelines were implemented. When a formal shipping container standard was implemented by the International Standards Organization (ISO), to ensure the safe and efficient transport of containers, its significant impact increased trade more than 790 percent over 20 years - an incredible case for the unification and optimization of an entire ecosystem to ensure its longevity.

To help today's growing enterprise buyer looking to harness the estimated 4ZB of data the world is generating, the open data community will need to work together to foster the support of standardization across Apache Hadoop to ensure confidence from new adopters in their investment - regardless of the industry they serve.

From platform distributions, to application and solution providers and system integrators, known standards in which to operate will not only help to sustain this piece of the Big Data ecosystem pie, but it will define how these pieces interoperate and integrate more simply for the benefit of the ever-important enterprise.

More Stories By John Mertic

John Mertic is Director of Program Management for ODPi and Open Mainframe Project at The Linux Foundation. Previously, he was director of business development software alliances at Bitnami. He comes from a PHP and Open Source background, being a developer, evangelist, and partnership leader at SugarCRM, board member at OW2, president of OpenSocial, and frequent conference speaker around the world. As an avid writer, he has published articles on IBM Developerworks, Apple Developer Connection, and PHP Architect, and authored the book The Definitive Guide to SugarCRM: Better Business Applications and the book Building on SugarCRM.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@DevOpsSummit Stories
With more than 30 Kubernetes solutions in the marketplace, it's tempting to think Kubernetes and the vendor ecosystem has solved the problem of operationalizing containers at scale or of automatically managing the elasticity of the underlying infrastructure that these solutions need to be truly scalable. Far from it. There are at least six major pain points that companies experience when they try to deploy and run Kubernetes in their complex environments. In this presentation, the speaker will detail these pain points and explain how cloud can address them.
While DevOps most critically and famously fosters collaboration, communication, and integration through cultural change, culture is more of an output than an input. In order to actively drive cultural evolution, organizations must make substantial organizational and process changes, and adopt new technologies, to encourage a DevOps culture. Moderated by Andi Mann, panelists discussed how to balance these three pillars of DevOps, where to focus attention (and resources), where organizations might slip up with the wrong focus, how to manage change and risk in all three areas, what is possible and what is not, where to start, and especially how new structures, processes, and technologies can help drive a new DevOps culture.
When building large, cloud-based applications that operate at a high scale, it's important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. "Fly two mistakes high" is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, discussed how this same philosophy can be applied to highly scaled applications, and can dramatically increase your resilience to failure.
As Cybric's Chief Technology Officer, Mike D. Kail is responsible for the strategic vision and technical direction of the platform. Prior to founding Cybric, Mike was Yahoo's CIO and SVP of Infrastructure, where he led the IT and Data Center functions for the company. He has more than 24 years of IT Operations experience with a focus on highly-scalable architectures.
CI/CD is conceptually straightforward, yet often technically intricate to implement since it requires time and opportunities to develop intimate understanding on not only DevOps processes and operations, but likely product integrations with multiple platforms. This session intends to bridge the gap by offering an intense learning experience while witnessing the processes and operations to build from zero to a simple, yet functional CI/CD pipeline integrated with Jenkins, Github, Docker and Azure.