Welcome!

@DevOpsSummit Authors: Elizabeth White, Pat Romanski, Liz McMillan, Yeshim Deniz, SmartBear Blog

Related Topics: @DevOpsSummit, Java IoT, Microservices Expo, Linux Containers, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Feed Post

Provisioning the Entire Stack | @DevOpsSummit #DevOps #Microservices

We’re currently in the process of stringing together applications at the various layers to do server provisioning

As we move into the world of complete data center automation, there is a whole new selection of issues that we are learning to resolve – from custom hardware to a variety of provisioning tools at each level of automation. These are not unexpected issues, but they certainly provide us with plenty to do while we’re trying to reduce the amount of busy work we have to do.

We’re currently in the process of stringing together applications at the various layers to do server provisioning, application provisioning, and (for internal apps at least) application deployment. The options out there are a pretty broad swath that runs from all-in-one solutions like Puppet with Razor to specialized solutions hooked together to run the entire automation process. The end result is the same, the differences are in implementation, vendor lock-in, and whether a full solution that can be implemented all at once is more important to an organization than a solution that has the most suited parts that can be implemented in parallel but have to be tied together.

Image complements of University Corporation for Atmospheric Research.

For those not familiar with myself or my employer StackIQ, the solutions available from my company run the gamut from specialized open source best-of-breed server provisioning (Stacki) to end-to-end Hadoop deployment. Which a given organization uses just depends upon which solution and what the organization’s relationship with StackIQ is.

As such, I am comfortable discussing the strengths and weaknesses of both styles of automation. After all, I’ve worked with both extensively, and can see the use cases that drive either one. There is no shortage of vendors in this space, and I will even mention a couple other than StackIQ, but my examples and use-cases will come from the various editions of Stacki, because I have worked on them more with that platform than others.

At the bottom level, pure-play server provisioning tools aim for best-of-breed usefulness. Open Source Stacki, for example, is focused on installing RHEL and CentoOS on target servers and configuring them to begin use. Of course few of the solutions on the market stop there, though a couple do. Stacki, for example, can be extended to install and configure agents from most of the application provisioning tools on the market. This means that server provisioning can hand off to application provisioning to achieve the full automation solution. It can also be extended to install specific applications as part of the OS install process. So a give server or set of servers can be configured to run Tomcat via installation, so that when Stacki finishes, the application server is installed and configured.

The drawback to this entry level is, of course, that such extensions must be developed by the user. While it is certainly possible to share extensions between users (for most tools anyway), and solicit feedback on the extensions being developed, there will still be work setting it up to work correctly. Some organizations – particularly those seeking a significant amount of control in their installations – find this to be an acceptable trade-off. Others would rather some of this foot work be done for them, or at least that they had ready access to people who specialize in this sort of thing.

That’s where the middle level comes in. This solution offers the functionality of the entry level, but adds either pre-built packages to install specific applications (like puppet agents or Tomcat) that can be easily marked as part of the install for a server or group of servers, or alternatively they offer the help of experts in setting these installations up – because once they’ve been configured, they (1) Don’t generally need to change much except to upgrade when new releases of the app come out and (2) Serve as examples that can be used to make other application customizations.

The issues with this solution are that it still takes time to get everything configured the first time, and it still requires a relatively intimate knowledge of the applications being deployed. For some shops these are not issues at all, for others they are show-stoppers that require significantly more investment than the organization is willing or able to make. That’s why there is the third level of integration for automation tools.

For some solutions in some organizations, the best answer is “push a button, get fully functional deployed servers running the software chosen.” A good example of this is big data. Deploying Hadoop by hand can be a daunting experience, yet timelines for Hadoop deployment far too often are tight. The third tier of integration is the system that knows what to do, and can juggle all the disparate pieces – hardware config, OS install, application install, configuration – at once. Push a button, reboot the machines intended for the Hadoop cluster, and a short time later there is a running Hadoop instance. I’m using Hadoop as an example here because of the glaring difference between when I’ve deployed it by hand and by automated provisioning (weeks to hours), but complex n-tier architectures stand to gain as much.

With these tools, an organization gets support built-in for a given set of applications, or gets the plug-ins to support a given application and adds them to one of the lower tiers. Then it is a simple case of defining (through command line, APIs, or spreadsheets) how the complex system should be deployed across target servers, and hitting a button. In a short time, the system is up and running with the target servers fully installed and doing their part. The issues with this solution are that there is a limit to the number of complex applications that have pre-packaged solutions, and of course, no internally developed application does. Professional services can help with this problem, but that adds expense and timeline to the project in a manner similar to the second tier of integration.

The best solution to these issues should be able to support all of the above integration tiers, but it should also be even more adaptable. There are several very good application provisioning tools on the market – Puppet, Chef, Saltstack, Ansible to name a few. A server provisioning tool, as the first step that runs in server-up datacenter automation, should be able to work reliably with all of these. Whether it enables users to hook up to them, or actually deploys the agents itself, the end result should be that a deployed server should be able to use the application provisioning that is currently in use at a given organization, and be able to change if that provisioning tool is replaced. Deeply integrated tools will work well with a single application provisioning tool, but tend toward vendor lock-in, because that tool is then harder to replace without replacing how servers are provisioned (which is no small chore in and of itself).

So when speccing out a datacenter automation or DevOps automation project, look for a server provisioning tool that can hand off to the application provisioning tool of choice, has some subset of application provisioning for complex systems (unless your application provisioning tool is already taking care of that of course), and can be fully automated. Partial automation is difficult at best, but it is the state of this growing market that some functionality is still not completely implemented in some products. It is easy enough to poke at products and determine if they suit the needs of your level of automation. A good example is that only about half of the products on the market make any attempt at RAID configuration as part of server provisioning. But if operators have to sit and configure RAID before an automated installation, then it’s not really fully automated, is it?

And of course I recommend that you consider automation. Partially because my employer plays in the space, but moreso because there are far more impactful things operations could be doing than sitting and watching an OS install, or typing in IP addresses, masks, and gateways.

More Stories By Don MacVittie

Don MacVittie is founder of Ingrained Technology, A technical advocacy and software development consultancy. He has experience in application development, architecture, infrastructure, technical writing,DevOps, and IT management. MacVittie holds a B.S. in Computer Science from Northern Michigan University, and an M.S. in Computer Science from Nova Southeastern University.

@DevOpsSummit Stories
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
For better or worse, DevOps has gone mainstream. All doubt was removed when IBM and HP threw up their respective DevOps microsites. Where are we on the hype cycle? It's hard to say for sure but there's a feeling we're heading for the "Peak of Inflated Expectations." What does this mean for the enterprise? Should they avoid DevOps? Definitely not. Should they be cautious though? Absolutely. The truth is that DevOps and the enterprise are at best strange bedfellows. The movement has its roots in the tech community's elite. Open source projects and methodologies driven by the alumni of companies like Netflix, Google and Amazon. This is a great thing for the evolution of DevOps. It can be alienating for Enterprise IT though. Learning about Netflix and their simian armies, or Facebook and their mind-melting scale is fascinating. Can you take it back to the office on Monday morning though?
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
Learn how to solve the problem of keeping files in sync between multiple Docker containers. In his session at 16th Cloud Expo, Aaron Brongersma, Senior Infrastructure Engineer at Modulus, discussed using rsync, GlusterFS, EBS and Bit Torrent Sync. He broke down the tools that are needed to help create a seamless user experience. In the end, can we have an environment where we can easily move Docker containers, servers, and volumes without impacting our applications? He shared his results so you can decide for yourself.
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examining how the Internet and the cloud has allowed for the democratization of IT, resulting in an increased demand for the cloud and the drive to develop new ways to utilize it.