Welcome!

@DevOpsSummit Authors: Zakia Bouachraoui, Stackify Blog, Jnan Dash, Liz McMillan, Janakiram MSV

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Article

Scaling Incident Management | @DevOpsSummit #DevOps #APM #Monitoring

Incident management is paramount to the success of any modern ITOps team

Scaling Incident Management
By Patrick O'Fallon

Incident management is paramount to the success of any modern ITOps team. However, much like growing a business, scaling incident management can also trigger growing pains. As the landscape of devices, applications, and systems grows - each requiring monitoring - so too, does the alert noise and complexity around management for on-call staff. With an increasing number of engineers on your team, it can be difficult to on-board and implement new notification policies and after-hours operations to ensure your team is efficient and load is fairly distributed. And the push towards hybrid models of IT and bimodal IT environments can also complicate incident management. Nevertheless, with a few tried and true techniques, you can scale incident management in a planned, deliberate, organized, and effective way.

Don't fall victim to your changing ITOps environment
Let's first understand the problem with an example where scaling becomes a serious issue.
You've finally dialed in your incident management process, only to shortly after learn that your company has bought a new business. Now your Ops team is taking over IT for the new environment, in addition to what you're already responsible for. At first glance, you think of the perfect scenario in which you can simply apply the same tools and methodology to this entirely new stack.

However, reality is rarely perfect - the new company may leverage a different tech stack and different incident management monitoring tools and methodologies. While this scenario is incredibly daunting, it's very similar to any growth scenario - whether it be growing your IT team, or adopting more agile and bimodal ITOps structures. Whichever scale scenario you may face, below are some ideas for any organization that is working on scaling their monitoring, incident management, and team.

Identify the main areas of scale
Are you implementing new hardware, software, or services? Are there new complexities within your future state ITOps environment? Has your engineering team just grown? Have you inherited an application in which code errors need to be reported? In all cases, you must identify the areas in which your ITOps team is being forced to scale your operations.

Monitoring Tools
Ensuring coverage of your monitoring tools across your entire stack is paramount to the success of scaling. To adopt to this change, don't be afraid to implement multiple or entirely new monitoring systems outside of your current stack. The goal of these systems is to gain full-stack visibility, and in many cases this requires implementing different monitoring tools in order to appropriately monitor disparate and new systems. But to truly support organized scale, there needs to be a way to normalize, de-dupe, correlate, and gain actionable insights from all this data. All the events generated by these monitoring tools must be centralized in a single hub, from which they can be triaged and routed to the right on-call engineer.

Noise Reduction
When monitoring is in place, the goal is then to understand the data for effective incident resolution. Adjusting the routing behavior across your monitoring tools and configuring the appropriate thresholding is a great next step to ensure your team does not experience alert fatigue once you have implemented new tools. Aggregating this data and suppressing or filtering out non-actionable alerts from paging within a common incident management system is critical to help reduce the noise and enrich the visibility of incidents across your entire stack.

Incident Management
A comprehensive incident management platform will help integrate data from all your tools and grow with you as you scale. It not only unifies all your disparate monitoring alerts into one common system, it supports growth in your engineering team without generating confusion around resource management. Moreover, it helps facilitate more accountability as well as more organized collaboration. As a bonus, you can leverage incident analytics to show your boss how well your ITOps team is managing and resolving outages.

Scale and complexity are not going away
The world of ITOps is evolving rapidly, but one thing is clear - IT teams are being ordered to scale their operations in almost every capacity. Legacy ITOps environments are transitioning to and adopting more hybrid and agile architectures and frameworks. Users are continually demanding faster and more reliable access to data across different devices. As a result, it's necessary for ITOps teams to be equipped with a plan for scaling. Incident management is now a necessity as the stakes of downtime get higher.

The post Scaling Incident Management appeared first on PagerDuty.

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

@DevOpsSummit Stories
Cloud-Native thinking and Serverless Computing are now the norm in financial services, manufacturing, telco, healthcare, transportation, energy, media, entertainment, retail and other consumer industries, as well as the public sector. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential. DevOpsSUMMIT at CloudEXPO expands the DevOps community, enable a wide sharing of knowledge, and educate delegates and technology providers alike.
Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developers don't think about, but you can even use Docker with ASP.NET.
If you are part of the cloud development community, you certainly know about “serverless computing,” almost a misnomer. Because it implies there are no servers which is untrue. However the servers are hidden from the developers. This model eliminates operational complexity and increases developer productivity. We came from monolithic computing to client-server to services to microservices to the serverless model. In other words, our systems have slowly “dissolved” from monolithic to function-by-function. Software is developed and deployed as individual functions – a first-class object and cloud runs it for you. These functions are triggered by events that follow certain rules. Functions are written in a fixed set of languages, with a fixed set of programming models and cloud-specific syntax and semantics. Cloud-specific services can be invoked to perform complex tasks. So for cloud-na...
Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Kubernetes was originally built by Google, leveraging years of experience with managing container workloads, and is now a Cloud Native Compute Foundation (CNCF) project. Kubernetes has been widely adopted by the community, supported on all major public and private cloud providers, and is gaining rapid adoption in enterprises. However, Kubernetes may seem intimidating and complex to learn. This is because Kubernetes is more of a toolset than a ready solution. Hence it’s essential to know when and how to apply the appropriate Kubernetes constructs.
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It's clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. That means serverless is also changing the way we leverage public clouds. Truth-be-told, many enterprise IT shops were so happy to get out of the management of physical servers within a data center that many limitations of the existing public IaaS clouds were forgiven. However, now that we've lived a few years with public IaaS clouds, developers and CloudOps pros are giving a huge thumbs down to the...