Welcome!

@DevOpsSummit Authors: Zakia Bouachraoui, Liz McMillan, Yeshim Deniz, Pat Romanski, Elizabeth White

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Article

Scaling Incident Management | @DevOpsSummit #DevOps #APM #Monitoring

Incident management is paramount to the success of any modern ITOps team

Scaling Incident Management
By Patrick O'Fallon

Incident management is paramount to the success of any modern ITOps team. However, much like growing a business, scaling incident management can also trigger growing pains. As the landscape of devices, applications, and systems grows - each requiring monitoring - so too, does the alert noise and complexity around management for on-call staff. With an increasing number of engineers on your team, it can be difficult to on-board and implement new notification policies and after-hours operations to ensure your team is efficient and load is fairly distributed. And the push towards hybrid models of IT and bimodal IT environments can also complicate incident management. Nevertheless, with a few tried and true techniques, you can scale incident management in a planned, deliberate, organized, and effective way.

Don't fall victim to your changing ITOps environment
Let's first understand the problem with an example where scaling becomes a serious issue.
You've finally dialed in your incident management process, only to shortly after learn that your company has bought a new business. Now your Ops team is taking over IT for the new environment, in addition to what you're already responsible for. At first glance, you think of the perfect scenario in which you can simply apply the same tools and methodology to this entirely new stack.

However, reality is rarely perfect - the new company may leverage a different tech stack and different incident management monitoring tools and methodologies. While this scenario is incredibly daunting, it's very similar to any growth scenario - whether it be growing your IT team, or adopting more agile and bimodal ITOps structures. Whichever scale scenario you may face, below are some ideas for any organization that is working on scaling their monitoring, incident management, and team.

Identify the main areas of scale
Are you implementing new hardware, software, or services? Are there new complexities within your future state ITOps environment? Has your engineering team just grown? Have you inherited an application in which code errors need to be reported? In all cases, you must identify the areas in which your ITOps team is being forced to scale your operations.

Monitoring Tools
Ensuring coverage of your monitoring tools across your entire stack is paramount to the success of scaling. To adopt to this change, don't be afraid to implement multiple or entirely new monitoring systems outside of your current stack. The goal of these systems is to gain full-stack visibility, and in many cases this requires implementing different monitoring tools in order to appropriately monitor disparate and new systems. But to truly support organized scale, there needs to be a way to normalize, de-dupe, correlate, and gain actionable insights from all this data. All the events generated by these monitoring tools must be centralized in a single hub, from which they can be triaged and routed to the right on-call engineer.

Noise Reduction
When monitoring is in place, the goal is then to understand the data for effective incident resolution. Adjusting the routing behavior across your monitoring tools and configuring the appropriate thresholding is a great next step to ensure your team does not experience alert fatigue once you have implemented new tools. Aggregating this data and suppressing or filtering out non-actionable alerts from paging within a common incident management system is critical to help reduce the noise and enrich the visibility of incidents across your entire stack.

Incident Management
A comprehensive incident management platform will help integrate data from all your tools and grow with you as you scale. It not only unifies all your disparate monitoring alerts into one common system, it supports growth in your engineering team without generating confusion around resource management. Moreover, it helps facilitate more accountability as well as more organized collaboration. As a bonus, you can leverage incident analytics to show your boss how well your ITOps team is managing and resolving outages.

Scale and complexity are not going away
The world of ITOps is evolving rapidly, but one thing is clear - IT teams are being ordered to scale their operations in almost every capacity. Legacy ITOps environments are transitioning to and adopting more hybrid and agile architectures and frameworks. Users are continually demanding faster and more reliable access to data across different devices. As a result, it's necessary for ITOps teams to be equipped with a plan for scaling. Incident management is now a necessity as the stakes of downtime get higher.

The post Scaling Incident Management appeared first on PagerDuty.

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

@DevOpsSummit Stories
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like "How is my application doing" but no idea how to get a proper answer.
Hackers took three days to identify and exploit a known vulnerability in Equifax’s web applications. I will share new data that reveals why three days (at most) is the new normal for DevSecOps teams to move new business /security requirements from design into production. This session aims to enlighten DevOps teams, security and development professionals by sharing results from the 4th annual State of the Software Supply Chain Report -- a blend of public and proprietary data with expert research and analysis.Attendees can join this session to better understand how DevSecOps teams are applying lessons from W. Edwards Deming (circa 1982), Malcolm Goldrath (circa 1984) and Gene Kim (circa 2013) to improve their ability to respond to new business requirements and cyber risks.
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, software-defined solution with rich machine intelligence.
So the dumpster is on fire. Again. The site's down. Your boss's face is an ever-deepening purple. And you begin debating whether you should join the #incident channel or call an ambulance to deal with his impending stroke. Yes, we know this is a developer's fault. There's plenty of time for blame later. Postmortems have a macabre name because they were once intended to be Viking-like funerals for someone's job. But we're civilized now. Sort of. So we call them post-incident reviews. Fires are never going to stop. We're human. We miss bugs. Or we fat finger a command - deleting dozens of servers and bringing down S3 in US-EAST-1 for hours - effectively halting the internet. These things happen.
The digital transformation is real! To adapt, IT professionals need to transform their own skillset to become more multi-dimensional by gaining both depth and breadth of a wide variety of knowledge and competencies. Historically, while IT has been built on a foundation of specialty (or "I" shaped) silos, the DevOps principle of "shifting left" is opening up opportunities for developers, operational staff, security and others to grow their skills portfolio, advance their careers and become "T"-shaped.