Welcome!

DevOps Journal Authors: Elizabeth White, Plutora Blog, AppDynamics Blog, Andreas Grabner, ITinvolve Blog

Blog Feed Post

Thinking About APM? 4 Key Considerations for Buy vs. Build Your Own

image_pdfimage_print

Most technology folks have heard Marc Andreessen’s provocative statement, “Software is eating the world.”  Whether you agree fully or not, you’re realizing that your business critical software applications increasingly drive both the top-line revenue growth and the bottom-line operational efficiency of your company – and often form the pillar of your business identity.  

Legacy monitoring systems you have in place, capturing and alerting on scores of infrastructure level metrics, have helped protect your technology investment to some degree.  I’ve worked for two leaders in that space, HP and BMC, so can personally attest to the real benefits of server, network and database monitoring that clients achieve.  However, as consumer demand for superior services and faster innovation accelerate, we see that the applications and associated business transactions are what end-users ultimately care about.  You can no longer afford slow response time, let alone application outage situations, as customers will delay or abandon purchases – or worse yet switch to a competitor when you have unacceptable application performance.  And unfortunately these “Yellow Light” or slow performance situations are the most challenging to detect and fix!

So you’ve come to the conclusion that you need a full-fledged application performance management (APM) solution.  The question some companies wrestle with at this point is: “Should we invest in an 3rd party APM solution or build it ourselves?”

Four key considerations should be:

  1. Upfront Costs – such as Initial Project Build & Software License cost.

  2. Ongoing, Annual Solution Costs – such as Server / Storage footprint, administrative maintenance & support, & agile development / release activities.

  3. Solution Capabilities Driving Benefits – chiefly, the ability to drive down the number of performance defects in production, as well as the MTTR when issues do occur.

  4. Opportunity Costs – personnel resources working on in-house APM, versus are there mature 3rd party APM solutions available for purchase.

1.  Upfront Costs

It’s difficult to estimate exactly how long it would take a company to develop a basic application monitoring tool in-house – but we’ll give it a logical shot.  Of course, on the plus side, the company would avoid spending money on a “commercial off-the-shelf” (COTS) 3rd party software application.  Based on experience for design, development, testing, and release, a good estimate for an in-house Initial Project Build is a team of 2-3 Engineers about 6 months to have a basic, log parsing and alerting tool ready.  A more robust tool for a medium to large sized deployment may be 2-3x this size and investment.  A gaming company we work with, when assessing an in-house build situation, estimated an APM product development lifecycle in the 12 to 18 month range.  Why?  APM functionality that involves tracing the user experience of distributed transactions, where every call needs to be traced across each service layer, is non-trivial technical work.  Also, you’ll need to factor in one-time hardware and prerequisite software purchasing costs.  So a ballpark cost from $400K to well into seven figures is reasonable.

What would be the upfront software licensing cost of a 3rd party APM solution?  Probably in a similar range, perhaps higher in certain cases.  Also, many APM companies offer lower annual SaaS subscription costs as an alternative to full upfront licensing payments – which add up to the same licensing fees over 3-5 years.  However, you should take into consideration that some solutions such as AppDynamics, which can be downloaded and installed via self-service within hours, provide immediate Time-to-Value versus waiting for a full software development lifecycle to occur for a custom built solution.

Advantage:  Cost = In-house (slight? depends on robustness of APM solution built), Time-to-Value = 3rd Party APM

2.  Ongoing, Annual Solution Costs

First, let’s determine the hardware & storage footprint required for the solution.  Typical in-house developed solutions architect for over-capacity as a rough estimate because of unknowns, and to avoid encountering limitations & performance issues.  A good estimate per environment (Dev, Test, Prod) may be 2 Large Servers and 16 TB of Storage for a starter in-house APM solution.  This cost might run in the $100K to $135K range per year.

For 3rd party APM solutions, the specs are well-known, validated, and published.  A leading APM solution like AppDynamics has been built and tuned via R&D by specialists over several years.  The footprint for a similar medium-sized deployment would be 1 Medium Server and 6 TB of Storage, for a rough cost of about $40-50K per year – or less than half of the in-house cost.

From an FTE support perspective for the in-house solution, you have to understand the administrative, support, & enhancement / new development labor required.  A good admin & support estimate would run about 1-2 FTEs, and new development might run 2 engineering FTEs to keep up with enhancement requests and coverage for new applications & technologies.  Remember, users will not expect the APM solution to stay static!  You might start with basic metric stores and time series data, but this will quickly run out of steam.  Next, you’ll want to build a baseline engine for the metric store based on load patterns and percentiles of metrics, as examples.  Demand for dashboarding and security access control requirements come into play, and require much design and testing work especially as the solution scales.  So this annual labor cost would run in the $375K plus range.

On top of that, in today’s Agile DevOps world, there are additional maintenance / revision labor costs each time a business application is released to production.  Appliances and/or monitoring agents need updating, and both application and business transaction topology maps likely need to be revised manually.  As the frequency of application release grows, often to a bi-weekly application release schedule, these are not insignificant tasks.  We estimate in a medium sized deployment, this could require about 2,000 labor hours per year to keep up, or about $100K.

In the AppDynamics APM world, these types of capabilities are already built into the solution.  So the maintenance per application release is zero since there is automated application discovery, mapping, and business transaction flows out of the box.  The ongoing FTE administrative & maintenance requirements for a medium-sized deployment are 1 FTE, or about $125K/year.  And new development is covered in the license costs via the hundreds of R&D professionals contributing to the various releases of the 3rd party software.

Advantage:  3rd Party APM (large, especially adding up multiple years)

3.  Solution Capabilities Driving Benefits

Next we look at the ability of an APM solution to provide benefits to your enterprise – which can be grouped into reducing costs, mitigating risks, and increasing or protecting revenue.  Two key performance metrics we suggest for measuring impact on cost, risk, and revenue are:

  1. # defects released to production

  2. Mean time to repair (MTTR) per performance issue

At AppDynamics, this is where we’ve invested our R&D dollars since 2008, and our industry-exceeding Net Promoter Score (NPS) of 84 – i.e., more than 8 in 10 customers would recommend us to a friend or colleague – is a testimony to our ability to achieve these benefits.

By leveraging AppDynamics in Pre-Production, our clients often report reduction in performance issues released to Production of 40%.  And by watching every line of code executed in Production, and measuring & scoring each transaction, we provide a “3 clicks to resolution” approach that often reduces MTTR per performance issue by 65% or more.  This is true of small application environments, as well as large deployments over 20,000 JVMs.

For an in-house solution, you have to assess what it would take to build similar APM capabilities to achieve these levels of defect and MTTR reduction.  How many years, developers, and dollars?  (And, as one client executive recently told us, “If I could do this, why wouldn’t my company be competing in the APM software space?!”)  Or alternatively and more likely, “let’s stitch something low-cost together” in-house.  Admittedly this sacrifices capability for cost cost, which translates into fewer features to address the MTTR and # of performance issue challenges you face.

For ballpark purposes, then, let’s credit the in-house solution in helping reduce both # of defects and MTTR up to 20%.  If we use an industry average cost per minute of slowness / downtime equating to $500 (inclusive of both labor and revenue protection factors), and there is one Sev1 performance issue per application per quarter – the difference between the in-house solution versus an APM solution would equate to over $1M per year for a medium sized deployment.

Advantage:  3rd Party APM (not close; and these add up year over year, too)

4.  Opportunity Costs

These costs deal chiefly with choosing what is most valuable for your developers to spend their time on.  Especially in today’s high-technology enterprises, there are excellent engineers capable of building fantastic tools across a wide range of areas – so it is tempting to initiate an in-house APM build project and get something out the door.  However, APM is not these engineers’ specialty and their talents are often better utilized on alternative software projects related to the core goods & services your company sells to your end-use customers that drive revenue.

This is an area we won’t attempt to quantify, as it’s more of a qualitative assessment and business decision specific to your organization.  But with a fairly mature and continually developing 3rd Party APM market, for most enterprises it’s logical to say….

Advantage:  3rd Party APM

4.  Summary

While the initial, upfront set of costs for an in-house vs. 3rd party APM solution purchase may be about the same (license vs build) – which leads some organizations to consider a “Do It Yourself” approach – there are significant ongoing annual costs for the care and feeding of an in-house APM solution compared to the 3rd party APM alternative.  These include the infrastructure footprint, as well as labor costs associated with administration, maintenance & enhancements.

The biggest differential in cost is typically related to the chief purpose of an APM solution – how often does it proactively reduce the number of production defects, and how fast does it help you resolve performance issues when they do inevitably occur?

Screen Shot 2014-06-19 at 8.47.02 AM

For a medium sized deployment, the total cost / benefit advantages of a 3rd party APM solution easily exceed $1M per year when compared to the in-house build alternative.  This benefit accumulates year over year.  And it’s worth mentioning here at AppDynamics, we achieve magnitudes of benefit even beyond other 3rd party APM solutions with lesser capabilities.  We’ve leveraged the feedback of our over 1,000 customers during the past several years to drive R&D and greater benefit realization.

For AppDynamics, these advantages stem from:

  1. The way our solution is architected to require minimal setup, upkeep and Time-to-Value, while providing ongoing Ease of Use.

  2. Key capabilities – such as transaction tracing across complex, distributed applications, in your data center and the cloud – which lead to significant improvement in KPIs such as # performance defects and MTTR.

  3. Our ability to intelligently scale to support the most complex and largest Pre-Production and Production environments.

  4. Thought-leadership expanding into our “Application Intelligence” platform with a host of new modules and capabilities.

So when assessing an in-house vs 3rd party APM solution, consider a multi-year TCO horizon and not just a short-term initial cost estimate.  Our personnel at AppDynamics standby to provide you help in not only getting a deep-dive on the APM market and our solution features, but also to analyze the value of APM choices via a detailed ROI assessment.

Thinking of trying a next generation APM solution rather than build it yourself? Try AppDynamics for free today!

The post Thinking About APM? 4 Key Considerations for Buy vs. Build Your Own written by appeared first on Application Performance Monitoring Blog from AppDynamics.

Read the original blog entry...

More Stories By AppDynamics Blog

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally. DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

@DevOpsSummit Stories
Cloud computing started a technology revolution; now DevOps is driving that revolution forward. By enabling new approaches to service delivery, cloud and DevOps together are delivering even greater speed, agility, and efficiency. No wonder leading innovators are adopting DevOps and cloud together! In his session at DevOps Summit, Andi Mann, Vice President of Strategic Solutions at CA Technologies, explored the synergies in these two approaches, with practical tips, techniques, research data, war stories, case studies, and recommendations on how to: Embrace, source, consume, and expose shared...
The term culture has had a polarizing effect among DevOps supporters. Some propose that culture change is critical for success with DevOps, but are remiss to define culture. Some talk about a DevOps culture but then reference activities that could lead to culture change and there are those that talk about culture change as a set of behaviors that need to be adopted by those in IT. There is no question that businesses successful in adopting a DevOps mindset have seen departmental culture change, the question remains, is culture the leading edge of this change? I posit that in large enterprises,...
CA Technologies released a new study – “DevOps: The Worst-Kept Secret to Winning in the Application Economy” – that reveals that 82% of enterprises in Asia Pacific and Japan (APJ) already have or plan to adopt a DevOps strategy, a 12 point increase from last year’s figure of 70%. DevOps is a methodology which helps foster collaboration between the teams that create and test applications (Dev) with those that maintain them in production environments (Ops). Vanson Bourne conducted the survey with 1,425 senior IT and business leaders worldwide, including 650 respondents from six APJ countries: A...
DevOps means different things to different people. Qubell defines DevOps as the ability for the developer teams to do what they need to do to have this level of self-service. At DevOps Summit, Stan Klimoff, CTO of Qubell, demos the enterprise DevOps platform.
In a world of ever-accelerating business cycles and fast-changing client expectations, the cloud increasingly serves as a growth engine and a path to new business models. Dynamic clouds enable businesses to continuously reinvent themselves, adapting their business processes, their service and software delivery and their operations to achieve speed-to-market and quick response to customer feedback. As the cloud evolves, the industry has multiple competing cloud technologies, offering on-premises and off-premises cloud platforms for both Infrastructure as a Service (IaaS) and Platform as a Servi...
"Our premise is Docker is not enough. That's not a bad thing - we actually love Docker. At ActiveState all our products are based on open source technology and Docker is an up-and-coming piece of open source technology," explained Bart Copeland, President & CEO of ActiveState Software, in this SYS-CON.tv interview at DevOps Summit at Cloud Expo®, held Nov 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

Leysin American School is an exclusive, private boarding school located in Leysin, Switzerland. Leysin selected an OpenStack-powered, private cloud as a service to manage multiple applications and provide development environments for students across the institution. Seeking to meet rigid data sovereignty and data integrity requirements while offering flexible, on-demand cloud resources to users, Leysin identified OpenStack as the clear choice to round out the school's cloud strategy. Additionally, the school sought a partner to provide OpenStack infrastructure deployment and operations expert...
“We are strong believers in the DevOps movement and our staff has been doing DevOps for large enterprise environments for a number of years. The solution that we build is intended to allow DevOps teams to do security at the speed of DevOps," explained Justin Lundy, Founder & CTO of Evident.io, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
DevOps is all about agility. However, you don't want to be on a high-speed bus to nowhere. The right DevOps approach controls velocity with a tight feedback loop that not only consists of operational data but also incorporates business context. With a business context in the decision making, the right business priorities are incorporated, which results in a higher value creation. In his session at DevOps Summit, Todd Rader, Solutions Architect at AppDynamics, discussed key monitoring techniques that facilitate higher value creation for your enterprise.
The move in recent years to cloud computing services and architectures has added significant pace to the application development and deployment environment. When enterprise IT can spin up large computing instances in just minutes, developers can also design and deploy in small time frames that were unimaginable a few years ago. The consequent move toward lean, agile, and fast development leads to the need for the development and operations sides to work very closely together. Thus, DevOps becomes essential for any ambitious enterprise today.
CloudBees, Inc., has announced a $23.5 million financing round, led by longtime CloudBees investor Lightspeed Venture Partners. Existing investors Matrix Partners, Verizon Ventures and Blue Cloud Ventures also participated in the round. The latest funding announcement follows earlier rounds of $4 million, $10.5 million and $10.8 million, bringing the total investment in CloudBees to just under $50 million since the company’s inception in 2010. Previous venture investment rounds were led by Matrix Partners, Lightspeed Venture Partners, Verizon Ventures with participation from Blue Cloud Vent...
The 4th International DevOps Summit, co-located with16th International Cloud Expo – being held June 9-11, 2015, at the Javits Center in New York City, NY – announces that its Call for Papers is now open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits, DevOps is correlated with 2...
“The year of the cloud – we have no idea when it's really happening but we think it's happening now. For those technology providers like Zentera that are helping enterprises move to the cloud - it's been fun to watch," noted Mike Loftus, VP Product Management and Marketing at Zentera Systems, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
"SOASTA built the concept of cloud testing in 2008. It's grown from rather meager beginnings to where now we are provisioning hundreds of thousands of servers on a daily basis on behalf of customers around the world to test their applications," explained Tom Lounibos, CEO of SOASTA, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With "smart" appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user's habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps, abiding by privacy concerns and making the concept a reality. These challenges can't be addressed w...
"Application monitoring and intelligence can smooth the path in a DevOps environment. In a DevOps environment you see constant change. If you are trying to monitor things in a constantly changing environment, you're going to spend a lot of your job fixing your monitoring," explained Todd Rader, Solutions Architect at AppDynamics, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
High-performing enterprise Software Quality Assurance (SQA) teams validate systems that are ready for use - getting most actively involved as components integrate and form complete systems. These teams catch and report on defects, making sure the customer gets the best software possible. SQA teams have leveraged automation and virtualization to execute more thorough testing in less time - bringing Dev and Ops together, ensuring production readiness. Does the emergence of DevOps mean the end of Enterprise SQA? Does the SQA function become redundant? In her session at DevOps Summit, Anne Hungat...
Docker offers a new, lightweight approach to application portability. Applications are shipped using a common container format and managed with a high-level API. Their processes run within isolated namespaces that abstract the operating environment independently of the distribution, versions, network setup, and other details of this environment. This "containerization" has often been nicknamed "the new virtualization." But containers are more than lightweight virtual machines. Beyond their smaller footprint, shorter boot times, and higher consolidation factors, they also bring a lot of new fea...
“This win means a great deal to us because it is decided by the readers – the people who understand how use of our technology enables new insights that drive the business,” said Matt Davies, senior director, EMEA marketing, Splunk. “Splunk Enterprise enables organizations to improve service levels, reduce operations costs, mitigate security risks, enhance DevOps collaboration, create new product and service offerings and obtain deeper insight into customer behavior. Being named Best Business Application underlines the value Operational Intelligence delivers to our customers.”
“DevOps is really about the business. The business is under pressure today, competitively in the marketplace to respond to the expectations of the customer. The business is driving IT and the problem is that IT isn't responding fast enough," explained Mark Levy, Senior Product Marketing Manager at Serena Software, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
15th Cloud Expo, which took place Nov. 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA, expanded the conference content of @ThingsExpo, Big Data Expo, and DevOps Summit to include two developer events. IBM held a Bluemix Developer Playground on November 5 and ElasticBox held a Hackathon on November 6. Both events took place on the expo floor. The Bluemix Developer Playground, for developers of all levels, highlighted the ease of use of Bluemix, its services and functionality and provide short-term introductory projects that developers can complete between sessions.
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
Between the compelling mockups and specs produced by your analysts and designers, and the resulting application built by your developers, there is a gulf where projects fail, costs spiral out of control, and applications fall short of requirements. In his session at DevOps Summit, Charles Kendrick, CTO and Chief Architect at Isomorphic Software, will present a new approach where business and development users collaborate – each using tools appropriate to their goals and expertise – to build mockups and enhance them all the way through functional prototypes, to final working applications. Lea...
Log data provides the most granular view into what is happening across your systems, applications, and end users. Logs can show you where the issues are in real-time, and provide a historical trending view over time. Logs give you the whole picture. Logentries, a log management and analytics service built for the cloud, has announced a new integration with Slack, the team communication platform, to enable real-time system and application monitoring. Users of both services can now receive real-time notifications, log-level details with timestamps, and event context inside of designated Slack ...