Welcome!

@DevOpsSummit Authors: Pat Romanski, Elizabeth White, Liz McMillan, Stackify Blog, Dana Gardner

Related Topics: @DevOpsSummit, Microservices Expo, @CloudExpo

@DevOpsSummit: Blog Post

Key Performance Metrics By @GrabnerAndi | @DevOpsSummit [#APM #DevOps]

Capture and analyze the metrics from the different application tiers and components in your application

Key Performance Metrics for Load Tests Beyond Response Time | Part I

Whether it is JMeter, SoapUI, Load Runner, SilkTest, Neotys or one of the cloud-based load testing solutions such as Keynote, Dynatrace (formerly Gomez) or others, breaking an application under heavy load is easy these days. Finding the problem based on automatically generated load testing reports is not. Can you tell me what is wrong based on the following reports?

Load Testing Reports alone are showing you that there is a problem - but not necessarily where you should look next

My Key Metrics from Web Server to Database
I've helped engineering organizations over the last 10 - 15 years to either run or analyze load tests. In this blog post I share my best practices and metrics I typically look when analyzing load testing results. I am not relying on the out-of-the box load testing reports, but instead I extend them based on the tools and capabilities, or put in an APM tool such as Dynatrace to capture this type of data while the load testing tool drives the load.

Some of the technical product screenshots in this blog are taken from data users of our Dynatrace Free Trial who shared data through my Share Your PurePath program. Thanks to all of them.

Now - if a load testing task is coming up for you I hope you find most of my described steps useful as I believe it will make analyzing your results easier. Feel free to use Dynatrace (or any other APM tool if you already have such a tool) in order to capture and analyze the following metrics from the different application tiers and components in your application:

From web server to database there are key performance metrics to look at instead of spending too much time in the load testing report

Now - let me go into the details of these metrics, where to capture them from, and what they tell us. In this blog I focus on the Web Server, Application Server, Hosts and the Application Layers. The next blog will focus on the Database as well as Errors and Logging.

1. Top Web Server Metrics
On the Web Server (Apache, IIS, Nginx, ...) the following key metrics have proven extremely valuable to identify problems in your deployment:

  • Busy and Idle Threads
    • Do you need more worker threads per web server?
    • Do you need more web servers?
    • Are threads busy for too long because of application performance hotspots?
  • Throughput
    • How many transactions / minute can we handle?
    • When do we need to scale out and add more web servers?
  • Bandwidth Requirements
    • Is the network the bottleneck?
    • Are our average pages too heavy?
    • Can we offload content to CDNs?

For example below we have a Web Server Process Health Dashboard- showing all of the metrics that are key for me. They get captured through a module placed in the Web Server:

Key metrics from your web server: worker threads, throughput and bandwidth

2. Top App Server Metrics
On the application server (Java, .NET, PHP) I focus on the following key metrics to identify any deployment or configuration problems on your application servers:

  • Load Distribution
    • How many transactions are handled by each JVM/CLR/PHP engine?
    • Are they equally load balanced?
    • Do we need more application servers to handle the load?
  • CPU Hotspots
    • How much CPU is needed for this tested load?
    • Is high CPU caused by bad programming and can be fixed?
    • Or do we need more CPU power?
  • Worker Threads
    • Is the number of worker threads correctly configured?
    • Are worker threads busy because the application servers are not ready?
    • Are there any web server modules that block these threads?
  • Memory Issues
    • Do we see bad memory patterns? Do we have a memory leak?
    • What's the impact of garbage collection on CPU and transaction throughput?

The following screenshot shows my Process Health Dashboard. All data is automatically captured via an agent and injected in your Java, .NET, PHP or node.js engine:

Key metrics from your app server: worker threads, CPU, memory and throughput

For insight on hosts and the application layers click here for the full article

More Stories By Andreas Grabner

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@DevOpsSummit Stories
ChatOps is an emerging topic that has led to the wide availability of integrations between group chat and various other tools/platforms. Currently, HipChat is an extremely powerful collaboration platform due to the various ChatOps integrations that are available. However, DevOps automation can involve orchestration and complex workflows. In his session at @DevOpsSummit at 20th Cloud Expo, Himanshu Chhetri, CTO at Addteq, will cover practical examples and use cases such as self-provisioning infrastructure/applications, self-remediation workflows, integrating monitoring and complimenting integrations between Atlassian tools and other top tools in the industry.
"Storpool does only block-level storage so we do one thing extremely well. The growth in data is what drives the move to software-defined technologies in general and software-defined storage," explained Boyan Ivanov, CEO and co-founder at StorPool, in this SYS-CON.tv interview at 16th Cloud Expo, held June 9-11, 2015, at the Javits Center in New York City.
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and cost-effective resources on AWS, coupled with the ability to deliver a minimum set of functionalities that cover the majority of needs – without configuration complexity.
As Marc Andreessen says software is eating the world. Everything is rapidly moving toward being software-defined – from our phones and cars through our washing machines to the datacenter. However, there are larger challenges when implementing software defined on a larger scale - when building software defined infrastructure. In his session at 16th Cloud Expo, Boyan Ivanov, CEO of StorPool, provided some practical insights on what, how and why when implementing "software-defined" in the datacenter.
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, which can process our conversational commands and orchestrate the outcomes we request across our personal and professional realm of connected devices.