Welcome!

@DevOpsSummit Authors: Elizabeth White, Yeshim Deniz, Pat Romanski, Liz McMillan, Stackify Blog

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Feed Post

Full-Stack Docker Performance Monitoring By @MayrAlois | @DevOpsSummit #DevOps #Containers

Docker containerization is increasingly being used in production environments

Full-Stack Docker Performance Monitoring: From Containers to Applications
By Alois Mayr

Docker containerization is increasingly being used in production environments. How can these environments best be monitored? Monitoring Docker containers as if they are lightweight virtual machines (i.e., monitoring the host from within the container), with all the common metrics that can be captured from an operating system, is an insufficient approach. Docker containers can’t be treated as lightweight virtual machines; they must be treated as what they are: isolated processes running on hosts. Why? Because they are processes that start and terminate quickly. Virtual machines aren’t designed to run for only a short time and then be terminated. Likewise, processes serve specific tasks while virtual machines typically serve multiple tasks.

Utilize Docker’s Remote API
Monitoring an environment at the container level is a great first step towards understanding the dynamics of containers in your environment. Many tools use the Docker Remote API to capture host resource consumption metrics related to CPU, memory, and network IO for each container. This is valuable information that operators can use when allocating host resources to containers.

Docker Stats CLI

Details of container dynamics in an environment can be captured by querying the Docker API of all Docker engines. For example, you can learn which hosts run containers that use a specific image. With the current move towards microservices, this becomes more important as Docker images are built for each service. You need to know on which machines the containers for a specific service are running.

Hosts and container overview

Docker containers and orchestration technologies like Docker Swarm, Mesos/Marathon, and Kubernetes offer means of deploying, running, and scaling applications and microservices. The whole Docker ecosystem is a fantastic enabler for running microservices in dynamic cloud-based environments.

But how can you know if the services you’ve deployed are okay and if they’re working as designed? This is where application performance management enters the game.

It’s what’s running inside that counts
When it comes to application monitoring, you’re mostly interested in the services running inside containers rather than the containers themselves. You need application-centric information to ensure that the applications served by your containers are running as expected. You need CPU-time breakdowns for your application at the method level. You also need to inspect database queries, measure throughput and response times for services, and track communication between microservices across containers and hosts.

Docker container insights with Ruxit

Monitoring microservices within containers
If you need to run your services at scale, Docker containers and orchestration tools are an ideal approach. No matter if services are stateless or stateful, load balancers send traffic to the respective containers once they’re properly configured.

Docker service visibility

To monitor the health of your application’s services, you need intuitive infographics that show you the most important metrics for each service. With this approach you can track throughput, average response time,failure rate, and most time consuming requests that are processed by all containers for each service.

If you need deep insights about a specific condition, select a time frame and analyze the metrics from that period in detail.

Find performance hotspots at the method level
Deep application performance analysis includes the ability to identify hotspots that contribute to the response time of a request. This enables you to pinpoint the service methods that consume the most CPU, disk, or network time for each request. In our example below, you can see the method that consumes the most CPU time for a Java service running in Docker containers.

Docker performance monitoring with method-level break down

Measure database query execution times and frequencies
Analyzing queries to and responses from databases is an essential aspect of performance tuning and therefore a core feature of application monitoring. This also holds true for monitoring applications that run in containers, no matter if the databases are served by other containers or not.

Inspecting all SQL statements and NoSQL queries sent by an application tells you about average query response times, execution frequency, numbers of fetched rows/executions, and failure rate. With this information you can optimize caching and query behavior on the application end, not to mention optimizing each database statement.

Database statement analysis

Track JVM metrics in Docker containers
Tracking Java heap memory metrics
enables you to see if your JVM’s garbage collection works as expected and if there is a memory shortage. Memory shortage is the #1 cause of increased garbage collection times. You can see how long a JVM is suspended due to garbage collection and then fine-tune memory settings accordingly. In our example below, you can see a JBoss process running within a Docker container on an AWS ECS cluster.

JVM metrics in containerized Java process

Full-stack Docker performance monitoring
Since you can use and run Docker containers virtually everywhere, and you can run almost anything within containers, monitoring needs to keep up with Docker’s dynamic and portable approach. Docker performance monitoring needs to cover many entities beyond just the container and application space.

For example, monitoring must be able to automatically handle the dynamics of virtual environments where virtual instances come and go frequently and where ephemeral Docker containers may only live for a few seconds or minutes. If you run an end-user facing web application in your environment you should also monitor user actions and detect Javascript errors in your customers’ browsers.

The table below shows you how monitoring various aspects of your environment, including Docker containers, can provide answers to different questions related to the performance of your applications.

What do you need for what?

Are all my machines healthy?
(CPU usage, memory, disk latency)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringWhich components communicate with one another?
(Network connections between processes)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringAre the processes responsive?
(Process response time and availability)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringDoes the network allow for proper process communication?
(Traffic, TCP requests, connection timeouts, retransmissions)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringAre the containers healthy?
(CPU usage, memory, network IO)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringWhich images have been deployed?
(Hosts with containers using same image)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringWhere are new services deployed?
(New instances, containers, service deployments)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringAre my application services responsive?
(Response time, failure rate, workload)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringWhich code parts are critical?
(CPU, disk, network time spent on a method, exceptions)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringDo the databases respond quickly?
(Query execution frequency, response time, and failure rate)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringAre the message queues fast enough?
(Message response time, failure rate)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringHow does heap memory usage change over time?
(Memory used in the generations)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringWhat is the average web response time experienced by users per region?
(Response time, number of user actions, Apdex rating)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb MonitoringAre my applications available and functional?
(Periodic availability checks and SLA reports)
Server MonitoringNetwork MonitoringDocker MonitoringApplication MonitoringWeb Monitoring

Monitoring data captured for the entities listed above must be put into context and analyzed along with all other entities and related dependencies. For example, user action duration (web monitoring) for customers in a specific region may be high despite the fact that the web servers and backend services show low CPU usage (server monitoring or Docker monitoring). Let’s assume that the network connections are also fast (network monitoring). The problem may be due to too few worker threads for the Apaches within the containers (application monitoring) or there may be an overloaded ESXi host with a high CPU ready time for the respective VM (cloud monitoring).

In other words, full-stack monitoring requires that you monitor all entities with a single solution that can analyze and interpret monitoring data from across your technology stack.

What’s next?
Go for Ruxit! Not convinced that Ruxit can really monitor all the entities that I’ve outlined in this post? Then test drive Ruxit for yourself! Simply sign up for the free trial, install Ruxit Agent on your Docker hosts, and you’ll be all set for deep, full-stack monitoring of your Docker environment.

The post Full-stack Docker performance monitoring: From containers to applications appeared first on #monitoringlife.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

@DevOpsSummit Stories
SYS-CON Events announced today that A&I Solutions has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 1999, A&I Solutions is a leading information technology (IT) software and services provider focusing on best-in-class enterprise solutions. By partnering with industry leaders in technology, A&I assures customers high performance levels across all IT environments including: mainframe, distributed, virtual, and cloud platforms. With expertise in the entire application lifecycle —Plan, Build, Test, Deploy, Operate, and Secure— A&I enables customers to transform their businesses by driving consistency and maximizing efficiency.
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Business Unit at CA Technologies, will share his vision about the true ‘DevOps Royalty’ and how it will take a new breed of digital cloud craftsman, architecting new platforms with a new set of tools to achieve it. He will also present a number of important insights and findings from a recent cloud and DevOps study – outlining the synergies high performance teams are exploiting to gain significant busin...
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs often get requests to connect these silos using technologies such as VPN; however, these tend to be difficult to manage and are not engineered for accessing business data from the cloud.
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT services.
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits, DevOps is correlated with 20% faster time-to-market, 22% improvement in quality, and 18% reduction in dev and ops costs, according to research firm Vanson-Bourne. It is changing the way IT works, how businesses interact with customers, and how organizations are buying, building, and delivering software.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and Embedded Systems worldwide. Supermicro is committed to protecting the environment through its “We Keep IT Green®” initiative and provides customers with the most energy-efficient, environmentally friendly solutions available on the market.
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value Stream Mapping for the development and operations tool chain by offering DevOps Tool Chain Integration and Traceability; DevOps Tool Chain Orchestration; and DevOps Insight and Intelligence. CollabNet also offers traditional application lifecycle management, ALM, for the enterprise through its TeamForge product.
Cloud promises the agility required by today’s digital businesses. As organizations adopt cloud based infrastructures and services, their IT resources become increasingly dynamic and hybrid in nature. Managing these require modern IT operations and tools. In his session at 20th Cloud Expo, Raj Sundaram, Senior Principal Product Manager at CA Technologies, will discuss how to modernize your IT operations in order to proactively manage your hybrid cloud and IT environments. He will be sharing best practices around collaboration, monitoring, configuration and analytics that will help you boost experience and optimize utilization of your modern IT Infrastructures.
SYS-CON Events announced today that Twistlock, the leading provider of cloud container security solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Twistlock is the industry's first enterprise security suite for container security. Twistlock's technology addresses risks on the host and within the application of the container, enabling enterprises to consistently enforce security policies, monitor and audit activity and identify and isolate threats in a container or cluster of containers.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus internal resources on core competencies.
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single threaded, you can effectively identify hot spots in your serverless code. In his session at 20th Cloud Expo, David Martin, Principal Product Owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to overcome the challenges of monitoring S3 and RDS. This presentation will provide an overview of necessary Amazon Lambda concepts and discuss how to integrate the monitoring data with other tools.
The 21st International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their online business and let Enzu manage their IT hosting infrastructure.
This talk centers around how to automate best practices in a multi-/hybrid-cloud world based on our work with customers like GE, Discovery Communications and Fannie Mae. Today’s enterprises are reaping the benefits of cloud computing, but also discovering many risks and challenges. In the age of DevOps and the decentralization of IT, it’s easy to over-provision resources, forget that instances are running, or unintentionally expose vulnerabilities.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In his Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, will explore the ways that Nutanix technologies empower teams to react faster than ever before and connect teams in ways that were either too complex or simply impossible with traditional infrastructures.
In his opening keynote at 20th Cloud Expo, Michael Maximilien, Research Scientist, Architect, and Engineer at IBM, will motivate why realizing the full potential of the cloud and social data requires artificial intelligence. By mixing Cloud Foundry and the rich set of Watson services, IBM's Bluemix is the best cloud operating system for enterprises today, providing rapid development and deployment of applications that can take advantage of the rich catalog of Watson services to help drive insights from the vast trove of private and public data available to enterprises.
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
As enterprise cloud becomes the norm, businesses and government programs must address compounded regulatory compliance related to data privacy and information protection. The most recent, Controlled Unclassified Information and the EU’s GDPR have board level implications and companies still struggle with demonstrating due diligence. Developers and DevOps leaders, as part of the pre-planning process and the associated supply chain, could benefit from updating their code libraries and design by incorporating changes. In her session at 20th Cloud Expo, Maria Horton, CEO of EmeSec, will discuss how addressing the new liabilities of sensitive data tracking for GDPR and CUI via code design can reap huge benefits for organizations and for developers that due just that whether in cloud, IoT or traditional environments.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that WineSOFT will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Based in Seoul and Irvine, WineSOFT is an innovative software house focusing on internet infrastructure solutions. The venture started as a bootstrap start-up in 2010 by focusing on making the internet faster and more powerful. WineSOFT’s knowledge is based on the expertise of TCP/IP, VPN, SSL, peer-to-peer, mobile browser, and live streaming solutions.
SYS-CON Events announced today that EARP Integration will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. EARP Integration is a passionate software house. Since its inception in 2009 the company successfully delivers smart solutions for cities and factories that start their digital transformation. EARP provides bespoke solutions like, for example, advanced enterprise portals, business intelligence systems and mobile applications for international enterprises across different sectors such as Energy and Utilities, GreenTech, MedTech, FinTech, Facility Management and Housing, Automotive Manufacturing, and Sport. EARP also cooperates with international software houses by providing them with highly qualified and well-selected, multilingual teams for bigger projects.
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
SYS-CON Events announced today that Carbonite will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Carbonite protects your entire IT footprint with the right level of protection for each workload, ensuring lower costs and dependable solutions with DoubleTake and Evault.