Welcome!

@DevOpsSummit Authors: Liz McMillan, Flint Brenton, Pat Romanski, Elizabeth White, Cameron Van Orman

Related Topics: @DevOpsSummit, Linux Containers, @CloudExpo

@DevOpsSummit: Blog Post

Hidden Exceptions & Application Performance Problems | @DevOps #APM #DevOps

"We dramatically improved application performance, average load times dropped from 500-800 milliseconds to 250-500

Case Study: How GeeksWithBlogs Found Hidden Exceptions & Application Performance Problems

Exceptions in .NET can be very expensive when it comes to CPU cycles. Read how Geeks with Blogs (http://geekswithblogs.net) used Stackify APM+ to improve the performance of their application.

About Geeks with Blogs (GWB)
Thousands of software developers use Geeks with Blogs (GWB) to host their own personal blog. The code base has slowly morphed and has been modified by several developers over 15 years. As a free service, GWB decided to use Stackify to try improve the overall page load times and CPU usage with the hopes of also being able to use fewer servers to cut hosting costs and improve application performance and user experience. Here’s their story:

How GWB uses Stackify APM+ to improve its application performance
We installed Stackify APM+ on our web servers and configured log4net to send errors and logs to Stackify. This allowed us to get visibility to errors being logged by log4net, basic server monitoring, key .NET performance counters, and full code profiling details via APM. I have to admit, it was pretty amazing to see all of this data in one place. By sending our log4net logs to Stackify we were able to immediately identify a couple simple bugs that needed to be fixed. The APM’s page level performance tracking confirmed that two pages (the blog list and individual blog page view) were generating the vast majority of all the traffic to the site. We also found a couple places in our code where we could do a better job of caching database queries. Out-of-the-box Stackify’s default monitoring included some .NET performance counters. One of which was the number of .NET CLR exceptions being thrown per minute. We noticed that the number was over 10,000 on each server but none of those errors were being logged via log4net. The exceptions were being caught and thrown away and we had no way to know what they were. This shows total errors being about 400,000 to 700,000 every 30 minutes. Yikes!<,p>

Log improvments

Preventing and finding hidden exceptions in your code
It is very common for developers to make helper methods for common tasks like grabbing a column from a SqlDataReader and converting it to the correct type as shown below. This type of code seems innocent but can potentially cause huge performance problems as you will see below.

public DateTime? GetDate(SqlDataReader reader, string columnName)
{
DateTime? value = null;
try
{
value = DateTime.Parse(reader[columnName].ToString());
}
catch
{
}
return value;
}

This code can easily throw multiple different exceptions:

  1. An invalid column name could have been passed in via the columnName variable
  2. The value in the data reader could be null
  3. The value in the data reader isn’t a valid DateTime and parsing fails

#2 and #3 are easy to avoid by refactoring the code as shown below. Checking for null and using TryParse eliminates the potential exceptions.

public DateTime? GetDate(SqlDataReader reader, string columnName)
{
try
{
var dbval = reader[columnName];
DateTime dt;
if (dbval != DBNull.Value && DateTime.TryParse(dbval.ToString(), out dt))
{
return dt;
}
}
catch {}
return null;
}

The root cause of our performance problems was actually #1, trying to access a column name that was not available in the SqlDataReader. We have several web pages that do different types of searches of blog posts and they use different stored procedures. A long time ago someone made the decision to have one code path that took the stored procedure results and populate them in to a common object that all the pages used. This worked perfect except all the stored procedures returned slightly different columns. This caused a lot of exceptions to occur for the missing columns, but we had no idea that the exceptions even existed since our code did not log the exceptions.

Stackify’s automatic APM exception collection
The only way we were able to find these exceptions was from Stackify’s APM+. Because it does code profiling within the CLR, it is able to get visibility to all exceptions that are thrown. So even though our code was swallowing the exceptions and not logging them via log4net, Stackify was able to give us visibility to them with no coding or logging changes. We were able to immediately find problems in a 15 year old application.

GWB’s Optimization Results
With a few code changes we were able to eliminate several exceptions that we had no idea existed until we used Stackify APM+. This cut CPU usage and enabled us to scale down the # of web servers we were using which will save us monthly hosting costs as well as to improve overall application performance. This chart shows how the number of errors has dramatically lowered.

Log rate improvment

Page load times improved as well. It is amazing how much of an effect hidden exceptions had on overall page load time across the app. Before the changes, average load times were 500-800 milliseconds and have dropped to 250-500ms depending on site traffic.

page load performance improvment

This chart below shows request volume by user satisfaction over the same time period. You can see that the yellow sluggish requests have lessened. Improve application performance

Exceptions are slow. Avoid them!
Exceptions in .NET are expensive, even when your users don’t know they are happening. The primary cause of it is capturing the stack trace at the time the exception is thrown. The .NET CLR has to stop and walk back through the calling frames and this requires CPU cycles and blocks the thread. At Stackify we would suggest never using exceptions to control application logic flow and to use good defensive coding techniques to avoid issues like those which Geeks with Blogs ran into.

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com

@DevOpsSummit Stories
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant that knows everything and can respond to your emotions and verbal commands!
SYS-CON Events announced today that B2Cloud will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. B2Cloud specializes in IoT devices for preventive and predictive maintenance in any kind of equipment retrieving data like Energy consumption, working time, temperature, humidity, pressure, etc.
Today traditional IT approaches leverage well-architected compute/networking domains to control what applications can access what data, and how. DevOps includes rapid application development/deployment leveraging concepts like containerization, third-party sourced applications and databases. Such applications need access to production data for its test and iteration cycles. Data Security? That sounds like a roadblock to DevOps vs. protecting the crown jewels to those in IT.
SYS-CON Events announced today that Cedexis will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Cedexis is the leader in data-driven enterprise global traffic management. Whether optimizing traffic through datacenters, clouds, CDNs, or any combination, Cedexis solutions drive quality and cost-effectiveness.
Many companies start their journey to the cloud in the DevOps environment, where software engineers want self-service access to the custom tools and frameworks they need. Machine learning technology can help IT departments keep up with these demands. In his session at 21st Cloud Expo, Ajay Gulati, Co-Founder, CTO and Board Member at ZeroStack, will discuss the use of machine learning for automating provisioning of DevOps resources, taking the burden off IT teams.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp empowers global organizations to unleash the full potential of their data to expand customer touchpoints, foster greater innovation and optimize their operations.
Is advanced scheduling in Kubernetes achievable? Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, will answer these questions and demonstrate techniques for implementing advanced scheduling. For example, using spot instances and cost-effective resources on AWS, coupled with the ability to deliver a minimum set of functionalities that cover the majority of needs – without configuration complexity.
SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http://www.ryobi-sol.co.jp/en/.
With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In his session at @DevOpsSummit at 21st Cloud Expo, Chris Van Tuin, Chief Technologist, NA West at Red Hat, will discuss: The top security risks with containers and how to manage these risks at scale including Images, Builds, Registry, Deployment, Hosts, Network, Storage, APIs, Monitoring/Logging, and Federation.
SYS-CON Events announced today that mruby Forum will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. mruby is the lightweight implementation of the Ruby language. We introduce mruby and the mruby IoT framework that enhances development productivity. For more information, visit http://forum.mruby.org/.
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capability.
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
SYS-CON Events announced today that Nihon Micron will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nihon Micron Co., Ltd. strives for technological innovation to establish high-density, high-precision processing technology for providing printed circuit board and metal mount RFID tags used for communication devices. For more information, visit http://www.nihon-micron.co.jp/.
SYS-CON Events announced today that Suzuki Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Suzuki Inc. is a semiconductor-related business, including sales of consuming parts, parts repair, and maintenance for semiconductor manufacturing machines, etc. It is also a health care business providing experimental research for dementia, etc. For more information, visit http://www.e-suzuki.co.jp/en/.
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data and analytics insights onto a single, holistic, display, focusing attention on what matters, when it matters.
Today most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes significant work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reduction in cost and increase in speed. Sometimes in order to reduce complexity teams compromise features or change requirements
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of engineers can navigate the Carrier Ecosystem for your IT team acting as an extension of your business, producing a hassle-free experience.
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
SYS-CON Events announced today that Daiya Industry will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Daiya Industry specializes in orthotic support systems and assistive devices with pneumatic artificial muscles in order to contribute to an extended healthy life expectancy. For more information, please visit https://www.daiyak.co.jp/en/.
Most of the time there is a lot of work involved to move to the cloud, and most of that isn't really related to AWS or Azure or Google Cloud. Before we talk about public cloud vendors and DevOps tools, there are usually several technical and non-technical challenges that are connected to it and that every company needs to solve to move to the cloud. In his session at 21st Cloud Expo, Stefano Bellasio, CEO and founder of Cloud Academy Inc., will discuss what the tools, disciplines, and cultural aspects are that enterprise companies are considering to get to the cloud and eventually transform the way they build software and services.
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.interface-amita.com/aboutus/interface_profile.asp.
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, visit http://www.krcnet.co.jp/eng_site/e_index.htm.
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.