Welcome!

@DevOpsSummit Authors: Liz McMillan, Elizabeth White, William Schmarzo, Yeshim Deniz, Pat Romanski

Related Topics: @DevOpsSummit, Microservices Expo, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Article

Real User #Monitoring | @DevOpsSummit #APM #DevOps #ContinuousDelivery

Enterprises are interested in understanding how they analyze performance to positively impact business metrics

With online viewership and sales growing rapidly, enterprises are interested in understanding how they analyze performance to positively impact business metrics. Deeper insight into the user experience is needed to understand why conversions are dropping and/or bounce rates are increasing or, preferably, to understand what has been helping these metrics improve.

The digital performance management industry has evolved as application performance management companies have broadened their scope beyond synthetic testing that simulates users loading specific pages at regular intervals to include web and mobile testing, and real user monitoring (RUM).  As synthetic monitoring gained popularity, performance engineers realized the variations that exist from real end users were not being captured. This led to the introduction of RUM - the process of capturing, analyzing and reporting data from a real end user's interaction with a website. RUM has been around for more than a decade, but the technology is still in its infancy.

Five factors contributing to the shift towards RUM to complement synthetic testing

Ability to measure third-party resources
Websites are complex, with many different resources affecting performance. While there is no way to reliably detect the number of third party scripts, the number of third-party components is growing, with the average web page now requesting over 30% of their resources from third party domains, as shown in Figure 1. These components have multiple purposes, including   tracking users, ad insertion, and  A/B testing. Understanding the impact these components have on the end user experience is critical.

Figure 1 - Growth in third party vs first party resources per page, 2011-2015

Mobile matters
With more users accessing applications primarily on mobile devices, understanding mobile performance is increasingly important. Metrics must be captured from desktop and mobile devices alike. Just because an application performs well on a desktop does not mean it will perform well on a mobile device. If you have or want to have mobile customers, ensure you are able to capture metrics from them. Mobile presents unique challenges, such as congestion and latency, that can have significant impacts on page performance.

With a growing  mobile user base, RUM is frequently correlated with bandwidth measured in the last mile, to determine whether the impact to performance is a result of unpredictable last mile conditions. This need is increasingly seen in many major Asian economies, where a large proportion of consumers' primary means of internet access is a mobile phone. Major eCommerce players in Asia report over 65% of transactions are made from mobile devices. With such a big customer base, monitoring performance on the mobile web and understanding the influence of carrier impact on performance is critical to doing business. Some businesses have therefore instrumented ability to profile expected levels of user experience as it relates to carrier impact on performance.

Validate performance for specific users or geographies
Synthetic measurements may not be available from all geographies. To understand why a service level agreement in a specific region is not being met, the only way to capture information may be through real users in that geographic location. Real user measurements also enable customers to validate whether issues reported by synthetic testing are widespread across user base or localized to geos or local to the synthetic test tools.

Continuous Delivery
As more organizations move to a continuous delivery model, synthetic tests may need to be frequently re-scripted. As the time to deliver and release content decreases, organizations are looking at ways to quickly gather performance data. Some have decided the fastest way to gather performance metrics on a just-released page or feature is through data from real users.

Native applications
As organizations evolve from mobile websites to native apps, the need to gather metrics from these applications becomes increasingly important.

What features should you look for in a RUM solution?
Knowing that you need a RUM solution is the first step.   The second step is identifying what features are required to meet your business needs.  With a variety of solutions available in the market, identifying the must-have and the nice-to-have features is important to find the best fit.  Here are a few features you should consider.

Real-time and actionable data
Most RUM tools  display insights in the dashboard for the user in near real-time.  This information can be coupled with near real time tracking information from business analytics tools like Google Analytics. Performance data from RUM solutions should be cross-checked against metrics such as site visits, conversions,user location and device/browser insights. Many website operators continuously monitor any changes in the business metrics since they are indicative of problems in performance; further, it enables them to minimize false positives or isolated issues in performance.

User experience timings
Trends in performance optimization testing have  moved away from metrics like time to first byte (TTFB) and page load towards measurements more accurately reflecting the user experience - such as start render and speed index.  A user does not necessarily care when the content on the bottom of the page has loaded - when critical resources have been loaded and the page appears usable is what matters. Ensure the metrics you are gathering accurately reflect what you are attempting to measure and optimize.

Granular information
While page-level metrics are a good start, they don't reveal  precisely what resources are causing content to load slowly, nor  the relevance of each metric. Combining resource timing on specific elements with where the resource is (above or below "the fold") can help organizations filter out the noise and collect actionable information. Intersection Observer can help you identify which resources are loading above or below the fold and prioritize what to do to remedy the impact.

Impact of ads
With large numbers of pages being populated with ads, understanding the impact of the ads is important. RUM tools can identify both the performance impact of an ad in terms of when the ad was fetched and how long it took to download, as well as user engagement - such as how many users watched a video ad in its entirety.

Correlation to business metrics
While there have been many articles describing the impact of performance on business in eCommerce companies - for example, impact on conversions - the same isn't true for media companies. Media companies are more interested in scroll depth, virality of content, and session length.  Soasta recently announced an Activity Impact Score as a way to correlate web performance to session length. Measurements like the Activity Impact Score help non-eCommerce companies measure and monitor engagement and how performance can negatively or positively impact user engagement. Further, with bonuses tied to metrics such as page views, organizations are increasingly scrutinizing RUM metrics and insist on verifying the integrity of these tools.

End device support & ease of measurement
With the plethora of device types and browsers on the market, you need to ensure the RUM solution implemented will capture traffic from the majority of your users. In some Asian countries, over 35% of browsers and devices are unknown, which presents an interesting challenge: should you just forget about these users, or find a way to reliably measure performance on these unknown devices?

Another important factor to consider is how easy is it to enable RUM measurements? Does it require manual instrumentation of every web page or is this automatically done by injection of a script?

End to end perspective
Frequently the performance issues can be anywhere in the delivery network or end user. The ability to zero in on the problem quickly requires correlation of metrics from the end user, last mile, delivery network and the server.

Dynamic thresholds and alerts
The connectivity of an end user's device can change throughout the day. At work, they may be browsing the internet on a high-speed connection; on the commute home, they may be on their mobile device with high latency and congestion; and at night, they may be at home on a DSL or fiber connection. Expecting the same level of performance at all times is unrealistic. Having the ability to set variable thresholds is more indicative of the real user experience.

What solutions exist today
In addition to commercial solutions like Soasta, New Relic, and Google Analytics' Site Speed, there are three specifications from the W3C that enable you to build your own solution - navigation timing, resource timing and user timing. Browser support for these specifications vary, with navigation timing having the greatest adoption, since it has been available the longest.

Navigation timing captures the timing of various events as a page loads, from the HTTP request until all content has been received, parsed, and executed by the browser. This provides high-level information on the overall page load time and can be used to get details on items such as DNS lookups and latency.

Figure 2 shows the various timings available from the navigation timing API:

Figure 2 - Navigation timing events

Among many metrics that can be computed using the navigation timing events, the following are most often used:

  • TimeToFirstByte = responseStart - requestStart
  • TimeToInteractive = domInteractive - requestStart
  • TimeToPageLoad = loadEventEnd - requestStart

While page-level information is helpful, you may want to know how various resources on a page perform. This is where the resource timing specification comes in. Resource timing enables you to collect complete timing information for any resource within a page,with some restrictions for security purposes.  The resource timings available for the request and response are shown in Figure 3.

Figure 3 - Resource timing events

Once resource and navigation timing specifications were available for all resources, the next step was to provide the ability to gather custom metrics to understand where an application is spending the most time. The user timing specification allows marks to be inserted in code enabling the  measurement of time deltas between various marks. This makes it possible to determine information like when a hero image is displayed, when fonts are loaded, and when scripts are done blocking.

Evolving quality measurements
As quality measurements evolve, they will become better at providing actionable insights that recommend specific improvements to mitigate performance bottlenecks - not only at the browser end point, but from an end-to-end perspective.

Increasingly, RUM measurements will leverage machine learning to more deeply understand traffic patterns and dynamically adapt to  changing patterns.

RUM measurements will evolve to include the time a given resource starts to execute and completes execution in the browser.

Also, device-agnostic solutions will no doubt emerge. Metrics need to be captured across the entire spectrum of user endpoints. Not gathering statistics from large percentages of users whose browsers don't support the technology leaves gaping blind spots in the visibility you have on the end user experience.

*    *    *

RUM gives organizations the ability to isolate and identify the cause of performance degradation in a web application, whether it is related to the browser, third-party content, the network provider, the CDN, or infrastructure. RUM is a piece of the puzzle; when used in conjunction with other tools and analytics, it can be used  to quickly recommend web application optimizations.

More Stories By Krishnan Manjeri

Krishnan is a seasoned product manager and is currently a Director of Product Management at InstartLogic responsible for Data Platform, Analytics and Performance. He has nearly 2 decades of experience in leading & delivering solutions, in various capacities from Engineering to Marketing and Product Management, for a variety of fortune 500 companies in the areas of Analytics, Telecommunication Networks, Application Delivery and Security. He has extensive experience leading cross-functional teams and delivering multi-million dollars in revenue in both the Enterprise and Service Provider. He has an MS in Computer Science from Case Western Reserve University and an MBA from Santa Clara University. He has a couple of patents in the area of Networking and Security.

@DevOpsSummit Stories
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs often get requests to connect these silos using technologies such as VPN; however, these tend to be difficult to manage and are not engineered for accessing business data from the cloud.
SYS-CON Events announced today that Outscale, a global pure play Infrastructure as a Service provider and strategic partner of Dassault Systèmes, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2010, Outscale simplifies infrastructure complexities and boosts the business agility of its customers. Outscale delivers a secure, reliable and industrial strength solution for its customers, which include solopreneurs, fast-growing startups, large multi-national companies and government agencies. The company is a strategic partner to Dassault Systémes, and today powers hundreds of organizations throughout North America, Europe and Asia. Outscale’s U.S. headquarters is located in Jersey City, New Jersey, and its global headquarters is in Saint-Cloud, France.
SYS-CON Events announced today that DivvyCloud will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. DivvyCloud software enables organizations to achieve their cloud computing goals by simplifying and automating security, compliance and cost optimization of public and private cloud infrastructure. Using DivvyCloud, customers can leverage programmatic Bots to identify and remediate common cloud problems in real time. DivvyCloud was founded by seasoned technologists who understand first-hand what is necessary to succeed in today’s fast-changing, multi-cloud world.
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single threaded, you can effectively identify hot spots in your serverless code. In his session at 20th Cloud Expo, David Martin, Principal Product Owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to overcome the challenges of monitoring S3 and RDS. This presentation will provide an overview of necessary Amazon Lambda concepts and discuss how to integrate the monitoring data with other tools.
Interested in leveling up on your Cloud Foundry skills? Join IBM for Cloud Foundry Days on June 7 at Cloud Expo New York at the Javits Center in New York City. Cloud Foundry Days is a free half day educational conference and networking event. Come find out why Cloud Foundry is the industry's fastest-growing and most adopted cloud application platform.
Most DevOps journeys involve several phases of maturity. Research shows that the inflection point where organizations begin to see maximum value is when they implement tight integration deploying their code to their infrastructure. Success at this level is the last barrier to at-will deployment. Storage, for instance, is more capable than where we read and write data. In his session at @DevOpsSummit at 20th Cloud Expo, Josh Atwell, a Developer Advocate for NetApp, will discuss the role and value extensible storage infrastructure has in accelerating software development activities, improve code quality, reveal multiple deployment options through automated testing, and support continuous integration efforts. All this will be described using tools common in DevOps organizations.
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a single platform to drive unprecedented simplicity in the data center. Customers can start with a base infrastructure and scale to multi-site and multi-geo infrastructures with predictable economics and performance.
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet the demands of Digital Transformation – including accelerating application delivery, closing feedback loops, enabling multi-channel delivery, empowering collaborative decisions, improving user experience, and ultimately meeting (and exceeding) business goals.
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
SYS-CON Events announced today that A&I Solutions has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 1999, A&I Solutions is a leading information technology (IT) software and services provider focusing on best-in-class enterprise solutions. By partnering with industry leaders in technology, A&I assures customers high performance levels across all IT environments including: mainframe, distributed, virtual, and cloud platforms. With expertise in the entire application lifecycle —Plan, Build, Test, Deploy, Operate, and Secure— A&I enables customers to transform their businesses by driving consistency and maximizing efficiency.
SYS-CON Events announced today that Tappest will exhibit MooseFS at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. MooseFS is a breakthrough concept in the storage industry. It allows you to secure stored data with either duplication or erasure coding using any server. The newest – 4.0 version of the software enables users to maintain the redundancy level with even 50% less hard drive space required. The software functions on all major operating systems and is used around the world by businesses, universities and NGOs. The MooseFS team provides on-site implementation and support for your operations.
SYS-CON Events announced today that EARP will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. "We are a software house, so we perfectly understand challenges that other software houses face in their projects. We can augment a team, that will work with the same standards and processes as our partners' internal teams. Our teams will deliver the same quality within the required time and budget just as our partners' teams would do. It helps our partners to expand their operations with our forces."
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
Regardless of what business you’re in, it’s increasingly a software-driven business. Consumers’ rising expectations for connected digital and physical experiences are driving what some are calling the "Customer Experience Challenge.” In his session at @DevOpsSummit at 20th Cloud Expo, Marco Morales, Director of Global Solutions at CollabNet, will discuss how organizations are increasingly adopting a discipline of Value Stream Mapping to ensure that the software they are producing is poised to offer continuous improvements to customers’ experience of products and services.
SYS-CON Events announced today that Outscale will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outscale's technology makes an automated and adaptable Cloud available to businesses, supporting them in the most complex IT projects while controlling their operational aspects. You boost your IT infrastructure's reactivity, with request responses that only take a few seconds.
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT services.
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits, DevOps is correlated with 20% faster time-to-market, 22% improvement in quality, and 18% reduction in dev and ops costs, according to research firm Vanson-Bourne. It is changing the way IT works, how businesses interact with customers, and how organizations are buying, building, and delivering software.
Cloud promises the agility required by today’s digital businesses. As organizations adopt cloud based infrastructures and services, their IT resources become increasingly dynamic and hybrid in nature. Managing these require modern IT operations and tools. In his session at 20th Cloud Expo, Raj Sundaram, Senior Principal Product Manager at CA Technologies, will discuss how to modernize your IT operations in order to proactively manage your hybrid cloud and IT environments. He will be sharing best practices around collaboration, monitoring, configuration and analytics that will help you boost experience and optimize utilization of your modern IT Infrastructures.
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Business Unit at CA Technologies, will share his vision about the true ‘DevOps Royalty’ and how it will take a new breed of digital cloud craftsman, architecting new platforms with a new set of tools to achieve it. He will also present a number of important insights and findings from a recent cloud and DevOps study – outlining the synergies high performance teams are exploiting to gain significant busin...
SYS-CON Events announced today that Twistlock, the leading provider of cloud container security solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Twistlock is the industry's first enterprise security suite for container security. Twistlock's technology addresses risks on the host and within the application of the container, enabling enterprises to consistently enforce security policies, monitor and audit activity and identify and isolate threats in a container or cluster of containers.
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value Stream Mapping for the development and operations tool chain by offering DevOps Tool Chain Integration and Traceability; DevOps Tool Chain Orchestration; and DevOps Insight and Intelligence. CollabNet also offers traditional application lifecycle management, ALM, for the enterprise through its TeamForge product.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus internal resources on core competencies.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and Embedded Systems worldwide. Supermicro is committed to protecting the environment through its “We Keep IT Green®” initiative and provides customers with the most energy-efficient, environmentally friendly solutions available on the market.