|By AppDynamics Blog||
|May 8, 2014 12:00 PM EDT||
I wouldn’t do website load/performance testing any more without having an APM tool in place. Period. Full stop. End of story.
I’ve been involved in website load testing for over 10 years, as a “end-user” when I was web operations manager for an online job board, as a team leader for a company providing cloud load testing services, and as a consultant on web performance with my own company DevOpsGuys. The difference in the value you get from load/performance testing with and without APM tools is enormous.
We’ve probably all seen those testing reports that are full of graphs of response time versus req/sec, CPU utilisation curves, disk IO throughput, error rates ad nauseam. I, to my eternal shame, have even written them… and whilst they are useful for answering the (very simplistic) question of “how many simulated requests/users can my website support before it falls over?” generating any real application insight from what are essentially infrastructure metrics is difficult. This type of test report rarely results in any corrective actions other than (1) “lets throw more hardware at it” or (2) “let’s shout at the devs that they have to fix something because the application is slow”. Quite often the report gets circular filed because no-one knows how to derive application insight and hence generate meaningful corrective actions at either the code, application stack configuration or infrastructure level. All that effort & expense is wasted.
So how are things different when using APM tools (like my preferred tool, AppDynamics)? Here are my top 4 reasons:
1. See the Big Picture (Systems Thinking)
“Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots.” - Peter Senge, “The Fifth Discipline”.
The “first way of DevOps” is systems thinking, and APM tools reinforce the systems thinking perspective by helping you see the big picture very clearly. You can see the interrelationships between the web tier, application tier, database servers, message queues, external cloud services etc. in real time while you’re testing rather than being focussed on the metrics for each tier individually. You can instantly see where the bottlenecks in your application are in the example below the 4306ms calls to Navision stand out!
2. Drill Down to the Code Level
One of my favourite things when load testing with APM tools is being able to drill down to the stack trace level and identify the calls that are the most problematic. Suddenly, instead of talking about infrastructure metrics like CPU, RAM and Disk we are talking about application metrics — this business transaction (e.g. web page or API request) generates this flow across the application and 75% of the time is spent in this method call which makes 3 database calls and 2 web service calls and its this database call that’s slow and here’s the exact SQL statement that was executed. The difference in the response you get from the developer’s when you give them this level of detail compared to “your application is slow when we hit 200 users” is fantastic — now you are giving them real, pinpoint actionable intelligence on how their application responds under load.
3. Iterate Faster
“the application was made 56x faster during a 12hr testing window”
Because you can move quickly to the code level in real-time while you test and because this facilitates better communication with the development team your load testing suddenly becomes a lot more collaborative, even if the load testing is being performed by an external 3rd party.
We generally have all the relevant parties on a conference call or HipChat chat session while we test and we are constantly exchanging information, screenshots, links to APM snapshots and the developers are often able to code fixes there and then because we can rapidly pinpoint the pain points.
If you’ve got a customer with an Agile mindset and continuous delivery capability it can enable you to do rapid test and fix cycles during testing, often multiple times times in a day. In one notable example, the application was made 56x faster during a 12hr testing window due to 4 application releases during that period.
4. Stop the “Blame Game”
“make the enemy poor performance, not each other…”
Traditionally in the old school (pre-APM tools) days, load tests were often conducted by external load testing consultancies who would come in, do the testing, and then deliver some big report on how things went.
The customer would assemble their team together in a conference room to go through the report, which often triggered the “blame game” – Ops blaming Dev, Dev blaming QA, QA blaming Ops, Ops blaming the hosting provider, the hosting provider blaming the customer’s code and around and around it would go.
But with the right APM tools in place we’ve found this negative team dynamic can be avoided.
As mentioned earlier, testing tends to become more collaborative because it’s easier to share the performance data in real time via the APM tool, and discussions become more evidence-based. It’s more about “what are we going to do about this problem we can see here in the APM tool” and less about trying to allocate blame when no-one really knows where the problem actually resides and they don’t want to be left holding the can. The system-thinking, holistic view of the application’s performance promulgated by the APM tool makes performance the enemy, not each other. And that means that the performance issues are likely to be fixed faster, and not ignored due to politics and infighting.
There are probably loads more reasons you can come up with for why load testing with APM tools are awesome (and I’d love you hear your thoughts in the comments), but I will leave you with one more bonus reason – because it’s fun. For me, using AppDynamics when I’m doing load testing and performance tuning has really bought the fun factor back into the work. It’s fun to see the load being applied to the system and to see (via AppDynamics) the effect across the entire application. It’s fun to work closer with the Dev & Ops teams (dare I say, “DevOps”!) and to share meaningful, actionable insights on where the problems lie, and it’s fun be able to rapidly iterate and show the performance improvements in real-time.
The post 4 Reasons Why You Should Use APM When You Load Test Your Website written by Steve Thair (DevOpsGuys) appeared first on Application Performance Monitoring Blog from AppDynamics.
For almost two decades, businesses have discovered great opportunities to engage with customers and even expand revenue through digital systems, including web and mobile applications. Yet, even now, the conversation between the business and the technologists that deliver these systems is strained, in large part due to misaligned objectives. In his session at DevOps Summit, James Urquhart, Senior Vice President of Performance Analytics at SOASTA, Inc., will discuss how measuring user outcomes – including how the performance, flow and content of your digital systems affects those outcomes – ca...
Oct. 3, 2015 11:00 PM EDT Reads: 261
Clearly the way forward is to move to cloud be it bare metal, VMs or containers. One aspect of the current public clouds that is slowing this cloud migration is cloud lock-in. Every cloud vendor is trying to make it very difficult to move out once a customer has chosen their cloud. In his session at 17th Cloud Expo, Naveen Nimmu, CEO of Clouber, Inc., will advocate that making the inter-cloud migration as simple as changing airlines would help the entire industry to quickly adopt the cloud without worrying about any lock-in fears. In fact by having standard APIs for IaaS would help PaaS expl...
Oct. 3, 2015 09:00 PM EDT Reads: 336
While testing is often ignored when it comes to DevOps - it could be the most important aspect of achieving true DevOps success. Without rethinking automated testing from the ground-up, the entire DevOps productivity gain cannot be realized. Large tech companies build their own rapid test automation that runs in minutes across functional, performance, security and other tests. In his session at DevOps Summit, Kevin Surace, CEO of Appvance, will discuss how we learn from these real-world successes and achieve a 95% time reduction in creating and running automated unified tests. Otherwise ent...
Oct. 3, 2015 06:00 PM EDT Reads: 284
SYS-CON Events announced today that Cloud Raxak has been named “Media & Session Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Raxak Protect automates security compliance across private and public clouds. Using the SaaS tool or managed service, developers can deploy cloud apps quickly, cost-effectively, and without error.
Oct. 3, 2015 01:15 PM EDT Reads: 557
DevOps is gaining traction in the federal government – and for good reasons. Heightened user expectations are pushing IT organizations to accelerate application development and support more innovation. At the same time, budgetary constraints require that agencies find ways to decrease the cost of developing, maintaining, and running applications. IT now faces a daunting task: do more and react faster than ever before – all with fewer resources.
Oct. 3, 2015 01:00 PM EDT Reads: 240
SYS-CON Events announced today that IBM Cloud Data Services has been named “Bronze Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. IBM Cloud Data Services offers a portfolio of integrated, best-of-breed cloud data services for developers focused on mobile computing and analytics use cases.
Oct. 3, 2015 12:00 PM EDT Reads: 357
The modern software development landscape consists of best practices and tools that allow teams to deliver software in a near-continuous manner. By adopting a culture of automation, measurement and sharing, the time to ship code has been greatly reduced, allowing for shorter release cycles and quicker feedback from customers and users. Still, with all of these tools and methods, how can teams stay on top of what is taking place across their infrastructure and codebase? Hopping between services and command line interfaces creates context-switching that slows productivity, efficiency, and may le...
Oct. 3, 2015 10:15 AM EDT Reads: 219
Docker is hot. However, as Docker container use spreads into more mature production pipelines, there can be issues about control of Docker images to ensure they are production-ready. Is a promotion-based model appropriate to control and track the flow of Docker images from development to production? In his session at DevOps Summit, Fred Simon, Co-founder and Chief Architect of JFrog, will demonstrate how to implement a promotion model for Docker images using a binary repository, and then show how to distribute them to any kind of consumer, being it a customer or a data center.
Oct. 3, 2015 10:00 AM EDT Reads: 612
SYS-CON Events announced today that JFrog, maker of Artifactory, the popular Binary Repository Manager, will exhibit at SYS-CON's @DevOpsSummit Silicon Valley, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Based in California, Israel and France, founded by longtime field-experts, JFrog, creator of Artifactory and Bintray, has provided the market with the first Binary Repository solution and a software distribution social platform.
Oct. 3, 2015 10:00 AM EDT Reads: 411
Mobile, social, Big Data, and cloud have fundamentally changed the way we live. “Anytime, anywhere” access to data and information is no longer a luxury; it’s a requirement, in both our personal and professional lives. For IT organizations, this means pressure has never been greater to deliver meaningful services to the business and customers.
Oct. 3, 2015 09:00 AM EDT Reads: 239
DevOps is speeding towards the IT world like a freight train and the hype around it is deafening. There is no reason to be afraid of this change as it is the natural reaction to the agile movement that revolutionized development just a few years ago. By definition, DevOps is the natural alignment of IT performance to business profitability. The relevance of this has yet to be quantified but it has been suggested that the route to the CEO’s chair will come from the IT leaders that successfully make the transition to a DevOps model. If this still seems foreign to you, I recommend reading up on D...
Oct. 3, 2015 06:00 AM EDT Reads: 14,004
The cloud has reached mainstream IT. Those 18.7 million data centers out there (server closets to corporate data centers to colocation deployments) are moving to the cloud. In his session at 17th Cloud Expo, Achim Weiss, CEO & co-founder of ProfitBricks, will share how two companies – one in the U.S. and one in Germany – are achieving their goals with cloud infrastructure. More than a case study, he will share the details of how they prioritized their cloud computing infrastructure deployments and the details they’ve learned. From performance to network configurations, they've got interesting...
Oct. 3, 2015 03:00 AM EDT Reads: 576
Decisions about budgets and resources are often made without IT even having a seat at the table. As technologist we understand the value of DevOps - but do your business counterparts? If they don't, your DevOps initiatives could lose funding before they start. In her session at DevOps Summit, Jeanne Morain, Strategist / Author at iSpeak Cloud, LLC, will provide insights on how to bridge the gap between business and technology leaders. Attendees will learn prescriptive guidance on balancing workloads, critical communication processes and considerations for building out a solid return-on-inves...
Oct. 3, 2015 02:00 AM EDT Reads: 479
The principles behind DevOps are not new - for decades people have been automating system administration and decreasing the time to deploy apps and perform other management tasks. However, only recently did we see the tools and the will necessary to share the benefits and power of automation with a wider circle of people. In his session at DevOps Summit, Bernard Sanders, Chief Technology Officer at CloudBolt Software, will explore the latest tools including Puppet, Chef, Docker, and CMPs needed to move from an insulated culture where automation is absent or hoarded to one where the power of ...
Oct. 1, 2015 10:30 PM EDT Reads: 286
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
Oct. 1, 2015 06:00 AM EDT Reads: 7,341
As a company adopts a DevOps approach to software development, what are key things that both the Dev and Ops side of the business must keep in mind to ensure effective continuous delivery? In his session at DevOps Summit, Mark Hydar, Head of DevOps, Ericsson TV Platforms, will share best practices and provide helpful tips for Ops teams to adopt an open line of communication with the development side of the house to ensure success between the two sides.
Sep. 30, 2015 05:45 PM EDT Reads: 448
At first adopted by enterprises to consolidate physical servers, virtualization is now widely used in cloud computing to offer elasticity and scalability. On the other hand, Docker has developed a new way to handle Linux containers, inspired by version control software such as Git, which allows you to keep all development versions. In his session at 17th Cloud Expo, Dominique Rodrigues, the co-founder and CTO of Nanocloud Software, will discuss how in order to also handle QEMU / KVM virtual machines versions, they have developed a new tool, called Vm_commit, which can create commits, backup ...
Sep. 30, 2015 04:30 PM EDT Reads: 506
Even though you are running an agile development process, that doesn’t necessarily mean that your performance testing is being conducted in a truly agile way. Saving performance testing for a “final sprint” before release still treats it like a waterfall development step, with all the cost and risk that comes with that. In this post, we will show you how to make load testing happen early and often by putting SLAs on the agile task board.
Sep. 30, 2015 01:00 PM EDT Reads: 521
Today, we are in the middle of a paradigm shift as we move from managing applications on VMs and containers to embracing everything that the cloud and XaaS (Everything as a Service) has to offer. In his session at 17th Cloud Expo, Kevin Hoffman, Advisory Solutions Architect at Pivotal Cloud Foundry, will provide an overview of 12-factor apps and migrating enterprise apps to the cloud. Kevin Hoffman is an Advisory Solutions Architect for Pivotal Cloud Foundry, and has spent the past 20 years building enterprise and mobile software for small businesses, massive enterprises, and everything in ...
Sep. 30, 2015 04:00 AM EDT Reads: 592
SYS-CON Events announced today that HPM Networks will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. For 20 years, HPM Networks has been integrating technology solutions that solve complex business challenges. HPM Networks has designed solutions for both SMB and enterprise customers throughout the San Francisco Bay Area.
Sep. 28, 2015 10:30 PM EDT Reads: 1,380
SYS-CON Events announced today that Logz.io has been named a "Bronze Sponsor" of SYS-CON's @DevOpsSummit Silicon Valley, which will take place November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Logz.io provides open-source software ELK turned into a log analytics platform that is simple, infinitely- scalable, highly available, and secure.
Sep. 28, 2015 05:15 PM EDT Reads: 714
DevOps delivers remarkable results. But does it help all of IT? Can traditional ‘mode 1’ IT benefit as much as innovative ‘mode 2’? How about the rest of your business? Or have you just shifted your bottleneck? And if so, what can you do about it? Improving dev and ops is necessary, but not sufficient. It often just shifts the burden sideways (e.g., to PMs, SQA, InfoSec, DBAs, NOC, etc.), upstream (to the PMO, Controller, Business Liaison, etc.), or downstream (to TechPubs, Service Desk, Training, etc.).
Sep. 28, 2015 04:45 PM EDT Reads: 631
Clutch is now a Docker Authorized Consulting Partner, having completed Docker's certification course on the "Docker Accelerator for CI Engagements." More info about Clutch's success implementing Docker can be found here. Docker is an open platform for developers and system administrators to build, ship and run distributed applications. With Docker, IT organizations shrink application delivery from months to minutes, frictionlessly move workloads between data centers and the cloud and achieve 20x greater efficiency in their use of computing resources. Inspired by an active community and trans...
Sep. 28, 2015 12:00 PM EDT Reads: 285
SYS-CON Events announced today that Alert Logic, the leading provider of Security-as-a-Service solutions for the cloud, has been named “Bronze Sponsor” of SYS-CON's 17th International Cloud Expo® and DevOps Summit 2015 Silicon Valley, which will take place November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Alert Logic provides Security-as-a-Service for on-premises, cloud, and hybrid IT infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions.
Sep. 27, 2015 11:00 PM EDT Reads: 2,289
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership ability. Many are unable to effectively engage and inspire, creating forward momentum in the direction...
Sep. 27, 2015 01:00 PM EDT Reads: 786