Click here to close now.

Welcome!

DevOps Journal Authors: Elizabeth White, Liz McMillan, Trevor Parsons, Matt LeRay, Pat Romanski

Related Topics: Silverlight, Java, SOA & WOA, .NET, Virtualization, Cloud Expo, DevOps Journal

Silverlight: Article

Combining Agile with Load and Performance Testing: What Am I in For?

Load and performance testing successfully in an Agile way can save an organization a lot in costly bug fixes

Agile software development isn't really a "new" trend anymore. I mean, the Agile Manifesto turns 13 years old next month and while that might be early adolescence in human years, it's downright ancient as far as trends in IT are concerned. However, one area that has yet to fully mature is the implementation of non-functional testing practices in a Continuous Testing sort of way that can keep pace with more Agile development teams. Load and performance testing definitely fall into that category. You might be a performance tester on a team that is just starting to do more iterative development or a more experienced Agile tester looking to add load and performance testing to your workflow; either way, you'll likely want to know what you're in for.

Agile development practices can help teams achieve faster time to market, adapt to changing requirements, provide a constant feedback loop, etc. The benefits of load and performance testing include determining how much load an application can handle before it crashes in production, when to add another server, when to reconfigure the network, where code needs to be optimized, etc.  What is less well known is the fact that the combination of the two practices can lead to additional benefits that go beyond just the sum of the benefits of each practice, i.e. 1+1=3.

Some of these benefits include:

Avoiding Late Performance Problem Discovery
When load and performance testing are pushed off until the end of a development cycle, there is often little to no time for developers to make changes. This can cause teams to push back release dates and delay getting features out the door that customers need. Alternatively, if the issues are minor, teams may decide to proceed and launch the application into production while accepting the heightened risks.  If the performance problems are more fundamental, they could even require painful architectural changes that could take weeks or months to implement.

Making Changes Earlier When They Are Cheaper
By including load and performance testing in Continuous Integration testing processes, organizations can catch performance issues early before they get much more costly to fix. Developers can instantly know that the new feature in the latest build has caused the application to no longer meet Service Level Agreements (SLAs). They can fix the problem then and there before it becomes exponentially more expensive. This is especially true on Agile teams when discovering a performance problem weeks later could mean that it actually occurred several builds ago which makes the task of pinpointing the root cause a nightmare.

Guaranteeing Users Get New Features, Not New Performance Issues
In some Agile organizations, changes are happening incredibly fast. It's possible for a new feature or some new functionality to get checked into source control, run through a Continuous Integration build, pass all of the automated tests, and get deployed to the production server in a matter of minutes. But if that code wasn't optimized to handle the number of simultaneous users seen at the worst peak times, it could cause the whole system to crash. Integrating load testing into the process before these changes are deployed to production can ensure that your users get all the goodies they want without the bad user experiences. This can save your company thousand or even millions in lost revenue from users switching to competitors' apps or bashing your brand because of the problems they experienced with your app.

In the same way that combining Agile with load testing can provide unique benefits, it can also present your teams with unique challenges they may not have experienced in the past.

Shorter Development Cycles Require More Tests in Less Time
Load and performance testing are usually pushed off until the end of a development cycle. With Agile, development, cycles are much shorter, and load & performance testing can get pushed off until the last day of a sprint or sometimes it's done every other sprint. This can often result in code being released without being adequately tested or user stories slipping to the next release once they have been tested. Conceptually the solution is to do the testing earlier in the development cycle, but that's easier said than done with many teams lacking the resources and tools to make it happen.

"Working" Code Does Not Always Perform Well
So much focus for developers on Agile teams is put on delivering "working" code, but is code really "working" if it fails when the application is under load? Should user stories and tasks really be marked as "done" if the code associated with them causes the application to crash with 100 users? What about 1,000? 100,000? The pressure to get code out the door is high, but so is the cost of having an application crash in production.

Developers Need Feedback NOW
Agile developers need to know more than just the fact that their code is causing performance issues: they need to know when their code started causing problems and what story they were working on when the issue started. It's a huge pain for developers to be forced to go back and fix code for a story they worked on weeks or months ago. It also means they can't spend time working on getting new features out the door. Detecting performance issues early in the cycle so you can deliver important feedback to developers quickly is crucial to saving costs.

Automating the Handoff from Dev to Ops Can Feel Risky
While DevOps and Continuous Deployment are still fairly young practices, the fear felt by operations teams that new changes in code will slow down or even crash the application when it is deployed in production has been around forever. Automating some of the testing in the Continuous Integration process can help to ease some of this fear, but without adequate performance testing included, the risk is still real. Ops teams know well the huge impact application downtime can have on the business.

As daunting as some of these challenges seem, don't lose sight of the benefits you'll receive. Load and performance testing successfully in an Agile way can save an organization a lot in costly bug fixes and could make a tester a hero to both development and operations teams alike.

Not quite sure how to get started? Stay tuned for my next blog on best practices for Agile load and performance testing.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@DevOpsSummit Stories
Thanks to Docker, it becomes very easy to leverage containers to build, ship, and run any Linux application on any kind of infrastructure. Docker is particularly helpful for microservice architectures because their successful implementation relies on a fast, efficient deployment mechanism – which is precisely one of the features of Docker. Microservice architectures are therefore becoming more popular, and are increasingly seen as an interesting option even for smaller projects, instead of being reserved to the largest, most complex application stacks.
DevOps is about increasing efficiency, but nothing is more inefficient than building the same application twice. However, this is a routine occurrence with enterprise applications that need both a rich desktop web interface and strong mobile support. With recent technological advances from Isomorphic Software and others, it is now feasible to create a rich desktop and tuned mobile experience with a single codebase, without compromising performance or usability.
The speed of software changes in growing and large scale rapid-paced DevOps environments presents a challenge for continuous testing. Many organizations struggle to get this right. Practices that work for small scale continuous testing may not be sufficient as the requirements grow. In his session at DevOps Summit, Marc Hornbeek, Sr. Solutions Architect of DevOps continuous test solutions at Spirent Communications, will explain the best practices of continuous testing at high scale, which is relevant to small scale DevOps, and if there is an expectation of growth as the number of build targe...
SYS-CON Events announced today Arista Networks will exhibit at SYS-CON's DevOps Summit 2015 New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Arista Networks was founded to deliver software-driven cloud networking solutions for large data center and computing environments. Arista’s award-winning 10/40/100GbE switches redefine scalability, robustness, and price-performance, with over 3,000 customers and more than three million cloud networking ports deployed worldwide.
Application metrics, logs, and business KPIs are a goldmine. It’s easy to get started with the ELK stack (Elasticsearch, Logstash and Kibana) – you can see lots of people coming up with impressive dashboards, in less than a day, with no previous experience. Going from proof-of-concept to production tends to be a bit more difficult, unfortunately, and it tends to gobble up our attention, time, and money. In his session at DevOps Summit, Otis Gospodnetić, co-author of Lucene in Action and founder of Sematext, will share the architecture and decisions behind Sematext’s services for handling larg...
Software is eating the world. Companies that were not previously in the technology space now find themselves competing with Google and Amazon on speed of innovation. As the innovation cycle accelerates, companies must embrace rapid and constant change to both applications and their infrastructure, and find a way to deliver speed and agility of development without sacrificing reliability or efficiency of operations. In her Day 2 Keynote DevOps Summit, Victoria Livschitz, CEO of Qubell, discussed how IT organizations can automate just-in-time assembly of application environments - each built fo...
SYS-CON Events announced today that Open Data Centers (ODC), a carrier-neutral colocation provider, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Open Data Centers is a carrier-neutral data center operator in New Jersey and New York City offering alternative connectivity options for carriers, service providers and enterprise customers.
Thanks to Docker, it becomes very easy to leverage containers to build, ship, and run any Linux application on any kind of infrastructure. Docker is particularly helpful for microservice architectures because their successful implementation relies on a fast, efficient deployment mechanism – which is precisely one of the features of Docker. Microservice architectures are therefore becoming more popular, and are increasingly seen as an interesting option even for smaller projects, instead of being reserved to the largest, most complex application stacks.
Security can create serious friction for DevOps processes. We've come up with an approach to alleviate the friction and provide security value to DevOps teams. In her session at DevOps Summit, Shannon Lietz, Senior Manager of DevSecOps at Intuit, will discuss how DevSecOps got started and how it has evolved. Shannon Lietz has over two decades of experience pursuing next generation security solutions. She is currently the DevSecOps Leader for Intuit where she is responsible for setting and driving the company’s cloud security strategy, roadmap and implementation in support of corporate innova...
In his session at DevOps Summit, Tapabrata Pal, Director of Enterprise Architecture at Capital One, will tell a story about how Capital One has embraced Agile and DevOps Security practices across the Enterprise – driven by Enterprise Architecture; bringing in Development, Operations and Information Security organizations together. Capital Ones DevOpsSec practice is based upon three "pillars" – Shift-Left, Automate Everything, Dashboard Everything. Within about three years, from 100% waterfall, Capital One now has 500+ Agile Teams delivering quality software via Agile and DevOps practices.
The cloud has transformed how we think about software quality. Instead of preventing failures, we must focus on automatic recovery from failure. In other words, resilience trumps traditional quality measures. Continuous delivery models further squeeze traditional notions of quality. Remember the venerable project management Iron Triangle? Among time, scope, and cost, you can only fix two or quality will suffer. Only in today's DevOps world, continuous testing, integration, and deployment upend the time metric, the DevOps cadence reinvents project scope, and cost metrics expand past software ...
DevOps tends to focus on the relationship between Dev and Ops, putting an emphasis on the ops and application infrastructure. But that’s changing with microservices architectures. In her session at DevOps Summit, Lori MacVittie, Evangelist for F5 Networks, will focus on how microservices are changing the underlying architectures needed to scale, secure and deliver applications based on highly distributed (micro) services and why that means an expansion into “the network” for DevOps.
Between the compelling mockups and specs produced by your analysts and designers, and the resulting application built by your developers, there is a gulf where projects fail, costs spiral out of control, and applications fall short of requirements. In his session at DevOps Summit, Charles Kendrick, CTO and Chief Architect at Isomorphic Software, will present a new approach where business and development users collaborate – each using tools appropriate to their goals and expertise – to build mockups and enhance them all the way through functional prototypes, to final working applications. Lea...
SYS-CON Media announced today that XebiaLabs launched a popular blog feed on DevOps Journal with close to 2,000 story reads in less than a day. DevOps Journal is focused on this critical enterprise IT topic in the world of cloud computing. DevOps Journal brings valuable information to DevOps professionals who are transforming the way enterprise IT is done.
XebiaLabs has announced record growth and major highlights for 2014. These milestones include: Triple-digit worldwide revenue growth. Record number of new customers including 3M, Allianz, American Express, Credit Agricole, Digital Globe, Electronic Arts, EMC, Expedia, Fandango / NBC Universal, GATX, General Electric, ING, KPMG, Liberty Mutual, Natixis, Paychex, Providence Health, Societe General, Thomson Reuters TIIA-CREF and Umpqua Bank.
VictorOps is making on-call suck less with the only collaborative alert management platform on the market. With easy on-call scheduling management, a real-time incident timeline that gives you contextual relevance around your alerts and powerful reporting features that make post-mortems more effective, VictorOps helps your IT/DevOps team solve problems faster.
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, will discuss why containers should be paired with new architectural practices such as microservices rather than mimicking legacy server virtualization workflows and architectures.
Security can create serious friction for DevOps processes. We've come up with an approach to alleviate the friction and provide security value to DevOps teams. In her session at DevOps Summit, Shannon Lietz, Senior Manager of DevSecOps at Intuit, will discuss how DevSecOps got started and how it has evolved. Shannon Lietz has over two decades of experience pursuing next generation security solutions. She is currently the DevSecOps Leader for Intuit where she is responsible for setting and driving the company’s cloud security strategy, roadmap and implementation in support of corporate innova...
SYS-CON Events announced today that SoftLayer, an IBM company, has been named “Gold Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place June 9-11, 2015 at the Javits Center in New York City, NY, and the 17th International Cloud Expo®, which will take place November 3–5, 2015 at the Santa Clara Convention Center in Santa Clara, CA. SoftLayer operates a global cloud infrastructure platform built for Internet scale. With a global footprint of data centers and network points of presence, SoftLayer provides infrastructure as a service to leading-edge customers ranging from ...
RingCentral Inc., has unveiled the RingCentral Connect Platform™, a set of tools and services to build, deploy, and manage custom integrations using RingCentral APIs. With this platform, developers can build out-of-the-box integrations with RingCentral to add powerful communication capabilities to business applications. "This is a significant step forward because businesses want to better connect with customers – and communications is at the heart of every customer interaction," said Vlad Shmunis, CEO, founder and chairman of RingCentral. "In the past, integrating communications into busine...
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
Qubell announced on Monday the availability of the first autonomic application management platform for cloud applications. Qubell enables applications to become adaptive, self-managed services that configure, heal, optimize and protect themselves in response to changes within dynamic cloud environments. With Qubell, managing applications as an autonomic system dramatically accelerates application release cycles, improves quality, eliminates manual operations and reduces outages and operating costs. The platform is designed for a broad range of web, commerce and big data applications with an em...
Basis Technologies, leaders in streamlining, accelerating and optimizing SAP systems, today announces the launch of theirDevOps toolset, focused on helping companies running SAP move from slow, linear, and manual development to safer, more agile and responsive development and test workflows. Building on the success of their market-leading Transport Express solution, the DevOps toolset comprises 4 flexible solutions that address specific goals of DevOps; Transport Expresso, ShiftLeft, DevMax and SnapOps.
SYS-CON Events announced today Sematext Group, Inc., a Brooklyn-based Performance Monitoring and Log Management solution provider, will exhibit at SYS-CON's DevOps Summit 2015 New York, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), search analytics (SSA), and search enhancement. They also provide exceptional Search and Big Data consulting services a...
SYS-CON Events announced today that Liaison Technologies, a leading provider of data management and integration cloud services and solutions, has been named "Silver Sponsor" of SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York, NY. Liaison Technologies is a recognized market leader in providing cloud-enabled data integration and data management solutions to break down complex information barriers, enabling enterprises to make smarter decisions, faster.