Welcome!

@DevOpsSummit Authors: Elizabeth White, Liz McMillan, Pat Romanski, Yeshim Deniz, Stackify Blog

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Feed Post

Putting the Test Back into DevOps | @DevOpsSummit #DevOps #BigData #Microservices

DevOps has moved from being something based on speculation about what development could be, to a real part of the developer role

Putting the Test Back into DevOps
by Justin Rohrman

DevOps has become synonymous with empowering developers to move faster and deliver more software, but has unintentionally moved software quality and testing into a corner. The pendulum that swings back and forth between very slow testing and staying far away from overly technical solutions has swung hard with DevOps, completely ignoring the value that testers can bring by forcing customers to test new code and creating teams that rely monitoring systems to find things that could have been caught before customers noticed.

DevOps has moved from being something based on speculation about what development could be, to a real part of the developer role. Let's see how we can swing this trend back to a more responsible place, still using DevOps to release software faster but with a level of quality that customers will be happy with.

Continuous Integration and Delivery
Continuous Integration
(CI) and Delivery (CD) are two of the most fundamental concepts built into DevOps. They have helped us talk about tools that will build new products every time a new line of code is added to the repository and then as soon as that build is done, deploy it to production. Some of the more technology focused companies like Github have pushed this idea as far as it can go and are deploying new software to production many times every day.

The side effect of this style of fast paced development and release is that any time for a real live person to try the software out before delivering is squeezed out. As soon as there is software that can run, it goes into production. Paying customers are the new testers, and monitoring and reporting systems are the new bug reports.

If we dial this back a little bit, we get a strategy where developers do CI on their own environment, getting a new build every time they commit to their local source code repository. And after checking in to the main repo, code is continuously deployed to a staging environment.

I have had a lot of success with continuous deployment to a headless API server on smaller teams. One team I worked on had two people working on the API platform that the rest of our product was built on. Each time they would commit code, a new build would go to that server. Normally before that would happen, we would sit together and talk about the changes and their concerns. I might start stubbing out a few automated checks, and writing down some test ideas.

We were still using DevOps concepts to build an API quickly, but we were doing it in a way that didn't force customers to deal with our problems.

Monitoring Is Only Part of the Solution
A few large companies are using complex API and Web monitoring systems running in the background, sucking up large amounts of log data from the API looking for a few important keywords like Exception, and Error. Every time these words are found, emails and text messages are sent to let the development staff know that something has gone wrong.

Maybe you have a completely componentized product where you can flip bits to turn features on and off quickly to reduce exposure to faults. Some companies have bits and pieces of product that are built as components, a handful have a large amount. Most have little or none. For smaller companies, this would mean spending equal amounts of time developing architecture as on building product. That isn't a ratio most founders and investors would want to see.

Monitoring and rollback systems are great for API products, but are usually not feasible for young companies that urgently need product to sell.

Landmines for Consumers and Holes in Your Revenue
Using monitoring without pre-production testing is dangerous for you and your consumers. Imagine you are releasing a new version of your API that will help customers to use Geolocation information on their phone to find stores nearby. The code under the hood of the API has some automated checks using faked data to see if stores were returned correctly, but there are only a few and they are pretty simple.

One of your first customers to use this feature live in the island paradise of Hawaii and it just so happens that there are lots of cities there with special characters in the name. A few hours after deploy, the server log files are blowing up with errors from people in Kāne‘ohe and ‘Ewa Gentry trying to find the closest Dunkin Donuts.

Emails start flying through the development office and the feature gets shut off minutes later, but the damage is done. By the time your support person gets in touch with a developer, and that developer investigates and takes action, your customers have given up. About half of those users uninstalled the app and are searching with Google instead.

"Testing new API changes can expose your product to black swan problems in a way that DevOps never will."

There are real consequences for using your customers to test new code. Placing a skilled tester in front of that API probably would have brought up questions like "What happens for cities with really long names, or special characters, or very small cities that might not be in all mapping systems". Testing new API changes can expose your product to black swan problems in a way that DevOps never will.

Cases like this actually happen and are good examples of where using DevOps concepts to deliver internally, rather than to production, would have saved a few customers and company money.

DevOps presents a powerful set of ideas that can help to deliver code to customers faster than we used to. They can also be a dangerous way to deliver bad code and buggy product much faster than we would like to. If we slow down a little bit by pairing DevOps themes with skilled testers, these ideas and tools can help to deliver software and API updates faster than would be possible without them without exposing our customers to new kinds of risk.

Do you have experience integrating DevOps with testing? We would love to hear your story.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

@DevOpsSummit Stories
SYS-CON Events announced today that DivvyCloud will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. DivvyCloud software enables organizations to achieve their cloud computing goals by simplifying and automating security, compliance and cost optimization of public and private cloud infrastructure. Using DivvyCloud, customers can leverage programmatic Bots to identify and remediate common cloud problems in real time. DivvyCloud was founded by seasoned technologists who understand first-hand what is necessary to succeed in today’s fast-changing, multi-cloud world.
SYS-CON Events announced today that Tintri, Inc, a leading provider of enterprise cloud infrastructure, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Tintri offers an enterprise cloud platform built with public cloud-like web services and RESTful APIs. Organizations use Tintri all-flash storage with scale-out and automation as a foundation for their own clouds – to build agile development environments for cloud native applications and to run mission-critical enterprise applications. Only Tintri enables organizations to guarantee the performance of their applications, automate common IT tasks to reduce operating expenses, troubleshoot across their infrastructure, and predict their needs to scale – the underpinnings of a modern data center.
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry’s single source for the cloud. Fusion’s advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including cloud communications, cloud connectivity, and cloud computing. Fusion’s innovative, yet proven cloud solutions lower our customers’ cost of ownership, and deliver new levels of security, flexibility, scalability, and speed of deployment.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus internal resources on core competencies.
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs often get requests to connect these silos using technologies such as VPN; however, these tend to be difficult to manage and are not engineered for accessing business data from the cloud.
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT services.
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits, DevOps is correlated with 20% faster time-to-market, 22% improvement in quality, and 18% reduction in dev and ops costs, according to research firm Vanson-Bourne. It is changing the way IT works, how businesses interact with customers, and how organizations are buying, building, and delivering software.
SYS-CON Events announced today that Hitachi Data Systems, a wholly owned subsidiary of Hitachi LTD., will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City. Hitachi Data Systems (HDS) will be featuring the Hitachi Content Platform (HCP) portfolio. This is the industry’s only offering that allows organizations to bring together object storage, file sync and share, cloud storage gateways, and sophisticated search and analytics to create a tightly integrated, simple and smart cloud-storage solution. HCP provides massive scale, multiple storage tiers, powerful security, cloud capabilities, and multitenancy with configurable attributes for each tenant, all backed by legendary Hitachi reliability. Designed to preserve data for long durations, HCP carries built-in data protection mechanisms and is designed to flue...
The 21st International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ product base consists of a wide variety of off-the-shelf PC/104 single board computers, computer-on-modules, touch panel computers, peripherals and industrial controllers. They also offer custom configurations and design services. Technologic Systems specializes in the ARM and X86 architectures, FPGA IP-core design, and open-source software support based on Linux, Android, and Windows, providing adv...
SYS-CON Events announced today that Carbonite will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Carbonite protects your entire IT footprint with the right level of protection for each workload, ensuring lower costs and dependable solutions with DoubleTake and Evault.
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet the demands of Digital Transformation – including accelerating application delivery, closing feedback loops, enabling multi-channel delivery, empowering collaborative decisions, improving user experience, and ultimately meeting (and exceeding) business goals.
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs often get requests to connect these silos using technologies such as VPN; however, these tend to be difficult to manage and are not engineered for accessing business data from the cloud.
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a single platform to drive unprecedented simplicity in the data center. Customers can start with a base infrastructure and scale to multi-site and multi-geo infrastructures with predictable economics and performance.
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value Stream Mapping for the development and operations tool chain by offering DevOps Tool Chain Integration and Traceability; DevOps Tool Chain Orchestration; and DevOps Insight and Intelligence. CollabNet also offers traditional application lifecycle management, ALM, for the enterprise through its TeamForge product.
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that Infranics will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Since 2000, Infranics has developed SysMaster Suite, which is required for the stable and efficient management of ICT infrastructure. The ICT management solution developed and provided by Infranics continues to add intelligence to the ICT infrastructure through the IMC (Infra Management Cycle) based on mathematical analysis and forecasting Big Data Analyze and Control.
SYS-CON Events announced today that HTBase will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. HTBase (Gartner 2016 Cool Vendor) delivers a Composable IT infrastructure solution architected for agility and increased efficiency. It turns compute, storage, and fabric into fluid pools of resources that are easily composed and re-composed to meet each application’s needs. With HTBase, companies can quickly provision resources and deploy unique, mission-critical, self-designed solutions to add-onto or create any type of infrastructure as per the business requirement. HTBase is the first company to enable a true multi-cloud strategy, enabling organizations to automate movement of data and workloads between private and public clouds. This means that organizations can now move data and workloads between pub...
SYS-CON Events announced today that Juniper Networks (NYSE: JNPR), an industry leader in automated, scalable and secure networks, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Juniper Networks challenges the status quo with products, solutions and services that transform the economics of networking. The company co-innovates with customers and partners to deliver automated, scalable and secure networks with agility, performance and value.
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on quality and value.
This talk centers around how to automate best practices in a multi-/hybrid-cloud world based on our work with customers like GE, Discovery Communications and Fannie Mae. Today’s enterprises are reaping the benefits of cloud computing, but also discovering many risks and challenges. In the age of DevOps and the decentralization of IT, it’s easy to over-provision resources, forget that instances are running, or unintentionally expose vulnerabilities.