Welcome!

@DevOpsSummit Authors: Elizabeth White, Yeshim Deniz, Pat Romanski, Liz McMillan, John Rauser

Related Topics: @DevOpsSummit, Java IoT, Linux Containers

@DevOpsSummit: Blog Feed Post

Getting Real About Memory Leaks | @DevOpsSummit #APM #DevOps #ContinuousTesting

Modern programming languages tend to separate the programmer from memory management

Getting Real About Memory Leaks
By Matt Heusser

Modern programming languages tend to separate the programmer from memory management; Java programmers don't have to deal with pointers; they just declare variables and let the built-in garbage collector do its thing.

These garbage collectors are smart, but not perfect; they typically work by object reference. When all the references to an object go out of memory, that object can go out of memory too. Yet if two objects point to each other, they will always have a reference count, and never go away. That means the code written in Javascript to run in a browser, or Objective-C to run on a hand-held phone, or ASP to run on a server could very well have a memory leak.

When memory leaks happen on modern Operating Systems, the Operating System isn't going to throw an error. Instead, it will just use more and more memory, eventually going to virtual memory on the disk. When that happens, performance falls apart. People in operations reboot the server and that "seems to fix it"; users close and restart their browser, or reboot their phone.

Consider the Following Example
My first job in software was to create an electronic audit of a telecommunications bill. Some other software, maintained by my coworker Jeff, would read accounts, phones, and calls from a database and then create a text file that could be interpreted by a printer, called a post-script file. If we were smart, we'd have had written the tool in two parts, first to create a bunch of excel-like text files, then a second piece to combine those into bills.

One day, about four months in, the President of our 30-person software company came downstairs to my desk and said "Matt, I want you to take over the telecom bills", doing maintenance and new feature development.

The program was written in C. It read a list of accounts and looped through every account. Not every active account, every account, because an inactive account could still have an unpaid balance. We calculated the balance by adding up all the debits and credits through all time. If that amount added up to zero, we could skip the account. That was no big deal when we were helping a new, competitive regional carrier get started, with forty or fifty accounts and billing for five or six months.

The competitive carrier, however, was good at, well, competing. By the time I took over they had eighteen hundred accounts, and would be in the tens of thousands (and three states) before I left. Performance would become a real problem, but not for the reasons I expected.

The first two problems that hit us were a pointer out of range, then a memory leak.

Memory Out of Bounds
The software needed to store a great deal of data in memory for each account (the buildings, phones, calls, costs, and taxes) then spit it out to disk in post-script format. There are two basic ways to do this in C: Either allocate memory with pointers, which is hard to track, or simulate it using arrays. We didn't have objects, Test Driven Development wasn't really a thing, and I was inheriting code designed to support a very small customer base. What we did have was arrays. So to support two thousand calls, we could do something like this:

struct call {

char inOrOutbound;

datetimestamp timeStart;

datetimestamp timeEnd;

integer billedMinutes;

double cost;

}

char buildings  [200][200];

char phones [200][200][1000][1000];

call calls [200][200][1000][1000];

What's represented here is up to 200 buildings per account, each of which can have 200 phones, each of which can have up to 1000 calls. The variable calls [5][10][25] would contain the fifth call for the tenth phone at the twenty fifth building. Actually, since C is zero-based, it would be the 6th call at the third phone at the eleventh building.

The software was basically a loop. For each account, for each building, for each phone, for each call. The calls needed to be sorted, then the results printed. Everything worked fine until in one run I got a memory out of bounds error. That's probably because the code was trying to access building 201, phone 201, or call 1001. The compiler has range checking built-in by default. To get around this, I moved the size of the data structure up as large as possible, which worked until we got an account that was too big for that.

Then I turned off range checking and declared a large amount of empty data structures, matching the originals in type, below the originals. That is an incredibly foolish thing to do, but it worked. Because the data structures were built into the compiled program (the ".exe file") and would be re-used for every account. Our little program would declare all the memory it needed one time, at the beginning, and use that.

Until we found the system slowing to a crawl on big accounts, like it was choking.

Enter The Memory Leaks
The data structures above are an over-simplification. We also needed to store the addresses of the building, the payments since the last bill, and so on. Like the examples above some of these were declared variables, put in the compiled code, "on the stack". Others were grabbed at run-time, using a C program call named "malloc" and its friend "free." Instead of declaring an array, we'd declare a character pointer, then malloc() some space at that pointer. When the function was done, we could free() the disk space.

If we remembered to do that.

As you've probably guessed, we had too many calls to malloc() and not enough to free(). That meant that memory use slowly crept up as we used the software. That's fine for fifty or even five hundred accounts; the memory use would go away when the program finished running. But, we started to get some very big accounts. I noticed that one big account would run, and performance would go down. The kicker, for me, was an out of memory error for our user. I went to Rich, the sysadmin, and asked for more memory for the user. He suggested I tune it instead. After carefully poring over the code, I found some mallocs() that weren't being freed. Adding the calls to free, the program did eventually finish. That account still took too long, but that's enough for today.

You might see how I struggled, but you might not see what that has to do with modern software delivery. Allow me to share just a bit more.

The Lesson for Today
These problems are silent killers, and fixing them is no longer as easy as searching through a program making sure every malloc() has a matching free().

If you write embedded medical software, or avionics, or anything in C, you are probably very familiar with these problems. If you don't, you might not be, but the problems are still there.

If your application degrades over time, the problem might just be memory leaks. The next logical question is how to find them.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

@DevOpsSummit Stories
It is ironic, but perhaps not unexpected, that many organizations who want the benefits of using an Agile approach to deliver software use a waterfall approach to adopting Agile practices: they form plans, they set milestones, and they measure progress by how many teams they have engaged. Old habits die hard, but like most waterfall software projects, most waterfall-style Agile adoption efforts fail to produce the results desired. The problem is that to get the results they want, they have to change their culture and cultures are very hard to change. To paraphrase Peter Drucker, "culture eats Agile for breakfast." Successful approaches are opportunistic and leverage the power of self-organization to achieve lasting change.
Cloud Expo, Inc. has announced today that Andi Mann and Aruna Ravichandran have been named Co-Chairs of @DevOpsSummit at Cloud Expo Silicon Valley which will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. "DevOps is at the intersection of technology and business-optimizing tools, organizations and processes to bring measurable improvements in productivity and profitability," said Aruna Ravichandran, vice president, DevOps product and solutions marketing, CA Technologies. "It's this results-driven combination of technology and business that makes me so passionate about DevOps and its future in the industry. I am truly honored to take on this co-chair role, and look forward to working with the DevOps Summit team at Cloud Expo and attendees to advance DevOps."
"When we talk about cloud without compromise what we're talking about is that when people think about 'I need the flexibility of the cloud' - it's the ability to create applications and run them in a cloud environment that's far more flexible,” explained Matthew Finnie, CTO of Interoute, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Wooed by the promise of faster innovation, lower TCO, and greater agility, businesses of every shape and size have embraced the cloud at every layer of the IT stack – from apps to file sharing to infrastructure. The typical organization currently uses more than a dozen sanctioned cloud apps and will shift more than half of all workloads to the cloud by 2018. Such cloud investments have delivered measurable benefits. But they’ve also resulted in some unintended side-effects: complexity and risk. End users now struggle to navigate multiple environments with varying degrees of performance. Companies are unclear on the security of their data and network access. And IT squads are overwhelmed trying to monitor and manage it all.
"We are a monitoring company. We work with Salesforce, BBC, and quite a few other big logos. We basically provide monitoring for them, structure for their cloud services and we fit into the DevOps world" explained David Gildeh, Co-founder and CEO of Outlyer, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 21st Int\ernational Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their online business and let Enzu manage their IT hosting infrastructure.
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud computing technologies. Get certified, manage the full lifecycle of your cloud-based resources, and build your knowledge based using Cloud Academy’s expert-created content, comprehensive Learning Paths, and innovative Hands-on Labs.
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex software systems for startups and enterprises. Since 2009 it has grown from a small group of passionate engineers and business managers to a full-scale mobile software company with over 200 developers, designers, quality assurance engineers, project managers in house, specializing in the world-class mobile and web development.
What's the role of an IT self-service portal when you get to continuous delivery and Infrastructure as Code? This general session showed how to create the continuous delivery culture and eight accelerators for leading the change. Don Demcsak is a DevOps and Cloud Native Modernization Principal for Dell EMC based out of New Jersey. He is a former, long time, Microsoft Most Valuable Professional, specializing in building and architecting Application Delivery Pipelines for hybrid legacy, and cloud native applications.
SYS-CON Events announced today that GrapeUp, the leading provider of rapid product development at the speed of business, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market across the USA and Europe, we work with a variety of customers from emerging startups to Fortune 1000 companies.
SYS-CON Events announced today that Ayehu will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara California. Ayehu provides IT Process Automation & Orchestration solutions for IT and Security professionals to identify and resolve critical incidents and enable rapid containment, eradication, and recovery from cyber security breaches. Ayehu provides customers greater control over IT infrastructure through automation. Ayehu solutions have been deployed by major enterprises worldwide, and currently, support thousands of IT processes across the globe. The company has offices in New York, California, and Israel.
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cloud marketplaces and DevOps are changing the economics of hosting and delivering software.
@DevOpsSummit at Cloud Expo taking place Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center, Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the application economy. With CA software at the center of their IT strategy, organizations can leverage the technology that changes the way we live - from the data center to the mobile device. CA's software and solutions help customers thrive in the new application economy by delivering the means to deploy, monitor and secure their applications and infrastructure.
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to the new world.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be looking at some significant engineering investment. On-demand, serverless computing enables developers to try out a fleet of devices on IoT gateways with ease. With a sensor simulator built on top of AWS Lambda, it’s possible to elastically generate device sensors that report their state to the cloud.
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 20th Cloud Expo, which will take place on June 6-8, 2017 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 additional third-party data centers across Europe. Its full-service Unified ICT platform serves international enterprises and many of the world’s leading service providers, as well as governments and universities.
SYS-CON Events announced today that Clouber will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Clouber offers Migration as a Service (MaaS) across Private and Public Cloud (AWS, Azure, GCP) including bare metal migration to cloud. Clouber’s innovative technology allows for migration projects to be completed in minutes instead of weeks. For more updates follow #clouberio
SYS-CON Events announced today that Striim will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Striim is pronounced "stream", with two i's for integration and intelligence. The company was founded in 2012 as WebAction, with a mission to help companies make data useful the instant it's born. The leaders behind the Striim platform thrive on building technology companies that raise expectations for how the world does business. The team include core executives from GoldenGate Software (acquired by Oracle in 2009), Informatica, Oracle, SnapLogic, Embarcadero Technologies, PubNub and WebLogic. It is led by Ali Kutay who was an angel investor, president and CEO of WebLogic, as well as Chairman and CEO of GoldenGate Software.
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet the demands of Digital Transformation – including accelerating application delivery, closing feedback loops, enabling multi-channel delivery, empowering collaborative decisions, improving user experience, and ultimately meeting (and exceeding) business goals.