@DevOpsSummit Authors: Ram Sonagara, Yeshim Deniz, Elizabeth White, Roger Strukhoff, Liz McMillan

Related Topics: @DevOpsSummit, Java IoT, @CloudExpo

@DevOpsSummit: Article

How to Lead with Analytics By @HoardingInfo | @DevOpsSummit [#DevOps]

The speed and performance of the modern software delivery pipeline brings a new set of challenges

Today, everything must be faster. Faster releases; faster and larger backlogs; faster complaints; faster bugs. The speed and performance of the modern software delivery pipeline brings a new set of challenges. How do you keep up? There really is only one option - automation and measurement. You need a method for automatically collecting data about your application and how it is being used. But also a good way to find the relevant information and act on it. And in order to do this your development team needs to lead with analytics first.

There are many types of analytics platforms. The breadth of each platform is determined by the amount and type of data they contain from the delivery pipeline.

Log management and analytics butters across all data produced by the pipeline. And for this reason it is one place the team can centralize the collection of new data. Data collection is great and relatively easy, but there are a few problems.

  1. Data stored without a purpose or way to analyze has no value.
  2. IT often owns Log analysis and sometimes developers have limited access or participation.

Yes, you can log everything; and you should. The reason you should is because getting data retroactively is expensive and difficult; and it might not provide value right away.  But another reason is to improve  communication and ownership around the log analysis service itself. Often with log analysis, Ops owns its implementation and usage. Depending on the culture of the teams this could hinder the development team from gaining value or insights from the tool. In a worse case scenario IT locks out all data and usage that is not system and infrastructure related. In the best case developers have their own level of access or even their own tools, and are storing data in the system as well.


Without everyone working in, storing, analyzing and sharing value from the log analysis service there are guaranteed gaps in communication and issue resolution. It is critical that the entire team use the system to ensure consistent communication and continuity between application layer and system layer issues. Not only will developers help enrich the platform, they also can benefit from bugs that are identified more quickly.

So how can the developers better use log analysis platform? Here are 10 things developers can do to lead with analytics.

In Your Code

  1. Exception Blocks: This is a simple idea, but not commonly done. In your exception blocks, log the error message or maybe even the entire stack trace.  This is a great way to create dashboards on error types. And historically measure the number and frequency of error types.
  2. Log Trigger Events: Log critical actions such as user activity. This data can later be related to user cohorts for marketing, product management, and growth hacking purposes. But also for the developers to optimize user flow. For example, events you expect a user to take, but they do not.  Or accidental actions such as a logout.
  3. Run Duration for Bigger Processes: For large process intensive or lengthy application processes, log start and stop times. It can then be correlated to pegged servers, outages or load issues if they arise. This data can help to identify areas where the product is slow to respond, or issues of application processes effecting infrastructure.
  4. Log Deprecated Functionality: When deprecated functionality is called, log it so the team can be alerted. This will help QA and Development know when outdated classes are being used by other team members by accident.
  5. Log flag Changes: Some developers like to use flags in their code to indicate different states. Create a function for changing flags so that it is logged and clear when flags are off or on. This allows the reporting or alerting of when a particular flag that should have been off, remained on in production.
  6. Log Component Usage: Applications are a treasure pile of 3rd party component libraries. These libraries pose a risk when they are not monitored. For all the components used, it is good to log their usage - component name and version.  There is a downside as this can be very chatty if not designed well. And it has to be backed by some good component tracking process outside of the collection, and from the insights.

With the Team

  1. Architect Your Logs: Make sure the development team agrees on a standard for logging key value pairs. A consistent language and format so that everyone on the team can understand it.
  2. Leverage Tags: Creating the logs is part of it, but create relevant tags once the logging happens. For example "User" for user data. "Error" for the above error exception data.
  3. Create Dashboards: If you log it, use it. Create dashboards give them convenient names and share them with the team. Also create alerting and email reports on regular basis. There does need to be a process to review and prune unused or old dashboards or you will have dashboard sprawl/confusion.
  4. Unify System to Application: Help the broader team understand the relationship of applications to the infrastructure they run on. For example apache functionality specific to a new version, ties to orchestration scripts or containers, etc. If you do this successful IT may not have to ask you about OS level configurations and backend code, they can fix it themselves.

For a team to be metrics- and results-driven, they need to lead with analytics. And to lead with analytics means to log everything and establish easy ways to leverage what is logged. Developers have a lot to gain from a valuable system and application logging service including reducing bugs, faster time to resolution, and better insight across infrastructure and application layers.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@DevOpsSummit Stories
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a multi-faceted approach of strategy and enterprise business development. Andrew graduated from Loyola University in Maryland and University of Auckland with degrees in economics and international finance.
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading these essential tips, please take a moment and watch this brief video from Sandy Carter.
DXWorldEXPO LLC announced today that "Miami Blockchain Event by FinTechEXPO" has announced that its Call for Papers is now open. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses. FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.