@DevOpsSummit Authors: Elizabeth White, Pat Romanski, Liz McMillan, Yeshim Deniz, Zakia Bouachraoui

Related Topics: @DevOpsSummit, Java IoT, Linux Containers

@DevOpsSummit: Blog Post

"New DevOps Plug-in" By @TrevParsons | @DevOpsSummit [#DevOps]

Logs are the most fine-grained data source for understanding today's system

As co-founder of Logentries I am often asked - "Why Logs?"  And I have to admit, upon first impression, ‘log management and analytics' does not seem like the sexiest space :). However at Logentries we are here to redefine that space, to provide a solution to access, manage and understand your log data that is easy to use, cost effective and intelligent (i.e., it does the hard work so you don't have to).  But that being said it still begs the question, "Why logs?"

Logs are the most fine-grained data source for understanding today's system. Unlike traditional monitoring and analytics tools which provide an aggregate view of what is happening in your system (such as server monitoring, application performance monitoring, web analytics etc.), logs capture every single event so that you can understand not only the general trends, but EXACTLY what happened, in what order, and by whom. Logs allow you to view this level of detail in real-time or to review it in a post-mortem fashion. At the same time, they can be rolled up into dashboards to give you a high level view of what is happening across your system. So in effect they can provide the best of both worlds: the low level detail of exactly what has happened as well as the high level trends across your systems.

However the biggest issue with many logging solutions today  is:

  • Too expensive: Keeping all that log data around for more that 30 days has been prohibitively expensive, so deep historical system understanding has been difficult to achieve with logs. People have instead turned to the traditional monitoring tools that give summary views that can span back indefinitely due to the ability to store this data in a much more cost effective manner vs. raw logs
  • Too difficult to use: Logging providers expect you to learn their query language, requiring deep technical skills and a lot of time on your hands to get value from them.
  • Too difficult to maintain: In particular open source or in house solutions are  difficult and costly to maintain and organizations quickly get frustrated with their in house logging solution.

At Logentries we address (and continue to address) each of the above points. We want you to send us all your data, and to make this available  in an easy-to-use, accessible and cost effective manner.

And sending us all of your data has just become even easier with our new Shinkin/Nagios and Diamond integrations:

  • Nagios Plug-in via Shinken: Shinken is an open source monitoring framework, that is compatible with your Nagios plugins, but improves some of the traditional issues with the Nagios framework (e.g. scalability). The Logentries NagiosLogentries Nagios Plug-in
  • Shinken module allows you to send results of your Nagios or Shinken health checks to Logentries such that you can get a real time view of the health of your infrastructure, correlated with your traditional log data. You can also easily maintain a history of your health checks which has always been difficult with tools like Nagios, so it's easier now to look back historically at any major issues and to identify and recurring themes.
  • Diamond: Diamond is a python daemon for collecting metrics. It also has a bunch of collectors that provide the ability to collect detailed performance metrics from your OS as well as from common components like Hadoop, Mongo, Kafka, MySQL, NetApp, RabbitMQ, Redis, AWS S3... The new Logentries Diamond handler allows you to stream all of these metrics into your Logentries account in real time so you can easily visualize them in dashboards and again correlate with any traditional logs from your systems or apps.

Check out these new IT and Dev Ops plug-in designed to continue to provide the deepest, most fine-grained view of your system-wide operational data.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@DevOpsSummit Stories
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with extensive global expertise as a strategist, technologist, innovator, marketer, and communicator. For over 30 years across five continents, he has built success with Fortune 500 corporations, vendors, governments, and as a leading research analyst and consultant.
The current environment of Continuous Disruption requires companies to transform how they work and how they engineer their products. Transformations are notoriously hard to execute, yet many companies have succeeded. What can we learn from them? Can we produce a blueprint for a transformation? This presentation will cover several distinct approaches that companies take to achieve transformation. Each approach utilizes different levers and comes with its own advantages, tradeoffs, costs, risks, and outcomes.
Contino is a global technical consultancy that helps highly-regulated enterprises transform faster, modernizing their way of working through DevOps and cloud computing. They focus on building capability and assisting our clients to in-source strategic technology capability so they get to market quickly and build their own innovation engine.
You want to start your DevOps journey but where do you begin? Do you say DevOps loudly 5 times while looking in the mirror and it suddenly appears? Do you hire someone? Do you upskill your existing team? Here are some tips to help support your DevOps transformation. Conor Delanbanque has been involved with building & scaling teams in the DevOps space globally. He is the Head of DevOps Practice at MThree Consulting, a global technology consultancy. Conor founded the Future of DevOps Thought Leaders Debate. He regularly supports and sponsors Meetup groups such as DevOpsNYC and DockerNYC.
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the benefits of the cloud without losing performance as containers become the new paradigm.