@DevOpsSummit Authors: Ram Sonagara, Yeshim Deniz, Elizabeth White, Roger Strukhoff, Liz McMillan

Related Topics: @DevOpsSummit, Java IoT, Microservices Expo, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

Log Aggregation Across Dev and Ops | @DevOpsSummit #DevOps #Docker #Containers #Microservices

Modern tools for log aggregation can be hugely enabling for DevOps approaches

Using Log Aggregation Across Dev & Ops: The Pricing Advantage
By Rob Thatcher, Co-founder and Principal Consultant at Skelton Thatcher Consulting.

Summary: the pricing of tools or licenses for log aggregation can have a significant effect on organizational culture and the collaboration between Dev and Ops teams.

Modern tools for log aggregation (of which Logentries is one example) can be hugely enabling for DevOps approaches to building and operating business-critical software systems. However, the pricing of an aggregated logging solution can affect the adoption of modern logging techniques, as well as organizational capabilities and cross-team collaboration. We need to choose our log aggregation tools carefully to make sure that we don't introduce unintended barriers or silos driven by unhelpful pricing.

Enabling Devops

Avoid ‘Singleton' Log Aggregation That Exists Only in Production
Organizations considering commercial tooling for log management are often sold on the idea that the single most important location for log collection is the Production (Live) environment. Whilst first class logging facilities for live environments are hugely valuable, if we fail to provide log message collection in upstream environments (QA, Dev, Test, Pre-Prod, etc.) we miss out on a significant opportunity to discover how the application and infrastructure behave before reaching Production.

One of the most common causes of these ‘singleton' tools - tools that exist only in Production - is that the tool is licensed per server. We see that per-server licenses tend to drive down the number of servers on which the tool (or agent) is installed; in this scenario, licenses are purchased only for Production machines, leaving upstream environments with no log aggregation. Contrast this with other approaches to tool licensing, such as charging per GB of data transferred, or number of messages/queries per month, which provide a much better scaling model for the tool, and encourage teams to use the tool in Dev or Test environments as well as in Production.

If our log aggregation tool is available only in Production, developers tend to view logging as ‘just an Ops thing' and have little incentive or reason to care about improving log messages or adding useful metrics. However, if developers and operations people all use the same tooling for searching logs across all environments (Dev, Test, Prod) then they can begin to collaborate on a common view of the system behavior, gaining valuable insights into improving operability.

Selling the Benefits of Log Aggregation
Working with a variety of clients over the past 18 months, we have found that the benefits of log aggregation are often not well understood by technical teams.

Time and time again we speak to developers and operations people who are astonished at how easy it is to search for and find information in logs once those logs are in a central log aggregation tool.

They never want to go back to manual log ‘scraping' again.

It's therefore crucial when evaluating log aggregation tools to find products that can as much demonstrate the fundamentals of log aggregation as show off a specific vendor's product. The Free tier or 30-day demo version of the tool needs to have sufficient capability so that we can demonstrate end-to-end value: event streams from a whole set of servers, server-level metrics, integration with time-series graphing tools and dashboarding tools, and pre-baked queries (we particularly like Logentries for its features here). Teams often need to convince budget holders to pay for what seems initially like a packed version of existing free tools but - if demonstrated well and adequately evangelized - rapidly becomes an obvious new core capability.

When we make sure that the subsequent rollout of log aggregation spans both development and IT operations, then we lay the foundations for a culture of shared responsibility where diagnostics and time-to-recovery are highly valued, a crucial feature of modern web systems.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@DevOpsSummit Stories
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a multi-faceted approach of strategy and enterprise business development. Andrew graduated from Loyola University in Maryland and University of Auckland with degrees in economics and international finance.
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading these essential tips, please take a moment and watch this brief video from Sandy Carter.
DXWorldEXPO LLC announced today that "Miami Blockchain Event by FinTechEXPO" has announced that its Call for Papers is now open. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses. FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.