Welcome!

@DevOpsSummit Authors: Liz McMillan, Elizabeth White, Pat Romanski, Karthick Viswanathan, PagerDuty Blog

Related Topics: @DevOpsSummit, Microservices Expo, Microsoft Cloud, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

Cache Tagging with Redis By @Stackify | @DevOpsSummit [#DevOps]

The process of using the tag is to get all members of the tag's SET and take action on each member

Implementing Cache Tagging with Redis on Azure based Applications

Redis has quickly become one of the top open source key value stores around. We at Stackify have been using Windows Azure's Managed Cache but have had a long history of intermittent problems with it. So we decided it was time to bite the bullet and give Redis a try, which Azure now supports and actually recommends it over their previous Managed Cache. We had one big problem though. Several important parts of our system relied on cache tags and Redis has no out-of-the-box support for tags.

Summary (TL;DR)

We use a Lua script to create a Redis "SET" for each tag that contains all the keys associated with the tag. We have some logic to ensure expiration of the tags corresponds properly with the keys and we have some cleanup code to eliminate old keys created for tags no longer needed. See Sample project on GitHub based on ServiceStack and C#.

How we use tags

We use tags for a few different purposes but the most important is to track which users are currently logged in to our system and what are they currently looking at the UI. Stackify collects monitoring metrics data as it flows into our system. We use SignalR to update the UI in real-time to show the latest monitoring data and alert status. It would be a huge waste of resources to send every result over SignalR to the UI if no one was there to see it.

For every user logged in we store  what they are viewing in cache sort of like this:

Cache key: Client1-User1-UniquePageID
Cache value: true  (not really used)

Related Tags:

  • ServerID1
  • ServerID2
  • ServerID3

When results come in we look for all cache keys that have a specific tag "ServerID1", for example, then we know by the cache keys that had that tag, what users are logged in and if we should send SignalR data via the UI.

How did we implement tagging

We chose to implement tagging using Redis' SET datatype. Redis SETs are unordered and are a true set - meaning duplicate values are not allowed. Their rough analog in the .NET framework is HashSet<T>. To achieve tagging functionality in Redis, we represent each tag as a SET whose values are the keys to the cache entries that have been associated with that tag. From our example it looks something like this:

Key: tag:ServerID1

Value: Redis "SET"

  • Client1-User1-UniquePageID
  • Client1-User2-UniquePageID
  • Client1-User3-UniquePageID

The process of using the tag is then to get all members of the tag's SET and take action (MGET, DEL, EXISTS, etc.) on each member accordingly. In order to keep our operations atomic, we've taken advantage of Redis' built-in scripting capabilities. Redis guarantees that execution of Lua scripts will happen atomically (just as any other single command does), so we don't have to worry about leaving our key-tag associations in an indeterminate state.

Once we implement our tagging Lua script in our cache library, adding cache items with tags is simple. Here's an example below of the simple C# call to PUT a key-value pair with the tags "red" and "blue" associated to it.

Note: We are utilizing the ServiceStack library for connecting to Redis and our sample project on GitHub  also utilizes the ServiceStack with some additional methods we have made around tags. The Lua scripts in our sample project could be used via any programming language.

Lua script to add the tags. Need to learn more about Redis and Lua? Check out this nice beginners guide: Lua: A Guide for Redis Users

Sample of getting related keys by tag from our sample project:

GetKeyByAnyTag is an extension method that we made to ServiceStack that does the following. You could extend this to not only return keys but also return the objects themselves from cache. As part of that you may want to ensure that those keys still exist or TTL has not expired on them. We have done several variations in our own libraries but we opted to keep this sample project simple.

Cleaning up the tag lists

One thing we do have to worry about in this scenario is tag-set growth. While Redis will happily persist a cache entry which never expires, for our purposes using it as a cache, almost everything we persist to Redis is given an explicit TTL. This presents a problem in our chosen tagging strategy, because Redis only allows the expiration of entire entries - expiration cannot be set to a single value within any of Redis' collection data types. This means our tag SETs will eventually contain values that point to keys that have expired. For very active tags with an unbounded set of associated keys, this can result in a large amount of invalid data (up to the point of consuming all allocated memory), and time wasted attempting to operate on cache entries that don't exist. It's a gotcha type of problem without a highly elegant solution -  you just have to choose a least-ugly solution and make it work. For us, that turned out to be a tag-cleanup Lua script which is triggered probabilistically by other tag-related cache operations:

The cleanup is accomplished by iterating through every key in Redis that is a tag and evaluating if it is still used. You will see advice elsewhere suggesting that Redis' KEYS operation should never be run in production. Our experience so far suggests that its performance is acceptable for this type of an operation that is only being called a few times per day.

Tag Naming Consistency

It is important that within your caching framework around Redis that you use a consistent format for your tags. For example, our developers may use "blue" and "red" as a tag but we actually prefix it with a "tag:" so it becomes "tag:blue" within Redis. This is important so we know which Redis keys are actually tags and not normal keys without the developers having to think about the formatting of the tag names. This is also important for cleaning up the tags no longer be used. You may also need to prefix your tags (and keys) with a customer number or some other identifier if you have a multi-tenant system.

So..how well does it work?

We have been using this new implementation for over a couple months now and so far it has worked very well for us. We are doing thousands of Redis queries per minute looking for "tags" the way we implemented it and seeing 3-4ms response times for our queries which has performed well for us thus far.

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com

@DevOpsSummit Stories
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, visit http://www.krcnet.co.jp/eng_site/e_index.htm.
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp empowers global organizations to unleash the full potential of their data to expand customer touchpoints, foster greater innovation and optimize their operations.
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data and analytics insights onto a single, holistic, display, focusing attention on what matters, when it matters.
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
One of the biggest challenges with adopting a DevOps mentality is: new applications are easily adapted to cloud-native, microservice-based, or containerized architectures - they can be built for them - but old applications need complex refactoring. On the other hand, these new technologies can require relearning or adapting new, oftentimes more complex, methodologies and tools to be ready for production. In his general session at @DevOpsSummit at 20th Cloud Expo, Chris Brown, Solutions Marketing Manager at Nutanix, explored how Nutanix is bringing these sides together - agility for pets, governance for cattle - in a single unified platform. With this combined approach, Nutanix removes silos - both technological and human - propelling your applications to a new level.
Most of the time there is a lot of work involved to move to the cloud, and most of that isn't really related to AWS or Azure or Google Cloud. Before we talk about public cloud vendors and DevOps tools, there are usually several technical and non-technical challenges that are connected to it and that every company needs to solve to move to the cloud. In his session at 21st Cloud Expo, Stefano Bellasio, CEO and founder of Cloud Academy Inc., will discuss what the tools, disciplines, and cultural aspects are that enterprise companies are considering to get to the cloud and eventually transform the way they build software and services.
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In his session at @DevOpsSummit at 21st Cloud Expo, Chris Van Tuin, Chief Technologist, NA West at Red Hat, will discuss: The top security risks with containers and how to manage these risks at scale including Images, Builds, Registry, Deployment, Hosts, Network, Storage, APIs, Monitoring/Logging, and Federation.
SYS-CON Events announced today that B2Cloud will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. B2Cloud specializes in IoT devices for preventive and predictive maintenance in any kind of equipment retrieving data like Energy consumption, working time, temperature, humidity, pressure, etc.
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits, DevOps is correlated with 20% faster time-to-market, 22% improvement in quality, and 18% reduction in dev and ops costs, according to research firm Vanson-Bourne. It is changing the way IT works, how businesses interact with customers, and how organizations are buying, building, and delivering software.
Your clients expect transactions to never fail, cloud access to be fast and always on, and their data to be protected - no exceptions. Hear about how Secure Service Container (SSC), an IBM-exclusive open technology, enables secure building and hosting of next-generation applications, both cloud and on-premises. SSC protects the full stack from external and insider threats, allows automatic encryption of data in-flight and at-rest, and is tamper-resistant during installation and runtime – with no changes to applications required.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of engineers can navigate the Carrier Ecosystem for your IT team acting as an extension of your business, producing a hassle-free experience.
SYS-CON Events announced today that Suzuki Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Suzuki Inc. is a semiconductor-related business, including sales of consuming parts, parts repair, and maintenance for semiconductor manufacturing machines, etc. It is also a health care business providing experimental research for dementia, etc. For more information, visit http://www.e-suzuki.co.jp/en/.
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http://www.ryobi-sol.co.jp/en/.
Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Kubernetes was originally built by Google, leveraging years of experience with managing container workloads, and is now a Cloud Native Compute Foundation (CNCF) project. Kubernetes has been widely adopted by the community, supported on all major public and private cloud providers, and is gaining rapid adoption in enterprises. However, Kubernetes may seem intimidating and complex to learn. This is because Kubernetes is more of a toolset than a ready solution. Hence it’s essential to know when and how to apply the appropriate Kubernetes constructs.
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capability.
SYS-CON Events announced today that Daiya Industry will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Daiya Industry specializes in orthotic support systems and assistive devices with pneumatic artificial muscles in order to contribute to an extended healthy life expectancy. For more information, please visit https://www.daiyak.co.jp/en/.
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.interface-amita.com/aboutus/interface_profile.asp.
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists examined how DevOps helps to meet the demands of Digital Transformation – including accelerating application delivery, closing feedback loops, enabling multi-channel delivery, empowering collaborative decisions, improving user experience, and ultimately meeting (and exceeding) business goals.
Cloud Expo, Inc. has announced today that Andi Mann and Aruna Ravichandran have been named Co-Chairs of @DevOpsSummit at Cloud Expo Silicon Valley which will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. "DevOps is at the intersection of technology and business-optimizing tools, organizations and processes to bring measurable improvements in productivity and profitability," said Aruna Ravichandran, vice president, DevOps product and solutions marketing, CA Technologies. "It's this results-driven combination of technology and business that makes me so passionate about DevOps and its future in the industry. I am truly honored to take on this co-chair role, and look forward to working with the DevOps Summit team at Cloud Expo and attendees to advance DevOps."