Welcome!

@DevOpsSummit Authors: Liz McMillan, Elizabeth White, David Linthicum, Zakia Bouachraoui, Yeshim Deniz

Related Topics: @DevOpsSummit, Linux Containers, @CloudExpo, @DXWorldExpo

@DevOpsSummit: Blog Post

Keepalived and HAProxy in AWS | @DevOpsSummit [#DevOps]

We're going to explore high availability and load balancing using Keepalived and HAProxy

Keepalived and HAProxy in AWS: An Exploratory Guide

This article originally published on the Logentries Blog.

We're going to explore high availability and load balancing using Keepalived and HAProxy.

Keepalived is a routing software designed to provide simple and robust facilities for load balancing and high-availability to Linux systems and Linux-based infrastructures.

HAProxy is an open source load balancer/reverse proxy generally used for load balancing web services, but also has the functionality to load balance TCP traffic.

Together, Keepalived and HAProxy provide some unique benefits

Specifically, they provide a low cost solution for high availability compared to proprietary hardware-based load balancers.

Keepalived and Haproxy in AWS

Keepalived uses the VRRP protocol to detect when HAProxy is down and fails with minimal downtime.

The following documentation is focused on setting up Keepalived in AWS (or potentially any cloud) where multicast is not supported, and you're using a debian-based OS.

Keepalived and HAProxy Setup

First, download the latest version of Keepalived.

(http://www.keepalived.org/software/keepalived-1.2.13.tar.gz)

You will need this version since the previous version in your repository is probably too old to support VRRP Unicast.

wget http://www.keepalived.org/software/keepalived-1.2.13.tar.gz
tar xf keepalived-1.2.13.tar.gz
cd keepalived-1.2.13/
./configure
make && make install

cp keepalived/etc/init.d/keepalived.init /etc/init.d/keepalived

mkdir /etc/keepalived/

Keepalived and HAProxy Architecture

We're going to assume two load balancer servers are running Keepalived and HAProxy, where one is the master, the other is a failover, and we have multiple web servers.

Keepalived and HAProxy AWS Architecture

In this scenario, we have an elastic IP on ServerA while ServerB is set as a passive server not being hit by the public internet. We also have five web servers running our app.

By default ServerA is set as MASTER state and ServerB is set as BACKUP, which means on startup the servers know their state, and also means on a failover/fail back scenario both servers know their role.

In a failure scenario where ServerA becomes unresponsive (i.e HAProxy process hangs, server dies etc.) ServerB will be unable to contact serverA. Therefore, ServerA enters a FAULT mode and ServerB becomes master. When ServerB becomes master it runs a script which will basically disassociate the Elastic IP from ServerA and assign it to ServerB. ServerB is now the only active HAProxy server and handles requests to the app servers.

Configs

A few online references seem to have incorrect configs for setting up Keepalived in a Unicast environment, however the following instructions seem to work correctly.

We need to set up our master and slave configs for Keepalived slightly different as they both have different roles to play in failover. The vrrp_script is used to verify the other party is functioning correctly. We also have in our notify_master section, /etc/keepalived/master.sh, a script which assigns the Elastic IP to the master in a case of failure or recovery.

Server A (master) (/etc/keepalived/keepalived.conf)

ServerB - BackupStarting Keepalived

Keepalived startup messages are in /var/log/syslog, to see if your process started up correctly you should see an output similar to below:

ServerA - Master

There are many ways to test connectivity, tshark, tcpdump etc, here are some sample commands:Verify Connectivity

tcpdump "ip proto 112" 
tshark -f "vrrp"

The reason we need to verify connectivity with either of the commands above is because Keepalived uses multicast, which is not supported in AWS. The commands verify our config is correctly sending via Unicast.


Verify Connectivity Keepalived Haproxy AWS tshark

Testing Failover

Assuming you have your failover.sh script setup correctly, on stopping HAProxy on ServerA, you should see it entering a FAULT state and see ServerB entering a MASTER state. When ServerB enters a master state it will run a script and reassign the Elastic IP to ServerB.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@DevOpsSummit Stories
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO Silicon Valley 2019 will cover all of these tools, with the most comprehensive program and with 222 rockstar speakers throughout our industry presenting 22 Keynotes and General Sessions, 250 Breakout Sessions along 10 Tracks, as well as our signature Power Panels. Our Expo Floor will bring together the leading global 200 companies throughout the world of Cloud Computing, DevOps, IoT, Smart Cities, FinTech, Digital Transformation, and all they entail. As your enterprise creates a vision and strategy that enables you to create your own unique, long-term success, learning about all the technologies involved is essential. Companies today not only form multi-cloud and hybrid cloud architectures, but create them with built-in cognitive capabilities.
In today's always-on world, customer expectations have changed. Competitive differentiation is delivered through rapid software innovations, the ability to respond to issues quickly and by releasing high-quality code with minimal interruptions. DevOps isn't some far off goal; it's methodologies and practices are a response to this demand. The demand to go faster. The demand for more uptime. The demand to innovate. In this keynote, we will cover the Nutanix Developer Stack. Built from the foundation of software-defined infrastructure, Nutanix has rapidly expanded into full application lifecycle management across any infrastructure or cloud .Join us as we delve into how the Nutanix Developer Stack makes it easy to build hybrid cloud applications by weaving DBaaS, micro segmentation, event driven lifecycle operations, and both financial and cloud governance together into a single unified st...
For far too long technology teams have lived in siloes. Not only physical siloes, but cultural siloes pushed by competing objectives. This includes informational siloes where business users require one set of data and tech teams require different data. DevOps intends to bridge these gaps to make tech driven operations more aligned and efficient.
Daniel Jones is CTO of EngineerBetter, helping enterprises deliver value faster. Previously he was an IT consultant, indie video games developer, head of web development in the finance sector, and an award-winning martial artist. Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams.
Hackers took three days to identify and exploit a known vulnerability in Equifax’s web applications. I will share new data that reveals why three days (at most) is the new normal for DevSecOps teams to move new business /security requirements from design into production. This session aims to enlighten DevOps teams, security and development professionals by sharing results from the 4th annual State of the Software Supply Chain Report -- a blend of public and proprietary data with expert research and analysis.Attendees can join this session to better understand how DevSecOps teams are applying lessons from W. Edwards Deming (circa 1982), Malcolm Goldrath (circa 1984) and Gene Kim (circa 2013) to improve their ability to respond to new business requirements and cyber risks.