High Availability

As more and more mission-critical applications move on the Internet, providing highly available services becomes increasingly important. One of the advantages of a clustered system is that it has hardware and software redundancy, because the cluster system consists of a number of independent nodes, and each node runs a copy of operating system and application software. High availability can be achieved by detecting node or daemon failures and reconfiguring the system appropriately, so that the workload can be taken over by the remaining nodes in the cluster.

In fact, high availability is a big field. An advanced highly available system may have a reliable group communication sub-system, membership management, quoram sub-systems, concurrent control sub-system and so on. There must be a lot of work to do. However, we can use some existing software packages to construct highly available LVS cluster systems now.

Working Principle

In general, there are service monitor daemons running on the load balancer to check server health periodically, as illustrated in the figure of LVS high availability. If there is no response for service access request or ICMP ECHO_REQUEST from a server in a specified time, the service monitor will consider the server is dead and remove it from the available server list at the load balancer, thus no new requests will be sent to this dead server. When the service monitor detects the dead server has recovered to work, the service monitor will add the server back to the available server list. Therefore, the load balancer can automatically mask the failure of service daemons or servers. Furthermore, administrators can also use system tools to add new servers to increase the system throughput or remove servers for system maintenance, without bringing down the whole system service.

Now the load balancer might become a single failure point of the whole system. In order to prevent the whole system from being out of service because of the load balancer failure, we need setup a backup (or several backups) of the load balancer. Two heartbeat daemons run on the primary and the backup respectively, they heartbeat the message like "I'm alive" each other through serial lines and/or network interfaces periodically. When the heartbeat daemon of the backup cannot hear the heartbeat message from the primary in the specified time, it will take over the virtual IP address to provide the load-balancing service. When the failed load balancer comes back to work, there are two solutions, one is that it becomes the backup load balancer automatically, the other is the active load balancer releases the VIP address, and the recover one takes over the VIP address and becomes the primary load balancer again.

The primary load balancer has state of connections, i.e. which server the connection is forwarded to. If the backup load balancer takes over without those connections information, the clients have to send their requests again to access service. In order to make load balancer failover transparent to client applications, we implement connection synchronization in IPVS, the primary IPVS load balancer synchronizes connection information to the backup load balancers through UDP multicast. When the backup load balancer takes over after the primary one fails, the backup load balancer will have the state of most connections, so that almost all connections can continue to access the service through the backup load balancer.

The availability of database, network file system or distributed file system is not addressed here.

Working Examples

There are several software packages in conjuction with LVS to provide high availability of the whole system, such as Red Hat Piranha, Keepalived, UltraMonkey, heartbeat plus ldirectord, and heartbeat plus mon.

The following examples of building highly available LVS systems are only for reference.

• Using Piranha to build highly available LVS systems
• Using Keepalived to build highly available LVS systems
• Using UltraMonkey to build highly available LVS systems
• Using heartbeat+mon+coda to build highly available LVS systems
• Using heartbeat+ldirectord to build highly available LVS systems

There must be many other methods to build highly available LVS systems, please drop me a message if you have your methods.

1. The Piranha Solution

Piranha is one of the clustering products from Red Hat Inc., it includes the IPVS kernel code, cluster monitoring tool and web-based cluster configuration tool.

The piranha monitoring tool has two main features:

• Heartbeating between active and backup load balancers.
• Checking availability of the services on each of real servers.

The piranha monitoring tool uses one process to heartbeat UDP messages between active and backup load balancers, and fork service monitoring processes so that each process is to monitoring one service at a real server.

Configuation example

Now we're going to using piranha to contruct a highly-available VS/NAT web cluster with two load balancers and three web servers. The topology is illustrated in the following figure. In the example, virtual IP address and gateway IP address are 10.23.8.80 and 172.18.1.254, which are floating between the two load balancers (LD1 and LD2), and the ip addresses of three real servers are 172.18.1.11, 172.18.1.12 and 172.18.1.13 respectively.

The configuration file of Piranha is the same at the LD1 and LD2. The configuration for our example looks like:

primary = 10.23.8.1
service = lvs
rsh_command = rsh
backup_active = 1
backup = 10.23.8.2
heartbeat = 1
heartbeat_port = 539
keepalive = 4
deadtime = 12
network = nat
nat_router = 172.18.1.254 eth1:0
nat_nmask = 255.255.255.0
reservation_conflict_action = preempt
debug_level = NONE
virtual web {
     active = 1
     address = 10.23.8.80 eth0:1
     vip_nmask = 255.255.255.255
     port = 80
     persistent = 600
     send = "GET / HTTP/1.0\r\n\r\n"
     expect = "HTTP"
     load_monitor = none
     scheduler = wlc
     protocol = tcp
     timeout = 6
     reentry = 15
     quiesce_server = 0
     server webserver1 {
         address = 172.18.1.11
         active = 1
         weight = 100
     }
     server webserver2 {
         address = 172.18.1.12
         active = 1
         weight = 100
     }
     server webserver3 {
         address = 172.18.1.13
         active = 1
         weight = 100
     }
}

2. The Keepalived Solution

Keepalived provides a strong and robust health checking for LVS clusters. It implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover.

Configuation example

Now we're going to using keepalived to contruct a highly-available VS/NAT web cluster with two load balancers and three web servers. The topology is illustrated in the following figure. In the example, virtual IP address and gateway IP address are 10.23.8.80 and 172.18.1.254, which are floating between the two load balancers (LD1 and LD2), and the ip addresses of three real servers are 172.18.1.11, 172.18.1.12 and 172.18.1.13 respectively.

In our example, the keepalived configuration file (/etc/keepalived/keepalived.conf) at the LD1 looks like:

vrrp_sync_group VG1 {
    group {
        VI_1
        VI_2
    }
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.23.8.80
    }
}
vrrp_instance VI_2 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.18.1.254
    }
}
virtual_server 10.23.8.80 80 {
    delay_loop 6
    lb_algo wlc
    lb_kind NAT
    persistence_timeout 600
    protocol TCP
    real_server 172.18.1.11 80 {
        weight 100
        TCP_CHECK {
            connect_timeout 3
        }
    }
    real_server 172.18.1.12 80 {
        weight 100
        TCP_CHECK {
            connect_timeout 3
        }
    }
    real_server 172.18.1.13 80 {
        weight 100
        TCP_CHECK {
            connect_timeout 3
        }
    }
}

The Keepalived configuration file at LD2 is similar to that of LD1, except to change the state of VI_1 and VI_2 from MASTER to BACKUP.

3. The UltraMonkey Solution

Ultra Monkey is a project to create load balanced and highly available services on a local area network using Open Source components on the Linux operating system, including heartbeat and ldirectord from the Linux-HA project.

Configuation example

Now we're going to using UltraMonkey to contruct a highly-available VS/NAT web cluster with two load balancers and three web servers. The topology is illustrated in the following figure. In the example, virtual IP address and gateway IP address are 10.23.8.80 and 172.18.1.254, which are floating between the two load balancers (LD1 and LD2), and the ip addresses of three real servers are 172.18.1.11, 172.18.1.12 and 172.18.1.13 respectively.

The configuration files of UltraMonkey are the same at LD1 and LD2. The configuration files for above examples are as follows:

/etc/ha.d/ha.cf:

logfacility     local0
keepalive 2
deadtime 10
warntime 10
initdead 10
nice_failback on
udpport 694
bcast   eth1
node    ld1
node    ld2

/etc/ha.d/haresources:

ld1 IPaddr::10.23.8.80/24/eth1 IPaddr::172.18.1.254/24/\
eth1 ldirectord::ldirectord.cf

/etc/ha.d/ldirectord.cf:

checktimeout=10
checkinterval=2
autoreload=no
logfile="local0"
quiescent=yes
virtual=10.23.8.80:80
	fallback=127.0.0.1:80
	real=172.18.1.11:80 masq
	real=172.18.1.12:80 masq
	real=172.18.1.13:80 masq
	service=http
	request="index.html"
	receive="Test Page"
	scheduler=wlc
	persistent=600
	protocol=tcp
	checktype=negotiate

4. The heartbeat+mon+coda solution

The high availability of virtual server can be provided by using of mon, heartbeat and coda software. The "mon" is a general-purpose resource monitoring system, which can be used to monitor network service availability and server nodes. The "heartbeat" code currently provides the heartbeats among two node computers through serial line and UDP heartbeats. Fake is IP take-over software by using of ARP spoofing. The architecture of this solution is illustrated in the following figure.

The server failover is handle as follows: The "mon" daemon is running on the load balancer to monitor service daemons and server nodes in the cluster. The fping.monitor is configured to detect whether the server nodes is alive every t seconds, and the relative service monitor is also configured to detect the service daemons on all the nodes every m seconds. For example, http.monitor can be used to check the http services; ftp.monitor is for the ftp services, and so on. An alert was written to remove/add a rule in the linux virtual server table while detecting the server node or daemon is down/up. Therefore, the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.

Now, the load balancer becomes a single failure point of the whole system. In order to mask the failure of the primary load balancer, we need setup a backup server of the load balancer. The "fake" software is used for the backup to takeover the IP addresses of the load balancer when the load balancer fails, and the "heartbeat" code is used to detect the status of the load balancer to activate/deactivate the "fake" on the backup server. Two heartbeat daemons run on the primary and the backup, they heartbeat the message like "I'm alive" each other through the serial line periodically. When the heartcode daemon of the backup cannot hear the "I'm alive" message from the primary in the defined time, it activates the fake to take over the Virtual IP address to provide the load-balancing service; when it receives the "I'm alive" message from the primary later, it deactivate the fake to release the Virtual IP address, and the primary comes back to work again.

However, the failover or the takeover of the primary load balancer will cause the established connection in the hash table lost in the current implementation, which will require the clients to send their requests again.

Coda is a fault-tolerant distributed file systems, a descendant of Andrew file system. The contents of servers can be stored in Coda, so that files can be highly available and easy to manage.

Configuation example

The following is an example to setup a highly available virtual web server via direct routing.

The failover of real servers

The "mon" is used to monitor service daemons and server nodes in the cluster. For example, the fping.monitor can be used to monitor the server nodes, http.monitor can be used to check the http services, ftp.monitor is for the ftp services, and so on. So, we just need to write an alert to remove/add a rule in the virtual server table while detecting the server node or daemon is down/up. Here is an example calleded lvs.alert, which takes virtual service(IP:Port) and the service port of real servers as parameters.

#!/usr/bin/perl
#
# lvs.alert - Linux Virtual Server alert for mon
#
# It can be activated by mon to remove a real server when the
# service is down, or add the server when the service is up.
#
#
use Getopt::Std;
getopts ("s:g:h:t:l:P:V:R:W:F:u");
$ipvsadm = "/sbin/ipvsadm";
$protocol = $opt_P;
$virtual_service = $opt_V;
$remote = $opt_R;
if ($opt_u) {
    $weight = $opt_W;
    if ($opt_F eq "nat") {
	$forwarding = "-m";
    } elsif ($opt_F eq "tun") {
	$forwarding = "-i";
    } else {
	 $forwarding = "-g";
    }
    if ($protocol eq "tcp") {
	system("$ipvsadm -a -t $virtual_service -r $remote -w $weight $forwarding");
    } else {
	system("$ipvsadm -a -u $virtual_service -r $remote -w $weight $forwarding");
    }
} else {
    if ($protocol eq "tcp") {
	system("$ipvsadm -d -t $virtual_service -r $remote");
    } else {
	system("$ipvsadm -d -u $virtual_service -r $remote");
    }
};

The lvs.alert is put under the /usr/lib/mon/alert.d directory. The mon configuration file (/etc/mon/mon.cf or /etc/mon.cf) can be configured to monitor the http services and servers in the cluster as follows.

#
# The mon.cf file
#
#
# global options
#
cfbasedir   = /etc/mon
alertdir   = /usr/lib/mon/alert.d
mondir     = /usr/lib/mon/mon.d
maxprocs    = 20
histlength = 100
randstart = 30s
#
# group definitions (hostnames or IP addresses)
#
hostgroup www1 www1.domain.com
hostgroup www2 www2.domain.com
#
# Web server 1
#
watch www1
    service http
	interval 10s
	monitor http.monitor
	period wd {Sun-Sat}
	    alert mail.alert wensong
	    upalert mail.alert wensong
	    alert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.1 -W 5 -F dr
	    upalert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.1 -W 5 -F dr
#
# Web server 2
#
watch www2
    service http
	interval 10s
	monitor http.monitor
	period wd {Sun-Sat}
	    alert mail.alert wensong
	    upalert mail.alert wensong
	    alert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.2 -W 5 -F dr
	    upalert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.2 -W 5 -F dr

Note that we need to set the paramter of lvs.alert like "lvs.alert -V 10.0.0.3:80 -R 192.168.0.3:8080" if the destination port is different in LVS/NAT.

Now the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.

The failover of the load balancer

In order to prevent the load balancer becoming a single failure point of the whole system, we need setup a backup of the load balancer and let them heartbeat each other periodically. Please read the GettingStarted document include the heartbeat package, it is simple to setup 2-node heartbeating system.

For an example, we assume that the two load balancers have the following addresses:

lvs1.domain.com (primary)	10.0.0.1
lvs2.domain.com (backup)	10.0.0.2
www.domain.com (VIP)	10.0.0.3

After install the heartbeat on both lvs1.domain.com and lvs2.domain.com, simply create the /etc/ha.d/ha.conf as follows:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#       hopfudge maximum hop count minus number of nodes in config
hopfudge 1
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport 1001
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lvs1.domain.com
node    lvs2.domain.com

The /etc/ha.d/haresources file is as follows:

lvs1.domain.com 10.0.0.3 lvs mon

The /etc/rc.d/init.d/lvs is as follows:

#!/bin/sh
#
# You probably want to set the path to include
# nothing but local filesystems.
#
PATH=/bin:/usr/bin:/sbin:/usr/sbin
export PATH
IPVSADM=/sbin/ipvsadm
case "$1" in
    start)
	if [ -x $IPVSADM ]
	then
	    echo 1 > /proc/sys/net/ipv4/ip_forward
	    $IPVSADM -A -t 10.0.0.3:80
	    $IPVSADM -a -t 10.0.0.3:80 -r 192.168.0.1 -w 5 -g
	    $IPVSADM -a -t 10.0.0.3:80 -r 192.168.0.2 -w 5 -g
	fi
	;;
    stop)
	if [ -x $IPVSADM ]
	then
	    $IPVSADM -C
	fi
	;;
    *)
	echo "Usage: lvs {start|stop}"
	exit 1
esac
exit 0

Finally, make sure that all the files are created on both the lvs1 and lvs2 nodes and alter them for own own environment, then start the heartbeat daemon on the two nodes.

Note that fake (IP takeover by Gratuitous Arp) is already included in the heartbeat package, so there is no need to setup "fake" separately. When the lvs1.domain.com node fails, the lvs2.domain.com will take over all the haresources of the lvs1.domain.com, i.e. taking over the 10.0.0.3 address by Gratuitous ARP, start the /etc/rc.d/init.d/lvs and /etc/rc.d/init.d/mon scripts. When the lvs1.domain.com come back, the lvs2 releases the HA resources and the lvs1 takes them back.

5. The heartbeat+ldirectord solution

The ldirectord (Linux Director Daemon) written by Jacob Rief is a stand-alone daemon to monitor services of real servers, currently http and https service. It is simple to install and it works with the heartbeat code. The latest version of ldirectord can be found at the CVS repository of heartbeat. See 'perldoc ldirectord' for all the information about ldirectord. Thank Jacob Rief for writing this great program!

The advantages of ldirectord over mon are as follows:

• The ldirectord is specially written for LVS monitoring.
  It reads configuration files like /etc/ha.d/xxx.cf, which contains all the information about the IPVS routing table configuration. When the ldirectord is up, the IPVS routing table will be configured properly. We can also save different virtual service configuration in multiple configuration files, so that it is possible to modify parameters of some services without bringing down other services.
• The ldirectored can be easily started and stopped by heartbeat.
  Put the ldirectord under the /etc/ha.d/resource.d/ directory, then we can add a line in the /etc/ha.d/haresources like:

      node1 IPaddr::10.0.0.3 ldirectord::www ldirectord::mail
  

Anyway, the ldirectord can also be started and stopped manually. We can also use it in our LVS clusters without the backup of load balancer.

Configuation example

Taking the example introduced in the heartbeat+mon+coda solution, we can configure the /etc/ha.d/www.cf as follows:

#
# The /etc/ha.d/www.cf for ldirectord
#
# the number of second until a real server is declared dead
timeout = 10
# the number of second between server checks
checkinterval = 10
#
# virtual = x.y.z.w:p
#     protocol = tcp|udp
#     scheduler = rr|wrr|lc|wlc
#     real = x.y.z.w:p gate|masq|ipip [weight]
#     ...
#
virtual = 10.0.0.3:80
     protocol = tcp
     scheduler = wlc
     real = 192.168.0.1:80 gate 5
     real = 192.168.0.2:80 gate 5
     request = "/.testpage"
     receive = "test page"

The /etc/ha.d/haresources file is simple as follows:

    lvs1.domain.com IPaddr::10.0.0.3 ldirectord::www

We need to create the .testpage file at the DocumentRoot directory of each web server.

    echo "test page" > .testpage

Start the heartbeat daemons on the primary and the backup. If there is anything wrong, we may check the /var/log/ha-log and /var/log/ldirector.log respectively.

Source from:http://www.linuxvirtualserver.org/HighAvailability.html