flyinweb's blog - 系统架构

keepalived+haproxy实现web服务的高可用和负载均衡

shanyiwan@live.com() — Fri, 23 Sep 2011 00:49:15 GMT

一、简介:
keepalived是一个类似于layer3, 4 & 5交换机制的软件，也就是我们平时说的第3层、第4层和第5层交换.Keepalived的作用是检测web服务器的状态,如果有一台web服务器死机,或工作出现故障，Keepalived将检测到,并将有故障的web服务器从系统中剔除,当web服务器工作正常后Keepalived自动将web服务器加入到服务器群中,这些工作全部自动完成,不需要人工干涉,需要人工做的只是修复故障的web服务器.

Haproxy 反向代理服务器,支持双机热备支持虚拟主机,但其配置简单，拥有非常不错的服务器健康检查功能,当其代理的后端服务器出现故障, HAProxy会自动将该服务器摘除,故障恢复后再自动将该服务器加入.新的1.3引入了frontend,backend,frontend根据任意HTTP请求头内容做规则匹配,然后把请求定向到相关的backend.

二、实验环境以及服务器信息:

1. OS:   RedHat AS 5.1
2. 软件列表:
      keepalived-1.2.2.tar.gz
      haproxy-1.4.13.tar.gz

3. 服务器信息:
Master server 10.10.0.99   (调度主服务器)
Slave server    10.10.0.98   (从调度服务器)
       VIP: 10.10.0.97                   (调度服务器的虚拟IP)

Real server:
10.10.0.96
10.10.0.95

三、keepalived、haproxy安装配置
1、安装keepalived
# tar zxvf keepalived-1.2.2.tar.gz
# cd keepalived-1.2.2
# ./configure –-prefix=/usr/local/keepalived
# make&&make install
# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
# mkdir /etc/keepalived
# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
# chkconfig –add keepalived
# chkconfig keepalived on
# vi /etc/keepalived/keepalived.conf

global_defs {
notification_email {
xxxxxx@domain.com
}
notification_email_from xxxxxx@domain.com
smtp_server mail.domain.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 51
priority 100
advert_int
authentication {
auth_type PASS
auth_pass 1111
    }
virtual_ipaddress {
10.10.0.97
}
}
virtual_server 10.10.0.97 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP
    real_server 10.10.0.95 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 10.10.0.96 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}
关于keepalived配置文件的选项解释可以去网上搜下由于选项参数居多这里就不说明了
# service keepalived start

2、安装haproxy
#   tar zxvf haproxy-1.4.13.tar.gz
#   mv haproxy-1.4.13 haproxy
#   cd haproxy
#   make TARGET=linux26
#   vi /usr/local/haproxy/conf/haproxy.conf
#   mkdir –p /var/chroot/haproxy

global
        chroot /var/chroot/haproxy
        daemon
        gid     0
        log 127.0.0.1 local3
        nbproc 2
        pidfile /var/run/haproxy-private.pid

        ulimit   -n        65535
        user    root
        maxconn         32000
        spread-checks           4
        tune.maxaccept          8
        tune.maxpollevents      100
defaults sxit
        log     global
        mode    http
        option httplog
        option dontlognull
        log 127.0.0.1 local3
        retries 3
        option redispatch
        maxconn     32000
        contimeout      5000
        clitimeout        50000
        srvtimeout       50000
listen sxit 0.0.0.0:80
        appsession JSESSIONID len 52 timeout 3h
        cookie SRV insert indirect nocache
        mode http
        stats enable
        stats hide-version
        stats uri /haproxy-stats
        stats realm Haproxy\ statistics
        stats auth sxit:sxit
        stats refresh 3s
        monitor-uri /haproxy_test
        balance roundrobin
        option httpclose
        option forwardfor
        option httpchk HEAD /index.html HTTP/1.0
        server s1 10.10.0.95:80 check inter 2000 weight 3
        server s3 10.10.0.96:80 check inter 2000 weight 3

# /usr/local/haproxy/sbin/haproxy –f /usr/local/haproxy/config/haproxy.config

如果启动没有报什么错误的话,就在浏览器上输入如下地址
http://10.10.0.97/haproxy-stats (查看服务器状态信息的页面,登录的时候输入上面设置的账号和密码sxit),页面状态如下：

测试负载均衡效果以及高可用性负载均衡测试:
   启动真实服务器的web服务,在2个真实服务器上创建2个首页文件,内容分别为test1和test2,如果在浏览器上访问web服务每次刷新既显示test1又显示tetst2就说明负载均衡已经生效了.
高可用性测试:
拔掉主调度服务器的网线或者关闭主调度服务器,看下VIP是否顺利的切换到从调度器,假如切换正常,那就说明keepalived已经成功生效了.

基于LVS的互联网应用架设攻略

shanyiwan@live.com() — Fri, 18 Mar 2011 07:44:36 GMT

可扩展、高可用服务网络设计方案实现技术细节。

网络环境

1、硬件：服务器、网络附属存储（NAS）和交换机。3个服务器用来做web,2个服务器
来做流媒体，1个服务器做LVS-DR,2个mysql服务器，一个邮件服务器，2个交换机，一个NETAPP NAS。
2、运行环境：流媒体windows,其他的都是linux。
3、逻辑结构：除数据库服务器和NETAPP存储外，其他的服务器都使用2个网络地址，一个公网地址和一个私有网络地址。设置为公网ip的网络接口连接在一个交换机，设置为私有网络ip的网络接另外一个交换机，处于安全和网络带宽考虑，网络存储设备和数据库只使用私有网络地址。网络拓扑图如下所示：

基本原理

传统模式下，用户的访问请求通过DNS服务器解析后，把服务请求转发给web服务器，取得数据后返回给用户。这种模式有2个麻烦：同时访问的用户增加到某个程度后，服务器不能提供所需的正常访问；遇到故障，所有的访问请求都将失败。要解决这样一个难题,LVS是上上之选。当我们采用lvs方案之后，更改dns服务器的记录，这样用户的访问将首先到达LVS控制器所在的服务器，LVS把请求按照某种算法转发给后面真正的服务器。那么数据的返还是怎样的一个过程呢？在采用DR方式的集群形式下，真实服务器直接把数据返还给用户而不再经过LVS控制器。访问数据的流向在上图中用带箭头的虚线标识出来了，这样设计使得结构更简单一些，lvs控制器的压力也小很多。

根据应用的实际情况考虑，本项目采用LVS/DR方式。

技术实现

先列出个相关服务器的ip地址：

一、修改DNS记录

www IN A 61.135.55.160
media IN A 61.135.55.161
修改bind完成后测试一下，看是否被正确的解析。注意：主机记录应该解析到虚拟地址。

二、配置LVS/DR

LVS/DR主要由控制器和真实服务器2部分构成，需要在控制器和真实服务器上做好配置才能提供正常的服务，下面分步来说明。

（一）安装控制器部分：安装好系统（我用的是centos 5）,指定ip地址61.135.55.100/24，关闭不必要的系统/网络服务（执行ntsysv用上下键和空白键来完成）。实现LVS/DR最重要的两个东西是ipvs内核模块和ipvsadm工具包，幸运的是，当前的发行版已经包含ipvs内核模块，不必再像旧的内核版本需要打这个补丁，ipvsadm需要从网上下载安装，下面总结一下这个过程：

1、检查内核模块，看ipvs 模块是否被加载

上图实现，ipvs模块没有被加载，可以手动加载，当然这不是必需的（当安装好ipvsadm包后，执行ipvsadm命令就会把ip_vs加载到系统内核）。执行命令 modprobe ip_vs 就可以把ip_vs模块加载到内核。现在再执行 lsmod –l | grep ip_vs 应该看见ip_vs模块被列出。

2、安装ipvsadm。Ipvsadm的官方下载地址为 http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz ，解压后先做一个链接文件,把目录/usr/src/kernels/2.6.18-8.el5-i686/ 链接为/usr/src/linux,不这样做的话，执行./configure脚本将报错。运行命令 ln –s /usr/src/kernels/2.6.18-8.el5-i686 /usr/src/linux 做好链接，再运行不带参数的脚本 ./configure，然后执行”make;make install”完成安装。

（二）控制器配置：既可以使用脚本也可以更改系统的配置文件 /etc/sysconfig/ipvsadm。在实际应用中，我建议用脚本，这样的话，维护和移植lvs会很方便。下面给出本案使用的lvs/dr脚本：

[root@mysql2 ~]# more /usr/local/bin/lvsdr #!/bin/bash RIP1=61.135.55.150RIP2=61.135.55.151RIP3=61.135.55.152VIP1=61.135.55.160VIP2=61.135.55.161/etc/rc.d/init.d/functions case"$1"instart) echo " start LVS of DirectorServer"# set the Virtual IP Address and sysctl parameter /sbin/ifconfig eth0:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up /sbin/ifconfig eth0:1 $VIP2 broadcast $VIP2 netmask 255.255.255.255 up /sbin/route add -host $VIP1 dev eth0:0/sbin/route add -host $VIP2 dev eth0:1echo "1">/proc/sys/net/ipv4/ip_forward #Clear IPVS table /sbin/ipvsadm -C #set LVS #Web Apache /sbin/ipvsadm -A -t $VIP1:80-s wlc -p 120/sbin/ipvsadm -a -t $VIP1:80-r $RIP1:80-g /sbin/ipvsadm -a -t $VIP1:80-r $RIP2:80-g /sbin/ipvsadm -a -t $VIP1:80-r $RIP3:80-g #Media (mms) /sbin/ipvsadm -A -t $VIP2:1755-s rr -p 3600/sbin/ipvsadm -a -t $VIP2:1755-r $RIP3:1755-g /sbin/ipvsadm -a -t $VIP2:1755-r $RIP4:1755-g /sbin/ipvsadm -A -t $VIP2:554-s rr -p 3600/sbin/ipvsadm -a -t $VIP2:554-r $RIP3:554 –g /sbin/ipvsadm -a -t $VIP2:554-r $RIP4:554 –g #Run LVS /sbin/ipvsadm ;; stop) echo "close LVS Directorserver"echo "0">/proc/sys/net/ipv4/ip_forward /sbin/ipvsadm -C /sbin/ifconfig eth0:0 down /sbin/ifconfig eth0:1 down ;; *) echo "Usage: $0 {start|stop}"exit 1esac

.RIP1=61.135.55.150到RIP3=61.135.55.152定义3个真实服务器的ip地址。
.VIP1=61.135.55.160，VIP2=61.135.55.161定义2个虚拟ip地址，一个作web服务的虚拟地址，一个做流媒体服务的虚拟地址。
. /etc/rc.d/init.d/functions,执行这个系统脚本，以取得运行其他脚本所需的环境和参数。
.case语句是一个多路选择，本脚本给出3个：start,stop及*,3选1。$1表示脚本带一个参数。
. /sbin/ifconfig eth0:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up把web服务所需的ip地址（虚拟地址）绑定在辅助接口eth0:0。在LVS方案中，虚拟ip地址与普通网络接口大大不同，这点需要特别注意。虚拟ip地址的广播地址是它本身，子网掩码是255.255.255.255。为什么要这样呢？因为有若干机器要使用同一个ip地址，用本身做广播地址和把子网掩码设成4个255就不会造成ip地址冲突了，否则lvs将不能正常转发访问请求。
. /sbin/route add -host $VIP1 dev eth0:0添加主机路由，这2条可有可无，较新的linux发行版能正确路由这个主机地址。
. echo "1" >/proc/sys/net/ipv4/ip_forward启用ip转发功能。
. /sbin/ipvsadm –C清空ipvs转发表。
. /sbin/ipvsadm -A -t $VIP1:80 -s wlc -p 120 添加一个虚拟服务，服务协议是tcp(-t)；服务类型是web（$VIP1:80）；-s 表示采用wlc这种调度算法转发数据包（调度算法包括：rr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）；-p表示连接的持续时间为120秒，这个会话时间是根据实际情况调整的，如果这个值设置得不合理，用户将得到非常糟糕的访问效果。下面举例简单说明一下：

从上图我们可以看出，随着时间的变化，用户的请求将可能被lvs转发到不同的服务器，而那些需要保持会话的请求将被丢失，导致访问不能进行。

. /sbin/ipvsadm -a -t $VIP1:80 -r $RIP1:80 –g 以直接路由的方式把请求转发到LVS后面的真实服务器。我曾企图把web（80端口）请求转发到真实服务器的其他端口（如8000），但不能如愿。

余下的行参照上面的解释，理解起来应该不是问题，因此不再一一说明。脚本写好后，把它放在目录/usr/local/bin,然后授与执行权限（chmod 700 /usr/local/bin/lvsdr），运行这个脚本，LVS/DR控制器部分就算配置好了。如果脚本不能正常运行，多半情况是脚本书写错误所致，如在windows用写字板写脚本再拷贝到linux,或者写丢了某个“;”等等。不管真实服务器端是否正确设置lvs,LVS/DR控制器都能独个运行。有2个方法检验LVS/DR是否正常运行了：（1）查看内核是否列出ip_vs模块；（2）直接运行ipvsadm –l看输出是否有转发规则。

（三）真实服务器配置虚拟ip地址。LVS可以把服务请求转发到各种各样的操作系统，在本案中有2种操作系统：centos和windows 2003 server。其他unix的处理跟centos(一种linux发行版)相似。

（1） centos服务器设置虚拟服务器：与LVS/DR控制类似，既可以修改配置文件也可以用脚本，相对来讲，还是脚本方便，下面是某个服务器设置虚拟ip地址的脚本： [root@WEB2 ~]# more /usr/local/bin/lvs #!/bin/bash #description:start realserver #chkconfig 2352626VIP1=220.194.55.160/etc/rc.d/init.d/functions case"$1"instart) echo " start LVS of REALServer"/sbin/ifconfig lo:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore echo "2">/proc/sys/net/ipv4/conf/all/arp_announce ;; stop) /sbin/ifconfig lo:0 down echo "close LVS Directorserver"echo "0">/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0">/proc/sys/net/ipv4/conf/lo/arp_announce echo "0">/proc/sys/net/ipv4/conf/all/arp_ignore echo "0">/proc/sys/net/ipv4/conf/all/arp_announce ;; *) echo "Usage: $0 {start|stop}"exit 1esac

. /sbin/ifconfig lo:0 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up 跟LVS/DR控制器一样，广播地址设置为虚拟地址本身，子网掩码4个255，不同的是，虚拟ip地址被绑定在环回（loopback）子接口，而不是物理接口的子接口。
. echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore 一共四行，其目的是关闭arp响应。

（2）windows服务器设置虚拟地址。windows下设置子网掩码为255.255.255.255比linux设置要麻烦些。要想在网上邻居本地连接的tcp/ip属性设置4个255掩码是不能得逞的，唯一的办法是修改注册表。默认状况下，windows并没有环回接口存在，配置之前得先安装这个“设备”。接下来介绍一下环回接口设置步骤：

①控制面板点击添加新硬件

选“网络适配器”，按“下一步”，选“Microsoft”及“Microsoft Loopback Adapter”

点击下一步安装好loopback adapter.。

②设置loopback的tcp/ip参数值。

先设置ip，把子网掩码设置成255.255.255.0。

设置ip地址的目的是方便在注册表中搜索loopback设置子网掩码的位置，我们用设置的虚拟ip地址做搜索关键字，很快就找到位置了。

搜索“61.135.55.160”

找到ip地址“61.135.55.160”所在的位置，在这个项的下方，有个项“SubnetMask”，它的值为255.255.255.0.

把其修改为255.255.255.255,但不幸的是，windows 2003 server 的注册表修改编辑方式是2进制，修改时需要技巧。在windows xp的注册表编辑器上修改好，然后转换到2进制方式，windows 2003 server 对照这个值更改即可。

换成2进制方式

修改好一个项（SubnetMask）后，按F3修改余下的几个项的SubnetMask 值为255.255.255.255，然后重启windows就可以生效了。 三、运行LVS/DR

在LVS/DR运行lvs脚本，在真实服务器上启用虚拟地址，就可以把整个LVS/DR运行起来了。

几个需要关注的问题

（一）控制器高可靠性。一个普遍的做法是使用HA,用2个服务器做双机。在条件有限的情况下，又考虑不增加网络结构的复杂性，可以把LVS/DR控制器脚本放在不同的服务器上，一旦当前使用的LVS/DR控制器出故障，立即启用其它服务器的控制器脚本，可以把停机时间控制在可以接受的范围。

（二）安全。出来在真实服务器上启用安全机制外，LVS/DR控制器也需要做防火墙策略的。下面是某个LVS/DR控制器的防火墙脚本，请大家参考：

[root@mysql2 ~]# more /usr/local/bin/firewall  
#!/bin/bash  
#this is a common firewall created by 2007-7-29  
 
#define some variable   
IPT=/sbin/iptables  
CONNECTION_TRACKING="1" 
INTERNET="eth0" 
CLASS_A="10.0.0.0/8" 
CLASS_B="172.16.0.0/12" 
CLASS_C="192.168.0.0/16" 
CLASS_D_MULTICAST="224.0.0.0/4" 
CLASS_E_RESERVED_NET="240.0.0.0/5" 
BROADCAST_SRC="0.0.0.0" 
BROADCAST_DEST="255.255.255.255" 
IPADDR=61.135.55.100  
LOOPBACK_INTERFACE="lo" 
 
#Remove any existing rules  
$IPT -F  
$IPT -X  
 
#setting default firewall policy  
$IPT --policy OUTPUT ACCEPT  
$IPT --policy FORWARD DROP  
$IPT -P INPUT DROP  
 
 
#stop firewall  
if [ "$1" = "stop" ]  
then  
echo "Filewall completely stopped!no firewall running!"  
exit 0  
fi  
 
#setting for loopback interface  
$IPT -A INPUT -i lo -j ACCEPT  
$IPT -A OUTPUT -o lo -j ACCEPT  
 
# Stealth Scans and TCP State Flags  
# All of the bits are cleared  
$IPT -A INPUT -p tcp --tcp-flags ALL NONE -j DROP  
# SYN and FIN are both set  
$IPT -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP  
# SYN and RST are both set  
$IPT -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP  
# FIN and RST are both set  
$IPT -A INPUT -p tcp --tcp-flags FIN,RST FIN,RST -j DROP  
# FIN is the only bit set, without the expected accompanying ACK  
$IPT -A INPUT -p tcp --tcp-flags ACK,FIN FIN -j DROP  
# PSH is the only bit set, without the expected accompanying ACK  
$IPT -A INPUT -p tcp --tcp-flags ACK,PSH PSH -j DROP  
# URG is the only bit set, without the expected accompanying ACK  
 
$IPT -A INPUT -p tcp --tcp-flags ACK,URG URG -j DROP  
 
# Using Connection State to By-pass Rule Checking  
if [ "$CONNECTION_TRACKING" = "1" ]; then  
$IPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT  
$IPT -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT  
$IPT -A INPUT -m state --state INVALID -j DROP  
$IPT -A OUTPUT -m state --state INVALID -j DROP  
fi  
 
##################  
# Source Address Spoofing and Other Bad Addresses  
 
# Refuse spoofed packets pretending to be from  
# the external interface.s IP address  
$IPT -A INPUT -i $INTERNET -s $IPADDR -j DROP  
 
# Refuse packets claiming to be from a Class A private network  
$IPT -A INPUT -i $INTERNET -s $CLASS_A -j DROP  
 
# Refuse packets claiming to be from a Class B private network  
$IPT -A INPUT -i $INTERNET -s $CLASS_B -j DROP  
 
# Refuse packets claiming to be from a Class C private network  
$IPT -A INPUT -i $INTERNET -s $CLASS_C -j DROP  
 
$IPT -A INPUT -i $INTERNET -s 0.0.0.0/8 -j DROP  
$IPT -A INPUT -i $INTERNET -s 169.254.0.0/16 -j DROP  
$IPT -A INPUT -i $INTERNET -s 192.0.2.0/24 -j DROP  
######################  
#setting access rules  
 
#enable ssh connect  
$IPT -A INPUT -i $INTERNET -p tcp --dport 22 -j ACCEPT  
$IPT -A INPUT -i $INTERNET -p tcp --dport 80 -j ACCEPT  
$IPT -A INPUT -i $INTERNET -p tcp --dport 1755 -j ACCEPT  
$IPT -A INPUT -i $INTERNET -p tcp --dport 554 -j ACCEPT  
$IPT -A INPUT -i $INTERNET -p udp --dport 554 -j ACCEPT  
$IPT -A INPUT -i $INTERNET -p tcp --dport 8080 -j ACCEPT  
$IPT -A INPUT -i $INTERNET -p udp --dport 1024:5000 -j ACCEPT

在这个防火墙脚本中，--dport 1024:5000 udp端口是用于mms协议的，在项目实施过程中，没有注意这个端口，结果导致流媒体服务请求不能被转发到真实服务器。

（三）数据同步。所有相同服务的服务器挂接共享服务器的同一个目录，写入数据实际上是写同一个文件或目录，因此不再需要rsycn这样占资源的同步工具。

（四）LVS/DR维护和监控。系统在运行过程中，某个真实服务器多运行的服务很可能出故障，但ipvsadm本身不探测这个事件，它仍然按照某种算法将一些用户的请求转发给出故障的服务器，导致一些用户不能正常访问。Ldirectord可以动态的处理这个麻烦，也可以自己写个小工具，定期到真实服务器获取访问返还状态码，根据返还状态码执行相关的ipvsadm维护操作。监控报警方面，Nagios(www.nagios.org)是非常好的选择，当然，监控系统最好放在LVS/DR环境之外，关于Nagios的细节，请参照我的文章” 《Nagios远程监控软件的安装与配置详解.》

作者：田逸

apache+jk+tomcat集群+session同步

shanyiwan@live.com() — Fri, 18 Dec 2009 02:00:38 GMT

作者：刘宇 liuyu.blog.51cto.com

msn群:mgroup49073@hotmail.com （linuxtone）

说明：借前人之鉴，写一篇关于tomcat集群及session同步的问题，首先介绍tomcat集成及做集群的原因和必要性，session同步的作用。然后包括各软件的安装配置。

原理：tomcat 做个WEB服务器有它的局限性，处理能力低，效率低。承受并发小（1000左右）。但目前有不少网站或者页面是JSP的。并采用了tomcat做为WEB，因此只能在此基础上调优。

目前采取的办法是Apache + Mod_JK + tomcat 来解决一部分请求，用户访问的是apache，但有jsp页面的时候才会去请求tomcat。如果量一大，那么tomcat无法承受，那么只能做tomat集群，Apache + Mod_JK 就是负载均衡器了。

Mod_JK2负载均衡可以把不同的jsp请求转发到不同的tomcat服务器，还可以侦测服务器存活。如果有条件可以给Mod_JK2做一个HA因为做完集群后压力就在JK上了。

简单拓仆图：

一、测试环境及软件安装：-----如果已经安装过的可以省略这步

linux 2.6 内核 centos 5.2

本例二台tomcat 不在同一机器。

先安装了所需的程序库

yum -y install gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel curl curl-devel

下载源码包：（本文不安装PHP）

apache2 和tomcat5都可以在 apache.org 下载

tomcat5需要JDK1.5的版本。

JK本文用的源码包tomcat-connectors

Apache 安装：

# tar zxvf httpd-2.2.8.tar.gz

#cd httpd-2.2.8

# ./configure --prefix=/usr/local/apache2 --enable-modules=so --enable-so

# make && make install

/usr/local/apache2/bin/apachectrl start

看到apache就配置成功了。

JDK的安装

#chmod 755 jdk-1_5_0_16-linux-i586.bin

#./jdk-1_5_0_16-linux-i586.bin

# mv jdk-1_5_0_16 /usr/local/

到此JDK已经安装完成

到/usr/bin目录下，把原用的java,javac文件删除:

#rm -rf java

#rm –rf javac

在/usr/bin 下建立软连接 java

# ln -s /usr/local/jdk-1_5_0_16/bin/java /usr/bin/java

# ln -s /usr/local/jdk-1_5_0_16/bin/javac /usr/bin/javac

# java –version

# javac -version

java version "1.4.2_08"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_08-b03)

Java HotSpot(TM) Client VM (build 1.4.2_08-b03, mixed mode)

为了方便下在的工作建立两个软链接：

ln -s /usr/local/jdk-1_5_0_16 /usr/local/jdk

ln -s /usr/local/jdk-1_5_0_16/jre /usr/local/jre

Tomcat的安装

#tar -zxvf apache-tomcat-5.5.27.tar.gz

# mv apache-tomcat-5.5.27 /usr/local/tomcat

设置环境变量:

#Set Environment by NetSeek

JAVA_HOME=/usr/local/jdk

export JAVA_HOME

JRE_HOME=/usr/local/jre

export JRE_HOME

CLASSPATH=/usr=/usr/local/tomcat/common/lib/:/usr/local/jdk/lib:/usr/local/jre/lib

export CLASSPATH

PATH=$PATH:/usr/local/tomcat/bin/:/usr/local/apache/bin:/usr/local/jdk/bin:/usr/local/jre/bin

export PATH

TOMCAT_HOME=/usr/local/tomcat

export TOMCAT_HOME

启动tomcat以检查是否存在错误：

#cd /usr/local/tomcat/bin

#./startup.sh

http://IP地址:8080 可以看到猫头：）成功了。

在另一台机器上也安装tomcat 并配置好。

JK 安装（整合apache tomcat）

# tar zxvf tomcat-connectors-1.2.27-src.tar.gz

#cd tomcat-connectors-1.2.27-src/native

# ./configure --with-apxs=/usr/local/apache2/bin/apxs --with-java-home=/usr/local/jdk

# make && make install

在/usr/local/apache2/modules/ 下会产生mod_jk.so

修改apache配置文件：

LoadModule jk_module modules/mod_jk.so

JkWorkersFile /usr/local/tomcat/conf/jk/workers.properties JK配置文件负载的配置

JkShmFile logs/mod_jk.shm

JkLogFile /usr/local/tomcat/logs/mod_jk.log 相关日志的设置

JkLogLevel info

JkMount /*.jsp controller 将jsp的请求转发给 controller(负载均衡控制器)

修改DoucmentRoot 与tomcat的目录一致。

本文修改为：/usr/local/tomcat/webapps/

二、调度器JK的配置

建立JK配置文件：

#vi /usr/local/tomcat/conf/jk/workers.properties

workers.tomcat_home=/usr/local/tomcat #指定tomcat的目录――如果2个tomcat在同一台机器该选项去掉

workers.java_home=/usr/local/jdk #指定jdk的目录

ps=/

worker.list=controller #指定负载名，这个可以随便起，与下面的 worker.controller.type=lb要对应

#========tomcat1======== 第一台tomcat的配置

worker.tomcat1.port=8009

worker.tomcat1.host=localhost #这里也可以写IP

worker.tomcat1.type=ajp13

worker.tomcat1.lbfactor=1 #权量数值越大，分配的机率就最小

#========tomcat2===========

worker.tomcat2.port=8009

worker.tomcat2.host=192.168.19.81 (tomcat2的IP)

worker.tomcat2.type=ajp13

worker.tomcat2.lbfactor=1

#========controller,负载均衡器=======

worker.controller.type=lb

worker.controller.balance_workers=tomcat1,tomcat2 # server.xml配置文件里的jvmRoute="tomcat1"指定的名字

worker.controller.sticky_session=1

三、tomcat集群配置，session 同步配置：
1、tomca1 tomcat 2 在同一台机器
Tomcat 1 （IP: 192.168.19.199）
A、修改Engine节点信息： <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
B、去掉<Cluster> <\Cluster> 的注释符
C、修改Cluster 节点信息

<Membership                 
className="org.apache.catalina.cluster.mcast.McastService"   
mcastBindAddress="192.168.19.199"                  
mcastAddr="224.0.0.1"                  
mcastPort="45564"      
mcastFrequency="500"                  
mcastDropTime="3000"/> 
<Receiver             
className="org.apache.catalina.cluster.tcp.ReplicationListener"                  
tcpListenAddress="192.168.19.199"                  
tcpListenPort="4001"                  
tcpSelectorTimeout="100"                  
tcpThreadCount="6"/>

Tomcat 2 （IP: 192.168.19.81）
A、修改Engine节点信息： <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">
B、去掉<Cluster> <\Cluster> 的注释符
C、修改Cluster 节点信息

<Membership         
className="org.apache.catalina.cluster.mcast.McastService" 
mcastBindAddress="192.168.19.199" 
mcastAddr="224.0.0.1" 
mcastPort="45564" 
mcastFrequency="500" 
mcastDropTime="3000"/> 
<Receiver            
className="org.apache.catalina.cluster.tcp.ReplicationListener"            
tcpListenAddress="192.168.19.199" 
tcpListenPort="4002" #一定要改  
tcpSelectorTimeout="100" 
tcpThreadCount="6"/>

D、将8080 8009 8082三个节点的端口改为
9080 9009 9082 避免与tomcat1端口冲突
注：这里的IP也可以不改

2、tomca1 tomcat 2 在不同机器上
Tomcat 1 （IP: 192.168.19.199）
A、修改Engine节点信息： <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
B、去掉<Cluster> <\Cluster> 的注释符
C、修改Cluster 节点信息

 <Membership              
className="org.apache.catalina.cluster.mcast.McastService" 
mcastBindAddress="192.168.19.199"                  
mcastAddr="224.0.0.1"                  
mcastPort="45564"                  
mcastFrequency="500" 
mcastDropTime="3000"/> 
<Receiver             
className="org.apache.catalina.cluster.tcp.ReplicationListener" 
tcpListenAddress="192.168.19.199"                  
tcpListenPort="4001"                  
tcpSelectorTimeout="100"                  
tcpThreadCount="6"/>

Tomcat 2 （IP: 192.168.19.81）
A、修改Engine节点信息： <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
B、去掉<Cluster> <\Cluster> 的注释符
C、修改Cluster 节点信息

<Membership 
className="org.apache.catalina.cluster.mcast.McastService"                  
mcastBindAddress="192.168.19.81"                  
mcastAddr="224.0.0.1"                  
mcastPort="45564"     mcastFrequency="500"                  
mcastDropTime="3000"/> 
<Receiver             
className="org.apache.catalina.cluster.tcp.ReplicationListener"              
 
tcpListenAddress="192.168.19.81"                  
tcpListenPort="4001"                  
tcpSelectorTimeout="100"                  
tcpThreadCount="6"/>

修改web应用里面WEB-INF目录下的web.xml文件，加入标签
<distributable/>
直接加在</web-app>之前就可以了
做tomcat集群必须需要这一步，否则用户的session就无法正常使用。

注意事项：1、mcastAddr="224.0.0.1" 这主广播地址因此需要开启网卡组播功能

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

分别在各机器上运行

2、查看端口情况：

Netstat –antl |grep 4001 同步监听的端口
tomcat1

tomcat2

3、测试广播：

java -cp tomcat-replication.jar MCaster 224.0.0.1 45564 Terminal1

java -cp tomcat-replication.jar MCaster 224.0.0.1 45564 Terminal2

如果不报错则能正常广播

tomcat-replication.jar 下载:http://cvs.apache.org/~fhanik/tomcat-replication.jar

如果是二台机器，可以用tcpdump 抓取包

五、测试集群及session同步

在webapps 下新建test 目录

目录下建WEB-INF目录下的web.xml文件

<web-app xmlns="http://java.sun.com/xml/ns/j2ee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee [url]http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd[/url]" 
    version="2.4"> 
       <display-name>TomcatDemo</display-name> 
       <distributable/> 
</web-app>

再在webapps 下建立print.jsp test.jsp

Print.jsp :

 <%  
   System.out.println("www.linuxtone.org   liuyu.blog.51cto.com");  
%>

test.jsp

<%@ page contentType="text/html; charset=GBK" %> 
<%@ page import="java.util.*" %> 
<html><head><title>Cluster App Test</title></head> 
<body> 
Server Info:  
<%  
out.println(request.getLocalAddr() + " : " + request.getLocalPort()+"<br>");%> 
<%  
  out.println("<br> ID " + session.getId()+"<br>");  
  String dataName = request.getParameter("dataName");  
  session.setAttribute("myname","session?");  
  if (dataName != null && dataName.length() > 0) {  
     String dataValue = request.getParameter("dataValue");  
     session.setAttribute(dataName, dataValue);  
  }  
  out.print("<b>Session P±?b>");  
  Enumeration e = session.getAttributeNames();  
  while (e.hasMoreElements()) {  
     String name = (String)e.nextElement();  
     String value = session.getAttribute(name).toString();  
     out.println( name + " = " + value+"<br>");  
         System.out.println( name + " = " + value);  
   }  
%> 
  <form action="index.jsp" method="POST"> 
    û³?<input type=text size=20 name="dataName"> 
     <br> 
    ?:<input type=text size=20 name="dataValue"> 
     <br> 
    <input type=submit> 
   </form> 
</body> 
</html>

重启所有的服务。

访问http://192.168.19.199/test/print.jsp

如图可以看出tomcat 集群配置完成

Session 复制的查看

在同一窗口，输入名称和值，

在日志可以看到如下结果. 这种结果表明成功了
tomcat1

tomcat 2

错误总汇：

skipping state transfer. No members active in cluster group."

mcastBindAddress 没有添加或添加得不对

?Unknown default host: ajp13

JK 配置不当

其它的不记得了。。。

本文出自 “seven” 博客，请务必保留此出处http://liuyu.blog.51cto.com/183345/113544

High Availability

shanyiwan@live.com() — Fri, 13 Nov 2009 03:25:11 GMT

High Availability

As more and more mission-critical applications move on the Internet, providing highly available services becomes increasingly important. One of the advantages of a clustered system is that it has hardware and software redundancy, because the cluster system consists of a number of independent nodes, and each node runs a copy of operating system and application software. High availability can be achieved by detecting node or daemon failures and reconfiguring the system appropriately, so that the workload can be taken over by the remaining nodes in the cluster.

In fact, high availability is a big field. An advanced highly available system may have a reliable group communication sub-system, membership management, quoram sub-systems, concurrent control sub-system and so on. There must be a lot of work to do. However, we can use some existing software packages to construct highly available LVS cluster systems now.

Working Principle

In general, there are service monitor daemons running on the load balancer to check server health periodically, as illustrated in the figure of LVS high availability. If there is no response for service access request or ICMP ECHO_REQUEST from a server in a specified time, the service monitor will consider the server is dead and remove it from the available server list at the load balancer, thus no new requests will be sent to this dead server. When the service monitor detects the dead server has recovered to work, the service monitor will add the server back to the available server list. Therefore, the load balancer can automatically mask the failure of service daemons or servers. Furthermore, administrators can also use system tools to add new servers to increase the system throughput or remove servers for system maintenance, without bringing down the whole system service.

Now the load balancer might become a single failure point of the whole system. In order to prevent the whole system from being out of service because of the load balancer failure, we need setup a backup (or several backups) of the load balancer. Two heartbeat daemons run on the primary and the backup respectively, they heartbeat the message like "I'm alive" each other through serial lines and/or network interfaces periodically. When the heartbeat daemon of the backup cannot hear the heartbeat message from the primary in the specified time, it will take over the virtual IP address to provide the load-balancing service. When the failed load balancer comes back to work, there are two solutions, one is that it becomes the backup load balancer automatically, the other is the active load balancer releases the VIP address, and the recover one takes over the VIP address and becomes the primary load balancer again.

The primary load balancer has state of connections, i.e. which server the connection is forwarded to. If the backup load balancer takes over without those connections information, the clients have to send their requests again to access service. In order to make load balancer failover transparent to client applications, we implement connection synchronization in IPVS, the primary IPVS load balancer synchronizes connection information to the backup load balancers through UDP multicast. When the backup load balancer takes over after the primary one fails, the backup load balancer will have the state of most connections, so that almost all connections can continue to access the service through the backup load balancer.

The availability of database, network file system or distributed file system is not addressed here.

Working Examples

There are several software packages in conjuction with LVS to provide high availability of the whole system, such as Red Hat Piranha, Keepalived, UltraMonkey, heartbeat plus ldirectord, and heartbeat plus mon.

The following examples of building highly available LVS systems are only for reference.

There must be many other methods to build highly available LVS systems, please drop me a message if you have your methods.

1. The Piranha Solution

Piranha is one of the clustering products from Red Hat Inc., it includes the IPVS kernel code, cluster monitoring tool and web-based cluster configuration tool.

The piranha monitoring tool has two main features:

Heartbeating between active and backup load balancers.
Checking availability of the services on each of real servers.

The piranha monitoring tool uses one process to heartbeat UDP messages between active and backup load balancers, and fork service monitoring processes so that each process is to monitoring one service at a real server.

Configuation example

Now we're going to using piranha to contruct a highly-available VS/NAT web cluster with two load balancers and three web servers. The topology is illustrated in the following figure. In the example, virtual IP address and gateway IP address are 10.23.8.80 and 172.18.1.254, which are floating between the two load balancers (LD1 and LD2), and the ip addresses of three real servers are 172.18.1.11, 172.18.1.12 and 172.18.1.13 respectively.

The configuration file of Piranha is the same at the LD1 and LD2. The configuration for our example looks like:

primary = 10.23.8.1
service = lvs
rsh_command = rsh
backup_active = 1
backup = 10.23.8.2
heartbeat = 1
heartbeat_port = 539
keepalive = 4
deadtime = 12
network = nat
nat_router = 172.18.1.254 eth1:0
nat_nmask = 255.255.255.0
reservation_conflict_action = preempt
debug_level = NONE
virtual web {
     active = 1
     address = 10.23.8.80 eth0:1
     vip_nmask = 255.255.255.255
     port = 80
     persistent = 600
     send = "GET / HTTP/1.0\r\n\r\n"
     expect = "HTTP"
     load_monitor = none
     scheduler = wlc
     protocol = tcp
     timeout = 6
     reentry = 15
     quiesce_server = 0
     server webserver1 {
         address = 172.18.1.11
         active = 1
         weight = 100
     }
     server webserver2 {
         address = 172.18.1.12
         active = 1
         weight = 100
     }
     server webserver3 {
         address = 172.18.1.13
         active = 1
         weight = 100
     }
}

2. The Keepalived Solution

Keepalived provides a strong and robust health checking for LVS clusters. It implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover.

Configuation example

Now we're going to using keepalived to contruct a highly-available VS/NAT web cluster with two load balancers and three web servers. The topology is illustrated in the following figure. In the example, virtual IP address and gateway IP address are 10.23.8.80 and 172.18.1.254, which are floating between the two load balancers (LD1 and LD2), and the ip addresses of three real servers are 172.18.1.11, 172.18.1.12 and 172.18.1.13 respectively.

In our example, the keepalived configuration file (/etc/keepalived/keepalived.conf) at the LD1 looks like:

vrrp_sync_group VG1 {
    group {
        VI_1
        VI_2
    }
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.23.8.80
    }
}
vrrp_instance VI_2 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.18.1.254
    }
}
virtual_server 10.23.8.80 80 {
    delay_loop 6
    lb_algo wlc
    lb_kind NAT
    persistence_timeout 600
    protocol TCP
    real_server 172.18.1.11 80 {
        weight 100
        TCP_CHECK {
            connect_timeout 3
        }
    }
    real_server 172.18.1.12 80 {
        weight 100
        TCP_CHECK {
            connect_timeout 3
        }
    }
    real_server 172.18.1.13 80 {
        weight 100
        TCP_CHECK {
            connect_timeout 3
        }
    }
}

The Keepalived configuration file at LD2 is similar to that of LD1, except to change the state of VI_1 and VI_2 from MASTER to BACKUP.

3. The UltraMonkey Solution

Ultra Monkey is a project to create load balanced and highly available services on a local area network using Open Source components on the Linux operating system, including heartbeat and ldirectord from the Linux-HA project.

Configuation example

Now we're going to using UltraMonkey to contruct a highly-available VS/NAT web cluster with two load balancers and three web servers. The topology is illustrated in the following figure. In the example, virtual IP address and gateway IP address are 10.23.8.80 and 172.18.1.254, which are floating between the two load balancers (LD1 and LD2), and the ip addresses of three real servers are 172.18.1.11, 172.18.1.12 and 172.18.1.13 respectively.

The configuration files of UltraMonkey are the same at LD1 and LD2. The configuration files for above examples are as follows:

/etc/ha.d/ha.cf:

logfacility     local0
keepalive 2
deadtime 10
warntime 10
initdead 10
nice_failback on
udpport 694
bcast   eth1
node    ld1
node    ld2

/etc/ha.d/haresources:

ld1 IPaddr::10.23.8.80/24/eth1 IPaddr::172.18.1.254/24/\
eth1 ldirectord::ldirectord.cf

/etc/ha.d/ldirectord.cf:

checktimeout=10
checkinterval=2
autoreload=no
logfile="local0"
quiescent=yes
virtual=10.23.8.80:80
	fallback=127.0.0.1:80
	real=172.18.1.11:80 masq
	real=172.18.1.12:80 masq
	real=172.18.1.13:80 masq
	service=http
	request="index.html"
	receive="Test Page"
	scheduler=wlc
	persistent=600
	protocol=tcp
	checktype=negotiate

4. The heartbeat+mon+coda solution

The high availability of virtual server can be provided by using of mon, heartbeat and coda software. The "mon" is a general-purpose resource monitoring system, which can be used to monitor network service availability and server nodes. The "heartbeat" code currently provides the heartbeats among two node computers through serial line and UDP heartbeats. Fake is IP take-over software by using of ARP spoofing. The architecture of this solution is illustrated in the following figure.

The server failover is handle as follows: The "mon" daemon is running on the load balancer to monitor service daemons and server nodes in the cluster. The fping.monitor is configured to detect whether the server nodes is alive every t seconds, and the relative service monitor is also configured to detect the service daemons on all the nodes every m seconds. For example, http.monitor can be used to check the http services; ftp.monitor is for the ftp services, and so on. An alert was written to remove/add a rule in the linux virtual server table while detecting the server node or daemon is down/up. Therefore, the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.

Now, the load balancer becomes a single failure point of the whole system. In order to mask the failure of the primary load balancer, we need setup a backup server of the load balancer. The "fake" software is used for the backup to takeover the IP addresses of the load balancer when the load balancer fails, and the "heartbeat" code is used to detect the status of the load balancer to activate/deactivate the "fake" on the backup server. Two heartbeat daemons run on the primary and the backup, they heartbeat the message like "I'm alive" each other through the serial line periodically. When the heartcode daemon of the backup cannot hear the "I'm alive" message from the primary in the defined time, it activates the fake to take over the Virtual IP address to provide the load-balancing service; when it receives the "I'm alive" message from the primary later, it deactivate the fake to release the Virtual IP address, and the primary comes back to work again.

However, the failover or the takeover of the primary load balancer will cause the established connection in the hash table lost in the current implementation, which will require the clients to send their requests again.

Coda is a fault-tolerant distributed file systems, a descendant of Andrew file system. The contents of servers can be stored in Coda, so that files can be highly available and easy to manage.

Configuation example

The following is an example to setup a highly available virtual web server via direct routing.

The failover of real servers

The "mon" is used to monitor service daemons and server nodes in the cluster. For example, the fping.monitor can be used to monitor the server nodes, http.monitor can be used to check the http services, ftp.monitor is for the ftp services, and so on. So, we just need to write an alert to remove/add a rule in the virtual server table while detecting the server node or daemon is down/up. Here is an example calleded lvs.alert, which takes virtual service(IP:Port) and the service port of real servers as parameters.

#!/usr/bin/perl
#
# lvs.alert - Linux Virtual Server alert for mon
#
# It can be activated by mon to remove a real server when the
# service is down, or add the server when the service is up.
#
#
use Getopt::Std;
getopts ("s:g:h:t:l:P:V:R:W:F:u");
$ipvsadm = "/sbin/ipvsadm";
$protocol = $opt_P;
$virtual_service = $opt_V;
$remote = $opt_R;
if ($opt_u) {
    $weight = $opt_W;
    if ($opt_F eq "nat") {
	$forwarding = "-m";
    } elsif ($opt_F eq "tun") {
	$forwarding = "-i";
    } else {
	 $forwarding = "-g";
    }
    if ($protocol eq "tcp") {
	system("$ipvsadm -a -t $virtual_service -r $remote -w $weight $forwarding");
    } else {
	system("$ipvsadm -a -u $virtual_service -r $remote -w $weight $forwarding");
    }
} else {
    if ($protocol eq "tcp") {
	system("$ipvsadm -d -t $virtual_service -r $remote");
    } else {
	system("$ipvsadm -d -u $virtual_service -r $remote");
    }
};

The lvs.alert is put under the /usr/lib/mon/alert.d directory. The mon configuration file (/etc/mon/mon.cf or /etc/mon.cf) can be configured to monitor the http services and servers in the cluster as follows.

#
# The mon.cf file
#
#
# global options
#
cfbasedir   = /etc/mon
alertdir   = /usr/lib/mon/alert.d
mondir     = /usr/lib/mon/mon.d
maxprocs    = 20
histlength = 100
randstart = 30s
#
# group definitions (hostnames or IP addresses)
#
hostgroup www1 www1.domain.com
hostgroup www2 www2.domain.com
#
# Web server 1
#
watch www1
    service http
	interval 10s
	monitor http.monitor
	period wd {Sun-Sat}
	    alert mail.alert wensong
	    upalert mail.alert wensong
	    alert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.1 -W 5 -F dr
	    upalert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.1 -W 5 -F dr
#
# Web server 2
#
watch www2
    service http
	interval 10s
	monitor http.monitor
	period wd {Sun-Sat}
	    alert mail.alert wensong
	    upalert mail.alert wensong
	    alert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.2 -W 5 -F dr
	    upalert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.2 -W 5 -F dr

Note that we need to set the paramter of lvs.alert like "lvs.alert -V 10.0.0.3:80 -R 192.168.0.3:8080" if the destination port is different in LVS/NAT.

Now the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.

The failover of the load balancer

In order to prevent the load balancer becoming a single failure point of the whole system, we need setup a backup of the load balancer and let them heartbeat each other periodically. Please read the GettingStarted document include the heartbeat package, it is simple to setup 2-node heartbeating system.

For an example, we assume that the two load balancers have the following addresses:

lvs1.domain.com (primary)	10.0.0.1
lvs2.domain.com (backup)	10.0.0.2
www.domain.com (VIP)	10.0.0.3

After install the heartbeat on both lvs1.domain.com and lvs2.domain.com, simply create the /etc/ha.d/ha.conf as follows:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#       hopfudge maximum hop count minus number of nodes in config
hopfudge 1
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport 1001
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    lvs1.domain.com
node    lvs2.domain.com

The /etc/ha.d/haresources file is as follows:

lvs1.domain.com 10.0.0.3 lvs mon

The /etc/rc.d/init.d/lvs is as follows:

#!/bin/sh
#
# You probably want to set the path to include
# nothing but local filesystems.
#
PATH=/bin:/usr/bin:/sbin:/usr/sbin
export PATH
IPVSADM=/sbin/ipvsadm
case "$1" in
    start)
	if [ -x $IPVSADM ]
	then
	    echo 1 > /proc/sys/net/ipv4/ip_forward
	    $IPVSADM -A -t 10.0.0.3:80
	    $IPVSADM -a -t 10.0.0.3:80 -r 192.168.0.1 -w 5 -g
	    $IPVSADM -a -t 10.0.0.3:80 -r 192.168.0.2 -w 5 -g
	fi
	;;
    stop)
	if [ -x $IPVSADM ]
	then
	    $IPVSADM -C
	fi
	;;
    *)
	echo "Usage: lvs {start|stop}"
	exit 1
esac
exit 0

Finally, make sure that all the files are created on both the lvs1 and lvs2 nodes and alter them for own own environment, then start the heartbeat daemon on the two nodes.

Note that fake (IP takeover by Gratuitous Arp) is already included in the heartbeat package, so there is no need to setup "fake" separately. When the lvs1.domain.com node fails, the lvs2.domain.com will take over all the haresources of the lvs1.domain.com, i.e. taking over the 10.0.0.3 address by Gratuitous ARP, start the /etc/rc.d/init.d/lvs and /etc/rc.d/init.d/mon scripts. When the lvs1.domain.com come back, the lvs2 releases the HA resources and the lvs1 takes them back.

5. The heartbeat+ldirectord solution

The ldirectord (Linux Director Daemon) written by Jacob Rief is a stand-alone daemon to monitor services of real servers, currently http and https service. It is simple to install and it works with the heartbeat code. The latest version of ldirectord can be found at the CVS repository of heartbeat. See 'perldoc ldirectord' for all the information about ldirectord. Thank Jacob Rief for writing this great program!

The advantages of ldirectord over mon are as follows:

The ldirectord is specially written for LVS monitoring.
It reads configuration files like /etc/ha.d/xxx.cf, which contains all the information about the IPVS routing table configuration. When the ldirectord is up, the IPVS routing table will be configured properly. We can also save different virtual service configuration in multiple configuration files, so that it is possible to modify parameters of some services without bringing down other services.
The ldirectored can be easily started and stopped by heartbeat.
Put the ldirectord under the /etc/ha.d/resource.d/ directory, then we can add a line in the /etc/ha.d/haresources like:
```
    node1 IPaddr::10.0.0.3 ldirectord::www ldirectord::mail
```

Anyway, the ldirectord can also be started and stopped manually. We can also use it in our LVS clusters without the backup of load balancer.

Configuation example

Taking the example introduced in the heartbeat+mon+coda solution, we can configure the /etc/ha.d/www.cf as follows:

#
# The /etc/ha.d/www.cf for ldirectord
#
# the number of second until a real server is declared dead
timeout = 10
# the number of second between server checks
checkinterval = 10
#
# virtual = x.y.z.w:p
#     protocol = tcp|udp
#     scheduler = rr|wrr|lc|wlc
#     real = x.y.z.w:p gate|masq|ipip [weight]
#     ...
#
virtual = 10.0.0.3:80
     protocol = tcp
     scheduler = wlc
     real = 192.168.0.1:80 gate 5
     real = 192.168.0.2:80 gate 5
     request = "/.testpage"
     receive = "test page"

The /etc/ha.d/haresources file is simple as follows:

    lvs1.domain.com IPaddr::10.0.0.3 ldirectord::www

We need to create the .testpage file at the DocumentRoot directory of each web server.

    echo "test page" > .testpage

Start the heartbeat daemons on the primary and the backup. If there is anything wrong, we may check the /var/log/ha-log and /var/log/ldirector.log respectively.

Source from:http://www.linuxvirtualserver.org/HighAvailability.html

“软”负载均衡

shanyiwan@live.com() — Fri, 23 Oct 2009 02:08:14 GMT

需求转而学习
很多时候不少做开发的同学都认为技术更新的快，新技术、新概念层出不穷，大家乐此不疲的去跟随着所谓的“技术趋势”走在风头浪尖上，但其实往往忘记了一个最重要的问题“满足客户需求”。其实技术就是为满足需求服务的，用最小的代价来满足用户的需求，以最简单高效的方式来达到目标，就是每个开发者应该追求的。（不要因为自己的架构很简单就脸红拿不出手，只要你在满足用户当前需求的基础上对未来有所考虑，那么化繁为简就是一种能力的表现）

SIP（服务集成平台）5.7版本中对于未来多个服务提供商，多种类型的服务，在每日几亿的调用压力下，需要找到一个解决方案：可以分流不同服务提供商的服务，分流不同类型的服务，服务隔离化来减少服务相互之间影响以及服务提供商之间的影响。

当前SIP的前端是通过硬件F5作负载均衡，因此是无状态无差别的服务负载，这也使得无法区分不同的服务提供商的服务请求和不同类型的服务请求，导致服务提供商之间的服务会产生相互影响（旺旺即时通信类API在峰值占用了大部分的服务处理资源，淘宝宝贝上传类API占用了大量的带宽）。近期还有更大的两类API将会接入，因此寻找一个服务可分流的方案势在必行。（当然过去也考虑通过三级域名配置在负载均衡上来解决这些问题，但是这样首先对于开发者来说不透明，其次也是一种比较僵化的设计方案，扩展和维护也有一定的难度）

在过去也尝试过Apache等Web容器自己的一些load balance特性，当然效果不是很好，和硬件基本无法比拟，而一些专有的“软”负载均衡方案和开源项目也没有深入的去了解，因此借着这次机会，好好深入的挖一挖“软”负载均衡。

“软”负载均衡
作为互联网应用，随时都需要做好用户量突然增大，访问量突然上升的准备。今年热门的词汇“云”我就不多说了，这里就简单说说服务器的横向扩展。其实和DB，文件系统等一样，当资源成为瓶颈的时候，就需要考虑如何通过扩展或者提升资源能力来满足用户的需求，这就是我们常说的横向扩展和纵向扩展。（对于横向扩展和纵向扩展的优劣大家应该都很清楚了，这里也不做赘述）横向扩展中就会要求使用负载均衡的能力，如何根据资源能力不同以及资源在运行期负荷动态变化将负载合理分配是判断负载均衡优劣的标准。

软件负载均衡一般通过两种方式来实现：基于操作系统的软负载实现和基于第三方应用的软负载实现。LVS就是基于Linux操作系统实现的一种软负载，HA Proxy就是基于第三应用实现的软负载。（后面会详细介绍这两种方式的使用）

最早期也是最原始的软负载均衡：“Round Robin DNS”，通过轮询方式在DNS绑定多个IP的情况下，将用户对于同一个域名的请求分配到后端不同的服务节点。这种方案的优点：配置简单，负载分配效率高。缺点：无法知晓后端服务节点服务情况（是否已经停止服务），无法保证在一个Session中多次请求由一个服务节点服务，每一个节点都要求有一个外网IP。

另一种较为常见的就是基于分发器的Load balance。服务使用者通过向分发器发起请求获得服务，分发器将请求分发给后端实际服务处理的节点，给客户提供服务，最常说的反向代理模式就是典型的分发器Load Balance。这类负载均衡处理可以基于应用级转发，也可以基于IP级别转发，当然基于应用转发效率和损耗比较大，同时分发器本身也会成为瓶颈。

LVS （Linux Virtual Server）
LVS是在Linux操作系统基础上建立虚拟服务器，实现服务节点之间的负载均衡。LVS主要是处理OSI模型中的4层消息包，根据一定的规则将请求直接转发到后端的服务处理节点，有较高转发效率。

Virtual Server是Load Balancer和一组服务器的逻辑组合统称，使用服务者只需要与Virtual Server进行交互就可以获得高效的服务。真实服务器和Load Balancer通过高速ＬＡＮ进行交互。Load Balancer能够将请求分发到不同的服务端，在一个虚拟IP下并行处理多个请求。

Virtual Server三种模式介绍
Virtual Server有三种基于IP级别的负载均衡实现方式：IP address translation（NAT）、Direct routing、IP Tunneling。

NAT(Network address translation)：由于IPV4的某些缺陷和安全原因，某些网段例如（10.0.0.0/255.0.0.0, 172.16.0.0/255.240.0.0 and 192.168.0.0/255.255.0.0）不能被用于互联网，因此常常被用作内部局域网，通过网络地址翻译的方式可以让这些网段的服务器访问互联网或者被互联网访问。网络地址翻译主要作用就是将一组ip地址映射到其他的一组ip地址，当映射比例为1:1的时候通常称作静态映射，而当映射地址为M:N(M>N)的时候（M为被映射地址数量，通常是内部ip），则成为动态映射。而对于Virtual Server的NAT模式来说，就是利用了NAT的特性，将内部的一组服务器通过映射到一个虚拟的IP，然后以一个外网虚拟服务节点的身份对外提供服务。

上图是一个实际的NAT范例，对外的服务IP为202.103.106.5，内部建立了虚拟IP为172.16.0.1，然后将内部其他两台实际服务的服务器172.16.0.2，172.16.0.3映射到172.16.0.1这个虚拟IP。客户端向202.103.106.5发起请求服务，Load Balancer查看请求数据包，如果是请求目标地址是注册的虚拟IP及监听端口的时候，那么通过NAT按照一定算法选择某一台实体服务器，再重写报文目标地址，转发请求到实际的目标服务器，当目标服务器处理完毕以后，将处理结果返回给Load Balancer，由Load Balancer修改源地址，返回给客户端。

IP Tunneling：IP管道技术是在IP报文上再次封装IP报文协议的一种技术。允许将一个目标为A的IP数据报文封装成为目标为B的IP数据报文，在特定的IP 管道中传输。

上图就是IP Tunneling模式的运作原理。首先客户端还是通过访问对外的一个服务IP请求服务，当Load Balancer接受到请求以后，检查VIP注册信息，然后根据算法选择实际的一台后台服务器，通过IP管道封装技术对IP报文再次封装，然后将消息通过IP管道转发到实际的服务器，实际的服务器通过解包处理请求，然后根据包体内实际的服务请求地址，将处理结果直接返回给客户端。

Direct routing：利用Load Balancer和实际服务器共享同一VIP，简单的通过修改消息报体目标MAC地址，转发请求，然后再通过实际服务器配置VIP为本地回环，直接处理消息报文，而不再转发，当处理完以后，直接将处理结果返回给客户端。

上图就是Direct Routing的运作流程，当外部请求到Load Balancer时，通过查找VIP注册信息，直接选择一台后端服务器作为新的目标地址，修改消息报文中的目标地址Mac地址，转发到目标服务器，目标服务器由于配置VIP在本地网卡回路中，因此直接处理消息，将处理完的结果直接返回给客户端。

Virtual Server三种模式的比较
下表是官方整理出的关于Virtual Server三种不同模式的区别：

	NAT	TUNNEL	DR
服务器要求	无要求	需要支持IP管道	无 arp组件（当前也有补丁）
网络要求	Private	LAN/WAN	LAN
可支持后端服务器节点数	较少（10-20）	较多	较多
服务网关	Load Balancer	本身	本身

NAT：根据其实现原理，可以知道这种模式对于操作系统，网络都没有太多的要求和约束，但是由于消息需要打解包，同时消息的响应都必须经过Load Balancer，因此Load Balancer自身成为了瓶颈，这样一个Load Balancer能够支持的后端服务节点数量就有限了。当然可以采用混合模式来解决这个问题，也就是通过TUNNEL或者DR模式作为前端模式串联起多个NAT模式Balancer。

TUNNEL：这种模式要求操作系统支持IP Tunnel，通过对IP报文再次封装转发，达到负载均衡的目的。设计这种模式的初衷是考虑，对于互联网很多服务来说，服务请求数据量和返回数据量是不对称的，返回的数据往往要远远大于请求的数据量，因此如果请求和返回都走Load Balancer会大量占用带宽，影响处理能力。IP Tunnel设计中请求是通过Load Balancer，但是返回是直接返回到客户端的，因此节省了返回的带宽，提高了请求处理的能力。

DR：这种模式要求Load Balancer和后端服务器处于同一个局域网段。DR模式处理消耗最小，消息转发和回复基本没有损耗，因此效率应该是最高的，但是约束是相对来说最多的。

Virtual Server三种模式实践
服务器资源：

10.2.225.128作为Load Balancer，10.2.225.136和10.2.225.139作为后端的服务器。

在10.2.225.136和10.2.225.139上分别安装Apache2.2.14，然后打开80端口，启动Web服务。

1．安装LVS配置工具(只需要在Load Balancer安装即可)

Linux操作系统当前大部分都已经支持Virtual Server，因此不需要再去编译内核，只需要下载和安装用户配置工具ipvsadmin。首先请查看自己Linux操作系统内核的版本，我的版本是2.6.18。从http://www.linux-vs.org/software/ipvs.html#kernel-2.6 下载了ipvsadm-1.24.tar.gz。安装步骤如下：

# tar –xzvf ipvsadm-1.24.tar.gz

# ln –s /usr/src/kernels/2.6.18-128.el5-i686 /usr/src/linux (这部必须作，因为在编译和安装的时候需要用到目录/usr/src/linux来获得必要的外部依赖库)

# cd ipvsadm-1.24

# make && make install

输入ipvsadm，如果看到以下内容证明安装成功了：

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

2． DR模式测试

a. 为10.2.225.128这台Load Balancer的网卡绑定虚拟IP。

ifconfig eth0:0 10.2.225.211 netmask 255.255.255.0 broadcast 10.2.225.211 up

b. 在Load Balancer上建立虚拟IP。

Ipvsadm –A –t 10.2.225.211:80 –s rr //221 IP没有实体机器在使用，因此作为虚拟IP使用，A表示建立虚拟IP，t表示是tcp协议，s表示负载算法，rr是Round Robin。

c. 在Load Balancer上建立虚拟IP与两个实体服务器的关系。

ipvsadm -a -t 10.2.225.211:80 -r 10.2.225.136:80 -g -w 1 //将虚拟IP和端口关联到实体机器上，a表示新建虚拟IP和实体机器的关联关系，r后面跟实体机器地址和端口，g表示使用DR模式，w是权重。

ipvsadm -a -t 10.2.225.211:80 -r 10.2.225.139 :80 -g -w 1

d. 在另外两台实体服务器上建立虚拟IP与网卡回环的绑定关系。

ifconfig lo:0 10.2.225.211 netmask 255.255.255.255 broadcast 10.2.225.211 up

下图就是配置好以后再输入ipvsadm查看到的虚拟配置信息

其中Route表示就是DR的模式，然后图中ActiveConn表示连接数，这里是访问一个发布在两台后端服务器的页面带来的连接数，访问页面的情况如下两张图：（主要是看返回页面中服务器ip打印的不同）

访问页面的结果和DR是一样的，对虚拟IP的访问，会轮询到不同的后端服务器上。

三种模式下的简单压力测试
简单的压力测试采用Apache ab，500并发用户，10w的请求总数。

	总耗时(s)	TPS(#/sec)
NAT	22.480	4448.34
TUNNEL	10.707	9339.80
DR	10.177	9825.68

可以看出NAT性能要比后两种差一倍。

LVS是从四层对消息进行负载转发的，很大程度上是直接利用操作系统的特性来实现消息转发，因此转发损耗有限，但是NAT模式中Load Balancer对于性能的影响很大，同时对于负载压力本身以及后台节点数目有限制。在LVS基础上有很多其他的开源软件，增加了心跳等机制来保证集群的健壮性。而LVS并不能满足根据应用参数或包体内容转发请求的场景，因此进一步研究HA-Proxy，这种基于基于应用级别在7层对消息作分析转发的负载均衡方案。

HA-Proxy
HA-Proxy相比LVS的使用要简单很多，功能方面也很丰富。HA-Proxy可以在4，7两层作负载均衡，4层大多用于邮件服务器、内部协议通信服务器等作负载均衡，7层用于Http分析负载转发。

在HA-Proxy官方网站可以下载配置说明文档(configuration.txt)和架构文件(architecture.txt)作为参考。具体的使用细节不做太多介绍，这里主要通过具体的配置来大致说一下HA-Proxy的结构。

HA-Proxy 组件图

HA-Proxy配置中分成四部分内容，当然这些组件不是必选的，可以根据需要选择部分作为配置。Defaults组件是配置默认参数的，这些参数可以被利用配置到frontend，backend，listen组件中（当这些组件某些参数没有被配置而在Defaults中配置了）。Frontend组件是接收请求的前端虚拟节点，就类似于LVS中配置了VIP的Load Balancer，Frontend可以直接指定后端指向那一个backend(可动态选择)。Backend是后端服务集群的配置，类似于LVS中的那些Real Server，一个Backend对应一个或者多个实体服务器。Listen是Frontend和Backend的组合体，可以直接定义一个类似于JBoss的 Server Farm。还有一些默认的配置可以通过在配置文件中配置或者在命令行中作为参数输入。

HA-Proxy安装和使用
安装HA-Proxy：

1．下载HA-Proxy安装包。

2．解压执行make TARGET=linux26（注意，TARGET后面根据本机操作系统内核版本来填写）

3． make install

4．目录下执行haproxy，如果有使用说明出现表示已经安装正常。

5．使用方式haproxy –f 配置文件地址。（例如 haproxy –f haproxy.cfg）

HA-Proxy日志配置说明：

HA-Proxy可以收集本机及其他后端服务器日志，但是需要在Load Balancer上作一些配置。

首先修改/etc/sysconfig/syslog文件，将SYSLOGD_OPTIONS="-m 0” 修改为SYSLOGD_OPTIONS="-m 0 -r -x"，支持收集远程服务器日志。

然后修改/etc/syslog.conf，增加如下语句：

#add by haproxy

local0.* /home/admin/tools/haproxy-1.3.17/haproxy.log // haproxy.log地址代表了需要存储日志的地址

执行service syslog restart，重新启动系统日志器

最后就是在HA-Proxy的配置中增加日志输出（具体可以参考后面的配置文件说明）

HA-Proxy配置文件说明：

下面的配置文件简单来说就是配置了根据请求参数的不同，将请求分别定向到后端的淘宝集群和阿里软件集群。具体配置文件(haproxy.cfg)如下：

log 127.0.0.1  local0 info  //日志输出配置，所有的日志都记录在本机，通过local0的系统日志器输出，这关系到前面我们做的配置  
    daemon  //以后台进程方式启动Ha-proxy  
    nbproc 2 //启动两个ha-proxy进程实例  
    pidfile /home/admin/tools/haproxy-1.3.17/haproxy.pid    // pid记录的文件  
 
defaults  //默认配置  
    mode http //默认采用http模式，可以配置tcp来做4层消息转发  
    option httplog //采用http日志格式  
    retries 3 //三次连接失败就认为是服务器不可用，主要是通过后面的check配置来实现服务器状态检查  
    maxconn 2000 //最大连接数  
    contimeout 5000 //连接超时时间  
    clitimeout 50000 //客户端连接超时时间  
    srvtimeout 50000 //服务端连接超时时间  
    stats uri /admin?stats //服务器状态统计查看页面  
    stats auth wenchu:wenchu  //服务器状态查看授权的用户名和密码设置，可以不设置  
    option httpchk HEAD /welcome.html HTTP/1.0  //服务器状态检查设置，这里是向每一个后端服务器请求/welcome.html页面来检查服务端健康状况。  
 
frontend http-in  //前端节点定义  
    bind :8181 //虚拟服务节点监听本地的8181端口  
    mode http   
    log  global  
    option httplog  
    option httpclose //每次请求完毕后主动关闭http通道，HA-Proxy不支持keep-alive模式，只能够模拟这种模式的实现  
    option forwardfor   //如果后端服务器需要获得客户端的真实IP需要配置次参数，将可以从Http Header中获得客户端IP  
    capture request header Host len 20 //此配置和一下的类似配置都是抓取Http请求中的参数记录到日志中。  
     capture request header User-Agent len 16  
    capture request  header Content-Length len 10        
    capture request  header Referer        len 20    
    capture response header Content-Length len 10  
 
    //控制策略的配置  
    acl api_taobao url_sub -i sip_apiname=taobao.  //在请求url中包含sip_apiname=taobao，则此控制策略返回true,否则为false  
    acl api_alisoft url_sub -i sip_apiname=alisoft. //在请求url中包含sip_apiname=alisoft，则此控制策略返回true,否则为false  
    acl invalid_req url_sub -i sip_apiname= //在请求url中包含sip_apiname=，则此控制策略返回true,否则为false  
    acl stat_req url_dir -i admin //在请求url中存在admin作为部分地址路径，则此控制策略返回true,否则返回false  
    block if !invalid_req !stat_req  //block表示阻止请求，返回403错误，当前表示如果不满足策略invalid_req，同时也不满足策略stat_req，则阻止请求。（就是要求URL中必需有参数sip_apiname，除非是查看服务器状态的URL）。  
    use_backend alisoft_server if api_alisoft  //如果是满足策略api_alisoft的情况，则使用alisoft_server作为后端服务集群。  
    use_backend taobao_server if api_taobao //如果是满足策略api_taobao的情况，则使用taobao_server作为后端服务集群。  
    default_backend alisoft_server //使用alisoft_server作为默认后端服务集群。  
 
   
backend alisoft_server //后端节点定义  
   mode http   
   balance roundrobin  //负载均衡策略配置  
   cookie SERVERID  //允许插入serverid到cookie中，serverid后面可以定义  
   server app1 10.2.225.139:80 cookie 1 check fall 5 weight 1  //真实服务器配置定义cookie 1表示serverid为1，check表示需要状态检查，fall 5表示失败五次就认为服务器状态不可用（不在接受请求），weight 1表示权重  
   server app2 10.2.225.136:80 cookie 2 check fall 5 weight 2  
 
backend taobao_server //后端节点定义  
   mode http  
   server app3 10.2.226.41:80 check fall 5

完成配置后，执行haproxy –f haproxy.cfg，后台进程就可以启动了，然后在浏览器中输入刚才定义的状态检查地址可以看到如下内容：

可以看到定义的前端和后端节点的状态。对于Ha-proxy很多配置这里面都没有使用，也没有详细讲解，使用者可以通过查看官方的配置文档了解细节。下面三个图片分别说明了对于sip_apiname不同的访问产生最后的结果。

上图的sip_apiname为alisoft.get.user，因此被定向到Alisoft集群，也就是136或者139上（这里是136处理了服务）。

上图的sip_apiname为taobao.get.user，因此被定向到Alisoft集群，也就是41上。

上图的sip_apiname没有传递，因此被拒绝访问，返回403错误。

HA-Proxy的压力测试结果
简单的压力测试采用Apache ab，500并发用户，10w的请求总数。

	总耗时(s)	TPS(#/sec)
LVS-NAT	22.480	4448.34
LVS-TUNNEL	10.707	9339.80
LVS-DR	10.177	9825.68
HA-2Node	21.387	4675.61
HA-5Node	27.371	3653.37

HA-2Node为配置了两个节点作为后段的服务节点，HA-5Node为配置了5个节点作为后端的服务处理节点。上面结果看到2个节点的HA反而比5个节点的速度来的快，同时HA在7层的转发和LVS-NAT性能相近。

HA-Proxy使用下来，总体上感觉比较简单，但功能却十分强大，但是性能方面来说需要关注在多节点和高压力的情况下的表现。

“软”负载均衡学习心得
从LVS三种模式中也看到了类似于分布式文件系统的一些设计经验，就是避免在管理资源过程中，让Manager成为了系统瓶颈。就好比LVS-NAT中的Load Balancer既负责请求分配同时也负责消息回复，成为了系统的关键节点，自身性能损耗比较大，加上算法对于数据采集的要求，自身稳定性和可用性下降，最后影响了整个架构。在HDFS中，Master的责任就和明晰，就是负责节点管理，不参与数据传输和通道建立，因此就可以很大程度上提升自身的效率。资源管理（申请，归还，状态检查等）和资源使用应该清晰的划分开来，这样可以让各个角色可以更好的独立的满足需求，防止由于其他功能影响到了“本职工作”。

就负载均衡效率来说，硬件实现负载均衡应该优于用软件实现负载均衡，就好比SSL硬件加速器要远优于SSL软件解析模块。但从另一个角度来看，分布式计算，分布式存储，分布式DB都采用横向扩展结合低成本资源的方式满足需求。而软件实现负载在很多情况下可以尽可能的降低成本，同时在性能损失较小的情况下实现硬件负载所支持的所有功能。因此在一定的环境下，部分采用软件来实现负载均衡能够增加可扩展性，提升配置灵活度，降低配置成本。

从LVS到HA-Proxy，可以发现不论从4层做转发还是7层做转发都会存在损失，而且LVS-NAT模式和HA-Proxy都会受到解析负载度和内容大小的影响。因此完全采用软件负载或者采用某一种配置的软件负载都不可行，通过将硬件负载和软件负载相结合，或者多种软件负载混合使用，可以更好的发挥软件负载灵活的优势，同时也不会因为转发损失影响性能。

Keepalived中文权威指南

shanyiwan@live.com() — Thu, 22 Oct 2009 08:28:26 GMT

中文权威指南

根据自己的研究和使用经验写的，可自由分发，但希望给我反馈，使这份文档更完善起来，目前的内容可能还不是非常完善。Keepalived权威指南.pdf

官方文档地址:

http://www.keepalived.org/documentation.html

本地下载：点击下载Keepalived权威指南

LVS NAT + Keepalived HOWTO

shanyiwan@live.com() — Thu, 22 Oct 2009 07:58:18 GMT

LVS NAT + Keepalived HOWTO

By Adam Fletcher (C) 2002, released under GPL

Install, testing and running of a Keepalived HA based LVS/NAT

1. keepalived - what is it?

From Alexandre Cassen, author of keepalived:

"The main goal of the keepalived project is to add a strong & robust keepalive facility to the Linux Virtual Server project. This project is written in C with multilayer TCP/IP stack checks. Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5. This framework gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the linux kernel via a setsockopt call to remove this server entrie from the LVS topology. In addition keepalived implements a VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover.

"keepalived is a project started to create a full-featured virtual router for Linux, which includes load balancing through Linux Virtual Server, failover via VRRP and health checks to monitor real servers. Essentially, it is a single package for doing what is typically done in Linux via lvs+mon+fake+hearbeat. With keepalived an administrator can quickly build a redundant load balancing solution without the hassle of using numerous packages and custom scripts.

keepalived will:

setup the IP addresses (i.e., no need to create the IP's with ifconfig, etc)
handle failover between load balancers (i.e., no need to run heartbeat to watch for failures, or write scripts to handle the failure)
do health checks on services, bringing them in and out of pools (no need to run mon+custom scripts)
send smtp notifications on transition from master to backup send smtp notifications on health check failures
do LVS NAT (no need to run lvs-configure, or do it by hand with ipvsadm)

keepalived is available from www.keepalived.org

Software used in this example:

Linux kernel 2.4.19 (from kernel.org)
LVS kernel patch, 1.0.6 for Linux kernel 2.4.19 (From www.linuxvirtualserver.org)
keepalived 0.7.1 (from www.keepalived.org)

Optional software:

ipvsadm 1.21 (from www.linuxvirtualserver.org)
OpenSSL (latest from openssl.org, I would recommend at least 0.9.6e or above due to security problems in any earlier version

2. Plan your network!

Draw out a logical diagram of your network, either by hand or with a tool like xfig or Visio. Planning your network saves hassle and time later! Make a list of the IP addresses you are going to use, any external router IPs you may need, the IP addresses of the machine you are going to load balancer and other related information.

3. Configuring your kernel

Configuring your kernel for LVS should be done according to the directions on www.linuxvirtualserver.org. Be sure you enable full NAT, and IP Forwarding.
After patching your kernel for the latest LVS and installing the new kernel and rebooting, you should turn on IP forwarding. Many Linux distributions allow you to do this through the system configuration editor (YaST2 on SuSE, linuxconf on Red Hat, for example), or you can do this in your keepalived startup scripts (we'll give some examples of this later). For now, just

echo "1" > /proc/sys/net/ipv4/ip_forwarding

as root.

4. Building ipvsadm (optional)

ipvsadm is a tool available from www.linuxvirtualserver.org that allows you to setup virtual servers by hand. It is also a useful debugging/status tool, so I recommend building this small tool.

5. Building keepalived

The quick version:

example-01:~ # tar xzvf keepalived-0.7.1.tar.gz; cd keepalived-0.7.1; ./configure; make; make install

Keepalived is very simple to build - grab the latest package from www.keepalived.org, untar, the run ./configure, then make, and make install.
For more information, read the INSTALL file shipped with keepalived.

If you have any trouble, or keepalived says it is not installing support for something you expected (such as SSL health checks), be sure to verify that you have the missing library or header file in the location keepalived expects it to be - for instance, the location of LVS's header file has changed in recent releases, so keepalived may not find the header in older versions of LVS.

6. Setting up keepalived: a simple network: 1 load balancer/virtual router, 1 real server on port 22 (ssh).

Now that we have keepalived built and installed, let's set up this network:

Client (on the internet somewhere) --> load balancer --> realserver

Load balancer IPs:

IP of load balancer's external interface(eth0): 192.168.1.9
external VIP of our realserver: 192.168.1.11
IP of load balancer's interface(eth1): 10.20.40.2
internal VIP our realserver will use as a default gateway: 10.20.40.1

Realserver:

IP: 10.20.40.10
be sure to set the default gateway to 10.20.40.1

Our first step is to configure keepalived. The typical location for
this file is /etc/keepalived/keepalived.conf

Note that keepalived, as of this writing, does not report errors in the
configuration file! This means if something is not right in the config file it may be difficult to notice. Try starting keepalived with the -d option, which will dump a config to syslog.

-- cut here --
! This is a comment
! Configuration File for keepalived

global_defs {
   ! this is who emails will go to on alerts
   notification_email {
        admins@example.com
    fakepager@example.com
    ! add a few more email addresses here if you would like
   }
   notification_email_from admins@example.com

   ! I use the local machine to relay mail
   smtp_server 127.0.0.1
   smtp_connect_timeout 30

   ! each load balancer should have a different ID
   ! this will be used in SMTP alerts, so you should make
   ! each router easily identifiable
   lvs_id LVS_EXAMPLE_01
}

! vrrp_sync_groups make sure that several router instances
! stay together on a failure - a good example of this is
! that the external interface on one router fails and the backup server
! takes over, you want the internal interface on the failed server
! to failover as well, otherwise nothing will work.
! you can have as many vrrp_sync_group blocks as you want.
vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}

! each interface needs at least one vrrp_instance
! each vrrp_instance is a group of VIPs that are logically grouped
! together
! you can have as many vrrp_instaces as you want

vrrp_instance VI_1 {
        state MASTER
        interface eth0

        lvs_sync_daemon_inteface eth0

    ! each virtual router id must be unique per instance name!
        virtual_router_id 51

    ! MASTER and BACKUP state are determined by the priority
    ! even if you specify MASTER as the state, the state will
    ! be voted on by priority (so if your state is MASTER but your
    ! priority is lower than the router with BACKUP, you will lose
    ! the MASTER state)
    ! I make it a habit to set priorities at least 50 points apart
    ! note that a lower number is lesser priority - lower gets less vote
        priority 150

    ! how often should we vote, in seconds?
        advert_int 1

    ! send an alert when this instance changes state from MASTER to BACKUP
        smtp_alert

    ! this authentication is for syncing between failover servers
    ! keepalived supports PASS, which is simple password
    ! authentication
    ! or AH, which is the IPSec authentication header.
    ! I don't use AH
    ! yet as many people have reported problems with it
        authentication {
                auth_type PASS
                auth_pass example
        }

    ! these are the IP addresses that keepalived will setup on this
    ! machine. Later in the config we will specify which real
        ! servers are behind these IPs
    ! without this block, keepalived will not setup and takedown the
    ! any IP addresses

        virtual_ipaddress {
                192.168.1.11
        ! and more if you want them
        }
}

! now I setup the instance that the real servers will use as a default
! gateway
! most of the config is the same as above, but on a different interface

vrrp_instance VI_GATEWAY {
        state MASTER
        interface eth1
        lvs_sync_daemon_inteface eth1
        virtual_router_id 52
        priority 150
        advert_int 1
        smtp_alert
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                10.20.40.1
        }
}

! now we setup more information about are virtual server
! we are just setting up one for now, listening on port 22 for ssh
! requests.

! notice we do not setup a virtual_server block for the 10.20.40.1
! address in the VI_GATEWAY instance. That's because we are doing NAT
! on that IP, and nothing else.

virtual_server 192.168.1.11 22 {
    delay_loop 6

    ! use round-robin as a load balancing algorithm
    lb_algo rr

    ! we are doing NAT
    lb_kind NAT
    nat_mask 255.255.255.0

    protocol TCP

    ! there can be as many real_server blocks as you need

    real_server 10.20.40.10 22 {

    ! if we used weighted round-robin or a similar lb algo,
    ! we include the weight of this server

        weight 1

    ! here is a health checker for this server.
    ! we could use a custom script here (see the keepalived docs)
    ! but we will just make sure we can do a vanilla tcp connect()
    ! on port 22
    ! if it fails, we will pull this realserver out of the pool
    ! and send email about the removal
        TCP_CHECK {
                connect_timeout 3
        connect_port 22
        }
    }
}

! that's all

-- cut here --

When you start keepalived with the -d flag, you should see this in /var/log/message (or equivalent):

Sep 12 14:13:11 example-01 Keepalived: ------< Global definitions >------
Sep 12 14:13:11 example-01 Keepalived: LVS ID = LVS_EXAMPLE_01
Sep 12 14:13:11 example-01 Keepalived: Smtp server = 127.0.0.1
Sep 12 14:13:11 example-01 Keepalived: Smtp server connection timeout = 100
Sep 12 14:13:11 example-01 Keepalived: Email notification from = admins@example.com, fakepager@example.com
Sep 12 14:13:11 example-01 Keepalived: Email notification = admins@example.com
Sep 12 14:13:11 example-01 Keepalived: ------< SSL definitions >------
Sep 12 14:13:11 example-01 Keepalived: Using autogen SSL context
Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Topology >------
Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_1
Sep 12 14:13:11 example-01 Keepalived:    Want State = MASTER
Sep 12 14:13:11 example-01 Keepalived:    Runing on device = eth0
Sep 12 14:13:11 example-01 Keepalived:    Virtual Router ID = 51
Sep 12 14:13:11 example-01 Keepalived:    Priority = 150
Sep 12 14:13:11 example-01 Keepalived:    Advert interval = 1sec
Sep 12 14:13:11 example-01 Keepalived:    Preempt Active
Sep 12 14:13:11 example-01 Keepalived:    Authentication type = SIMPLE_PASSWORD
Sep 12 14:13:11 example-01 Keepalived:    Password = example
Sep 12 14:13:11 example-01 Keepalived:    VIP count = 1
Sep 12 14:13:11 example-01 Keepalived:      VIP1 = 192.168.1.11/32
Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_GATEWAY
Sep 12 14:13:11 example-01 Keepalived:    Want State = MASTER
Sep 12 14:13:11 example-01 Keepalived:    Runing on device = eth1
Sep 12 14:13:11 example-01 Keepalived:    Virtual Router ID = 52
Sep 12 14:13:11 example-01 Keepalived:    Priority = 150
Sep 12 14:13:11 example-01 Keepalived:    Advert interval = 1sec
Sep 12 14:13:11 example-01 Keepalived:    Preempt Active
Sep 12 14:13:11 example-01 Keepalived:    Authentication type = SIMPLE_PASSWORD
Sep 12 14:13:11 example-01 Keepalived:    Password = example
Sep 12 14:13:11 example-01 Keepalived:    VIP count = 1
Sep 12 14:13:11 example-01 Keepalived:      VIP1 = 10.20.40.1/32
Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Sync groups >------
Sep 12 14:13:11 example-01 Keepalived: VRRP Sync Group = VG1, MASTER
Sep 12 14:13:11 example-01 Keepalived:    monitor = VI_1
Sep 12 14:13:11 example-01 Keepalived:    monitor = VI_GATEWAY
Sep 12 14:13:11 example-01 Keepalived: ------< LVS Topology >------
Sep 12 14:13:11 example-01 Keepalived: System is compiled with LVS v1.0.4
Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.11, VPORT = 22
Sep 12 14:13:11 example-01 Keepalived:    delay_loop = 10, lb_algo = rr
Sep 12 14:13:11 example-01 Keepalived:    protocol = TCP
Sep 12 14:13:11 example-01 Keepalived:    lb_kind = NAT
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.11, RPORT = 22, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived: ------< Health checkers >------
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.11:22
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = TCP_CHECK
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 22

Let's see what ipvsadm has to say about this, after keepalived starts up:

example-01:~ # ipvsadm
IP Virtual Server version 1.0.4 (size=65536)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP 192.168.1.11:ssh rr
-> 10.20.40.10:ssh            Masq    1      0          0
example-01:~ #

And finally, we should see the new IP addresses in our IP address list:

example-01:~ # ip addr list
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:e0:81:21:bb:1c brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.9/24 brd 192.168.1.254 scope global eth0
    inet 192.168.1.11/32 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:e0:81:21:bb:1d brd ff:ff:ff:ff:ff:ff
    inet 10.20.40.2/24 brd 10.20.40.255 scope global eth1
    inet 10.20.40.1/32 scope global eth1
example-01:~ #

ipvsadm, ip addr list, and starting keepalived with the -d option are good ways to verify your config is working.

7. Failover

With our basic config from above, we can easily move to a failover situation. All you have to do is setup keepalived on another box, copy over the keepalived.conf, change the lvs_id, change any priorities down 50 points, states to BACKUP, and run keepalived. You'll see in the logs on the backup server that the server accepts it's BACKUP state, and if you unplug the network cable(s) from the MASTER server, the BACKUP server takes over the MASTER state.

For the example, use the config file from the simple example above on the MASTER machine. On the BACKUP machine, use this config file:

-- cut here --
! This is a comment
! Configuration File for keepalived

global_defs {
   ! this is who emails will go to on alerts
   notification_email {
        admins@example.com
    fakepager@example.com
    ! add a few more email addresses here if you would like
   }
   notification_email_from admins@example.com

   ! I use the local machine to relay mail
   smtp_server 127.0.0.1
   smtp_connect_timeout 30

   ! each load balancer should have a different ID
   ! this will be used in SMTP alerts, so you should make
   ! each router easily identifiable

   ! this is router 2
   lvs_id LVS_EXAMPLE_02
}

! vrrp_sync_groups make sure that several router instances
! stay together on a failure - a good example of this is
! that the external interface on one router fails and the backup server
! takes over, you want the internal interface on the failed server
! to failover as well, otherwise nothing will work.
! you can have as many vrrp_sync_group blocks as you want.
vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}

! each interface needs at least one vrrp_instance
! each vrrp_instance is a group of VIPs that are logically grouped
! together
! you can have as many vrrp_instaces as you want

vrrp_instance VI_1 {
        ! we are the failover
        state BACKUP
        interface eth0

        lvs_sync_daemon_inteface eth0

    ! each virtual router id must be unique per instance name!
        ! instance names are the same on MASTER and BACKUP, so the
        ! virtual router_id is the same as VI_1 on the MASTER
        virtual_router_id 51

    ! MASTER and BACKUP state are determined by the priority
    ! even if you specify MASTER as the state, the state will
    ! be voted on by priority (so if your state is MASTER but your
    ! priority is lower than the router with BACKUP, you will lose
    ! the MASTER state)
    ! I make it a habit to set priorities at least 50 points apart
    ! note that a lower number is lesser priority -
    ! lower gets less vote
        priority 100

    ! how often should we vote, in seconds?
        advert_int 1

    ! send an alert when this instance changes state from
    ! MASTER to BACKUP
        smtp_alert

    ! this authentication is for syncing between failover servers
    ! keepalived supports PASS, which is simple
    ! password authentication
    ! or AH, which is the ipsec authentication header.
    ! I don't use AH
    ! yet as many people have reported problems with it
        authentication {
                auth_type PASS
                auth_pass example
        }


        virtual_ipaddress {
                192.168.1.11
        ! and more if you want them
        }
}

! now I setup the instance that the real servers will use as a default
! gateway
! most of the config is the same as above, but on a different interface

vrrp_instance VI_GATEWAY {
        state BACKUP
        interface eth1
        lvs_sync_daemon_inteface eth1
        virtual_router_id 52
        priority 100
        advert_int 1
        smtp_alert
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                10.20.40.1
        }
}

! now we setup more information about are virtual server
! we are just setting up one for now, listening on port 22 for ssh
! requests.

! notice we do not setup a virtual_server block for the 10.20.40.1
! address in the VI_GATEWAY instance. That's because we are doing NAT
! on that IP, and nothing else.

virtual_server 192.168.1.11 22 {
    delay_loop 6

    ! use round-robin as a load balancing alogorithm
    lb_algo rr

    ! we are doing NAT
    lb_kind NAT
    nat_mask 255.255.255.0

    protocol TCP

    ! there can be as many real_server blocks as you need

    real_server 10.20.40.10 22 {

    ! if we used weighted round-robin or a similar lb algo,
    ! we include the weight of this server

        weight 1

    ! here is a health checker for this server.
    ! we could use a custom script here (see the keepalived docs)
    ! but we will just make sure we can do a vanilla tcp connect()
    ! on port 22
    ! if it fails, we will pull this realserver out of the pool
    ! and send email about the removal
        TCP_CHECK {
                connect_timeout 3
        connect_port 22
        }
    }
}

! that's all

-- cut here --

Notice how little is different between the MASTER and BACKUP config file - just the lvs_id directive, the priorities, and the state directive. That's it, that's all. Make sure these are different but everything else is the same.

Once you startup keepalived on the MASTER and the BACKUP, you should be able to kill keepalived on the MASTER server and watch the BACKUP take over in the logs on the BACKUP server.

If you did an ip addr list on the backup server, you won't see the VIPs until the backup server takes over the MASTER state.

8. Setting up keepalived: a more complicated network: 2 VIPs (1 http/https, 1 ssh) with 2 real servers in each.

Load balancer IPs:

IP of load balancers's external interface(eth0): 192.168.1.9
external VIP of our http/https realservers: 192.168.1.11
external VIP of our ssh realservers: 192.168.1.12
IP of load balancer's interface(eth1): 10.20.40.2
internal VIP our realserver will use as a default gateway: 10.20.40.1

Realserver 1 (http, https):

IP: 10.20.40.10
be sure to set the default gateway to 10.20.40.1

Realserver 2 (http, https):

IP: 10.20.40.11
be sure to set the default gateway to 10.20.40.1

Realserver 3 (ssh):

IP: 10.20.40.20
be sure to set the default gateway to 10.20.40.1

Realserver 4 (ssh):

IP: 10.20.40.21
be sure to set the default gateway to 10.20.40.1

A few oddities occur with this setup. In particular, you'll want to learn to use the "genhash" command that comes with keepalived to generate MD5 sums for the HTTP_GET and the SSL_GET service checks. Also, you'll want to setup persistence on the https - persistence will allow your clients to always connect to the same realserver, in case you have something like a shopping cart that's state is maintained on the realserver.

genhash is simple to use. Let's say we have a test.html on our web servers, and use that for service checks.

example-01:~ # genhash -s 192.168.1.11 -p 80 -u /test.html
-----------------------[    HTTP Header Buffer    ]-----------------------
0000 48 54 54 50 2f 31 2e 31 - 20 32 30 30 20 4f 4b 0d   HTTP/1.1 200 OK.
0010 0a 44 61 74 65 3a 20 54 - 68 75 2c 20 31 32 20 53   .Date: Thu, 12 S
0020 65 70 20 32 30 30 32 20 - 31 39 3a 34 31 3a 35 39   ep 2002 19:41:59
0030 20 47 4d 54 0d 0a 53 65 - 72 76 65 72 3a 20 41 70    GMT..Server: Ap
0040 61 63 68 65 2f 32 2e 30 - 2e 33 39 20 28 55 6e 69   ache/2.0.39 (Uni
0050 78 29 20 6d 6f 64 5f 73 - 73 6c 2f 32 2e 30 2e 33   x) mod_ssl/2.0.3
0060 39 20 4f 70 65 6e 53 53 - 4c 2f 30 2e 39 2e 36 20   9 OpenSSL/0.9.6
0070 50 48 50 2f 34 2e 32 2e - 31 0d 0a 4c 61 73 74 2d   PHP/4.2.1..Last-
0080 4d 6f 64 69 66 69 65 64 - 3a 20 54 75 65 2c 20 30   Modified: Tue, 0
0090 33 20 53 65 70 20 32 30 - 30 32 20 31 37 3a 34 31   3 Sep 2002 17:41
00a0 3a 31 31 20 47 4d 54 0d - 0a 45 54 61 67 3a 20 22   :11 GMT..ETag: "
00b0 31 65 35 35 66 2d 34 32 - 2d 64 33 36 63 33 62 63   1e55f-42-d36c3bc
00c0 30 22 0d 0a 41 63 63 65 - 70 74 2d 52 61 6e 67 65   0"..Accept-Range
00d0 73 3a 20 62 79 74 65 73 - 0d 0a 43 6f 6e 74 65 6e   s: bytes..Conten
00e0 74 2d 4c 65 6e 67 74 68 - 3a 20 36 36 0d 0a 43 6f   t-Length: 66..Co
00f0 6e 6e 65 63 74 69 6f 6e - 3a 20 63 6c 6f 73 65 0d   nnection: close.
0100 0a 43 6f 6e 74 65 6e 74 - 2d 54 79 70 65 3a 20 74   .Content-Type: t
0110 65 78 74 2f 68 74 6d 6c - 3b 20 63 68 61 72 73 65   ext/html; charse
0120 74 3d 49 53 4f 2d 38 38 - 35 39 2d 31 0d 0a 0d 0a   t=ISO-8859-1....
-----------------------[ HTTP Header Ascii Buffer ]-----------------------
HTTP/1.1 200 OK
Date: Thu, 12 Sep 2002 19:41:59 GMT
Server: Apache/2.0.39 (Unix) mod_ssl/2.0.39 OpenSSL/0.9.6 PHP/4.2.1
Last-Modified: Tue, 03 Sep 2002 17:41:11 GMT
ETag: "1e55f-42-d36c3bc0"
Accept-Ranges: bytes
Content-Length: 66
Connection: close
Content-Type: text/html; charset=ISO-8859-1

-----------------------[       HTML Buffer        ]-----------------------
0000 3c 48 54 4d 4c 3e 0a 3c - 42 4f 44 59 3e 0a 54 68   <HTML>.<BODY>.Th
0010 69 73 20 69 73 20 61 20 - 74 65 73 74 20 70 61 67   is is a test pag
0020 65 20 66 6f 72 20 6d 6f - 6e 69 74 6f 72 69 6e 67   e for monitoring
0030 2e 0a 3c 2f 42 4f 44 59 - 3e 0a 3c 2f 48 54 4d 4c   ..</BODY>.</HTML
0040 3e 0a                   -                           >.
-----------------------[    HTML MD5 resulting    ]-----------------------
0000 42 28 34 d1 d2 b9 72 ee - e9 e5 b8 75 e4 bd 8c 33   B(4...r....u...3
-----------------------[ HTML MD5 final resulting ]-----------------------
422834d1d2b972eee9e5b875e4bd8c33

example-01:~#

That very last string is what you need to keep track, as you will use this in your service check setup below.

Now for the config file:

-- cut here --
! This is a comment
! Configuration File for keepalived

global_defs {
   ! this is who emails will go to on alerts
   notification_email {
        admins@example.com
    fakepager@example.com
    ! add a few more email addresses here if you would like
   }
   notification_email_from admins@example.com

   ! I use the local machine to relay mail
   smtp_server 127.0.0.1
   smtp_connect_timeout 30

   ! each load balancer should have a different ID
   ! this will be used in SMTP alerts, so you should make
   ! each router easily identifiable
   lvs_id LVS_EXAMPLE_01
}

! takes over, you want the internal interface on the failed server
! to failover as well, otherwise nothing will work.
! you can have as many vrrp_sync_group blocks as you want.
vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}

! now we setup more information about are virtual server
! we are just setting up one for now, listening on port 22 for ssh
! requests.
! each interface needs at least one vrrp_instance
! each vrrp_instance is a group of VIPs that are logically grouped
! together
! you can have as many vrrp_instaces as you want

vrrp_instance VI_1 {
        state MASTER
        interface eth0
        lvs_sync_daemon_inteface eth0
    ! each virtual router id must be unique per instance name!
        virtual_router_id 51
        priority 150
    ! how often should we vote, in seconds?
        advert_int 1
        smtp_alert
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                192.168.1.11
        192.168.1.12
        ! and more if you want them
        }
}

! now I setup the instance that the real servers will use as a default
! gateway
! most of the config is the same as above, but on a different interface

vrrp_instance VI_GATEWAY {
        state MASTER
        interface eth1
        lvs_sync_daemon_inteface eth1
        virtual_router_id 52
        priority 150
        advert_int 1
        smtp_alert
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                10.20.40.1
        }
}

! vrrp_sync_groups make sure that several router instances
! stay together on a failure - a good example of this is
! that the external interface on one router fails and the backup server

! notice we do not setup a virtual_server block for the 10.20.40.1
! address in the VI_GATEWAY instance. That's because we are doing NAT
! on that IP, and nothing else.

virtual_server 192.168.1.12 22 {
    delay_loop 6

    ! use round-robin as a load balancing algorithm
    lb_algo rr

    ! we are doing NAT
    lb_kind NAT
    nat_mask 255.255.255.0

    protocol TCP

    ! there can be as many real_server blocks as you need

    real_server 10.20.40.20 22 {

    ! if we used weighted round-robin or a similar lb algo,
    ! we include the weight of this server

        weight 1

    ! here is a health checker for this server.
    ! we could use a custom script here (see the keepalived docs)
    ! but we will just make sure we can do a vanilla tcp connect()
    ! on port 22
    ! if it fails, we will pull this realserver out of the pool
    ! and send email about the removal
        TCP_CHECK {
                connect_timeout 3
        connect_port 22
        }
    }
    real_server 10.20.40.21 22 {

    ! if we used weighted round-robin or a similar lb algo,
    ! we include the weight of this server

        weight 1
        TCP_CHECK {
                connect_timeout 3
        connect_port 22
        }
    }
}

virtual_server 192.168.1.11 80 {
    delay_loop 10
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol TCP

! use this to specify which host keepalived asks for during an HTTP GET
    virtualhost www.example.com

    real_server 10.20.40.10 80 {
        weight 1
        HTTP_GET {

        ! for the path, don't include the host if you use
        ! a virtualhost
                url {
                        path /test.html

            ! the results from genhash go here
                        digest 422834d1d2b972eee9e5b875e4bd8c33
                }
                connect_timeout 10
                connect_port 80
                ! keepalived will retry this many times before a failure
                ! is marked
                nb_get_retry 3
                ! each retry will occur after this delay
                delay_before_retry 10
        }
    }
    real_server 10.20.40.11 80 {
        weight 1
        HTTP_GET {
                url {
                        path /test.html
                        digest 422834d1d2b972eee9e5b875e4bd8c33
                }
                connect_timeout 10
                nb_get_retry 3
                delay_before_retry 10
                connect_port 80
        }
    }
}

virtual_server 192.168.1.11 443 {
    delay_loop 10
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol TCP
    virtualhost www.example.com
    real_server 10.20.40.10 443 {
        weight 1
        SSL_GET {
                url {
                        path /test.html
                        digest 422834d1d2b972eee9e5b875e4bd8c33
                }
                connect_timeout 10
                connect_port 80
                nb_get_retry 3
                delay_before_retry 10
        }
    }
    real_server 10.20.40.11 443 {
        weight 1
        SSL_GET {
                url {
                        path /test.html
                        digest 422834d1d2b972eee9e5b875e4bd8c33
                }
                connect_timeout 10
                nb_get_retry 3
                delay_before_retry 10
                connect_port 80
        }
    }
}
! that's all

-- cut here --

Okay let's see what keepalived -d shows us:

Sep 12 14:13:11 example-01 Keepalived: ------< Global definitions >------
Sep 12 14:13:11 example-01 Keepalived: LVS ID = LVS_EXAMPLE_01
Sep 12 14:13:11 example-01 Keepalived: Smtp server = 127.0.0.1
Sep 12 14:13:11 example-01 Keepalived: Smtp server connection timeout = 100
Sep 12 14:13:11 example-01 Keepalived: Email notification from = admins@example.com, fakepager@example.com
Sep 12 14:13:11 example-01 Keepalived: Email notification = admins@example.com
Sep 12 14:13:11 example-01 Keepalived: ------< SSL definitions >------
Sep 12 14:13:11 example-01 Keepalived: Using autogen SSL context
Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Topology >------
Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_1
Sep 12 14:13:11 example-01 Keepalived:    Want State = MASTER
Sep 12 14:13:11 example-01 Keepalived:    Runing on device = eth0
Sep 12 14:13:11 example-01 Keepalived:    Virtual Router ID = 51
Sep 12 14:13:11 example-01 Keepalived:    Priority = 150
Sep 12 14:13:11 example-01 Keepalived:    Advert interval = 1sec
Sep 12 14:13:11 example-01 Keepalived:    Preempt Active
Sep 12 14:13:11 example-01 Keepalived:    Authentication type = SIMPLE_PASSWORD
Sep 12 14:13:11 example-01 Keepalived:    Password = example
Sep 12 14:13:11 example-01 Keepalived:    VIP count = 2
Sep 12 14:13:11 example-01 Keepalived:      VIP1 = 192.168.1.11/32
Sep 12 14:13:11 example-01 Keepalived:      VIP1 = 192.168.1.12/32
Sep 12 14:13:11 example-01 Keepalived: VRRP Instance = VI_GATEWAY
Sep 12 14:13:11 example-01 Keepalived:    Want State = MASTER
Sep 12 14:13:11 example-01 Keepalived:    Runing on device = eth1
Sep 12 14:13:11 example-01 Keepalived:    Virtual Router ID = 52
Sep 12 14:13:11 example-01 Keepalived:    Priority = 150
Sep 12 14:13:11 example-01 Keepalived:    Advert interval = 1sec
Sep 12 14:13:11 example-01 Keepalived:    Preempt Active
Sep 12 14:13:11 example-01 Keepalived:    Authentication type = SIMPLE_PASSWORD
Sep 12 14:13:11 example-01 Keepalived:    Password = example
Sep 12 14:13:11 example-01 Keepalived:    VIP count = 1
Sep 12 14:13:11 example-01 Keepalived:      VIP1 = 10.20.40.1/32
Sep 12 14:13:11 example-01 Keepalived: ------< VRRP Sync groups >------
Sep 12 14:13:11 example-01 Keepalived: VRRP Sync Group = VG1, MASTER
Sep 12 14:13:11 example-01 Keepalived:    monitor = VI_1
Sep 12 14:13:11 example-01 Keepalived:    monitor = VI_GATEWAY
Sep 12 14:13:11 example-01 Keepalived: ------< LVS Topology >------
Sep 12 14:13:11 example-01 Keepalived: System is compiled with LVS v1.0.4
Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.11, VPORT = 22
Sep 12 14:13:11 example-01 Keepalived:    delay_loop = 10, lb_algo = rr
Sep 12 14:13:11 example-01 Keepalived:    protocol = TCP
Sep 12 14:13:11 example-01 Keepalived:    lb_kind = NAT
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.20, RPORT = 22, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.21, RPORT = 22, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.12, VPORT = 80
Sep 12 14:13:11 example-01 Keepalived:    VirtualHost = www.example.com
Sep 12 14:13:11 example-01 Keepalived:    delay_loop = 10, lb_algo = rr
Sep 12 14:13:11 example-01 Keepalived:    protocol = TCP
Sep 12 14:13:11 example-01 Keepalived:    lb_kind = NAT
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.10, RPORT = 80, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.11, RPORT = 80, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived: VIP = 192.168.1.12, VPORT = 443
Sep 12 14:13:11 example-01 Keepalived:    VirtualHost = www.example.com
Sep 12 14:13:11 example-01 Keepalived:    delay_loop = 10, lb_algo = rr
Sep 12 14:13:11 example-01 Keepalived:    persistence timeout = 360
Sep 12 14:13:11 example-01 Keepalived:    protocol = TCP
Sep 12 14:13:11 example-01 Keepalived:    lb_kind = NAT
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.10, RPORT = 443, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived:    RIP = 10.20.40.11, RPORT = 443, WEIGHT = 1
Sep 12 14:13:11 example-01 Keepalived: ------< Health checkers >------
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.20:22
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = TCP_CHECK
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 22
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.21:22
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = TCP_CHECK
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 22
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.10:80
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = HTTP_GET
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 80
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Nb get retry = 3
Sep 12 14:13:11 example-01 Keepalived:    Delay before retry = 10
Sep 12 14:13:11 example-01 Keepalived:    Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.11:80
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = HTTP_GET
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 80
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Nb get retry = 3
Sep 12 14:13:11 example-01 Keepalived:    Delay before retry = 10
Sep 12 14:13:11 example-01 Keepalived:    Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.10:443
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = SSL_GET
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 443
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Nb get retry = 3
Sep 12 14:13:11 example-01 Keepalived:    Delay before retry = 10
Sep 12 14:13:11 example-01 Keepalived:    Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33
Sep 12 14:13:11 example-01 Keepalived: 10.20.40.11:443
Sep 12 14:13:11 example-01 Keepalived:    Keepalive method = SSL_GET
Sep 12 14:13:11 example-01 Keepalived:    Connection port = 443
Sep 12 14:13:11 example-01 Keepalived:    Connection timeout = 10
Sep 12 14:13:11 example-01 Keepalived:    Nb get retry = 3
Sep 12 14:13:11 example-01 Keepalived:    Delay before retry = 10
Sep 12 14:13:11 example-01 Keepalived:    Checked url = /test.html, digest = 422834d1d2b972eee9e5b875e4bd8c33

And ipvasdm:

example-01:~# ipvsadm
IP Virtual Server version 1.0.4 (size=65536)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP 192.168.1.12:ssh rr
-> 10.20.40.20:ssh        Masq    1      0          0
-> 10.20.40.21:ssh               Masq    1      0          1
TCP 192.168.1.11:http rr
-> 10.20.40.10:http         Masq    1      0          0
-> 10.20.40.11:http         Masq    1      0          0
TCP 192.168.1.11:http rr persistent 360
-> 10.20.40.10:https            Masq    1      0          0
-> 10.20.40.11:https            Masq    1      1          0

example-01:~ # ip addr list
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:e0:81:21:bb:1c brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.9/24 brd 192.168.1.254 scope global eth0
    inet 192.168.1.11/32 scope global eth0
    inet 192.168.1.12/32 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:e0:81:21:bb:1d brd ff:ff:ff:ff:ff:ff
    inet 10.20.40.2/24 brd 10.20.40.255 scope global eth1
    inet 10.20.40.1/32 scope global eth1

Remeber, if we wanted to put this in a failover config, we could just add another box with the same config file (modified with a different lvs_id, state and priority) and start up keepalived on the backup box.

9. Example startup script for SuSE 8.0

#! /bin/sh
# Copyright (c) 1995-2002 SuSE Linux AG, Nuernberg, Germany.
# All rights reserved.
#
# Author of template: Kurt Garloff <feedback@suse.de>
# Modified for keepalived by Adam Fletcher <adamf+keepalived@csh.rit.edu>
#
# /etc/init.d/keepalived
#
#
# LSB compliant service control script; see http://www.linuxbase.org/spec/
#
# System startup script for some example service or daemon keepalived (template)
#
### BEGIN INIT INFO
# Provides: keepalived
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 3 5
# Default-Stop:   0 1 2 6
# Description:    Start keepalived to allow XY and provide YZ
#       continued on second line by '#<TAB>'
### END INIT INFO
#
# Note on Required-Start: It does specify the init script ordering,
# not real dependencies. Depencies have to be handled by admin
# resp. the configuration tools (s)he uses.

# Source SuSE config (if still necessary, most info has been moved)
test -r /etc/rc.config && . /etc/rc.config

# Check for missing binaries (stale symlinks should not happen)
KEEPALIVED_BIN=/usr/local/sbin/keepalived
test -x $KEEPALIVED_BIN || exit 5
# Shell functions sourced from /etc/rc.status:
#      rc_check         check and set local and overall rc status
#      rc_status        check and set local and overall rc status
#      rc_status -v     ditto but be verbose in local rc status
#      rc_status -v -r ditto and clear the local rc status
#      rc_failed        set local and overall rc status to failed
#      rc_failed <num> set local and overall rc status to <num><num>
#      rc_reset         clear local rc status (overall remains)
#      rc_exit          exit appropriate to overall rc status
#      rc_active        checks whether a service is activated by symlinks
. /etc/rc.status

# First reset status of this service
rc_reset

# Return values acc. to LSB for all commands but status:
# 0 - success
# 1 - generic or unspecified error
# 2 - invalid or excess argument(s)
# 3 - unimplemented feature (e.g. "reload")
# 4 - insufficient privilege
# 5 - program is not installed
# 6 - program is not configured
# 7 - program is not running
#
# Note that starting an already running service, stopping
# or restarting a not-running service as well as the restart
# with force-reload (in case signalling is not supported) are
# considered a success.

case "$1" in
    start)
        echo -n "Starting keepalived"
        ## Start daemon with startproc(8). If this fails
        ## the echo return value is set appropriate.

        # NOTE: startproc returns 0, even if service is
        # already running to match LSB spec.
        startproc $KEEPALIVED_BIN -d

        # Remember status and be verbose
        rc_status -v
        ;;
    stop)
        echo -n "Shutting down keepalived"
        ## Stop daemon with killproc(8) and if this fails
        ## set echo the echo return value.

        killproc -TERM $KEEPALIVED_BIN
        # masquerade the rest of the 10.20.40.0/24 network
        # through the external IP

        # Remember status and be verbose
        rc_status -v
        ;;
    try-restart)
        ## Stop the service and if this succeeds (i.e. the
        ## service was running before), start it again.
        ## Note: try-restart is not (yet) part of LSB (as of 0.7.5)
        $0 status >/dev/null && $0 restart

        # Remember status and be quiet
        rc_status
        ;;
    restart)
        ## Stop the service and regardless of whether it was
        ## running or not, start it again.
        $0 stop
        $0 start

        # Remember status and be quiet
        rc_status
        ;;
    force-reload)
        ## Signal the daemon to reload its config. Most daemons
        ## do this on signal 1 (SIGHUP).
        ## If it does not support it, restart.

        echo -n "Reload service keepalived"
        ## if it supports it:
        killproc -HUP $KEEPALIVED_BIN
        touch /var/run/keepalived.pid
        rc_status -v

        ## Otherwise:
        #$0 stop && $0 start
        #rc_status
        ;;
    reload)
        ## Like force-reload, but if daemon does not support
        ## signalling, do nothing (!)

        # If it supports signalling:
        echo -n "Reload service keepalived"
        killproc -HUP $KEEPALIVED_BIN
        touch /var/run/keepalived.pid
        rc_status -v

        ## Otherwise if it does not support reload:
        #rc_failed 3
        #rc_status -v
        ;;
    status)
        echo -n "Checking for service keepalived: "
        ## Check status with checkproc(8), if process is running
        ## checkproc will return with exit status 0.

        # Return value is slightly different for the status command:
        # 0 - service running
        # 1 - service dead, but /var/run/ pid file exists
        # 2 - service dead, but /var/lock/ lock file exists
        # 3 - service not running

        # NOTE: checkproc returns LSB compliant status values.
        checkproc $KEEPALIVED_BIN
        rc_status -v
        ;;
    probe)
        ## Optional: Probe for the necessity of a reload,
        ## print out the argument which is required for a reload.

        test /etc/keepalived/keepalived.conf -nt /var/run/keepalived.pid && echo reload
        ;;
    *)
        echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}"
        exit 1
        ;;
esac
rc_exit

10. Troubleshooting

Run keepalived with the -d option and watch your /var/log/messages
Look at the output of ipvsadm
Look at the output of ip addr list

Important caveats:

Turn on IP forwarding (echo "1" > /proc/sys/net/ipv4/ip_forward)
Get the latest LVS - don't go to a mirror of linuxvirtualserver.org unless you are sure it has the latest version (i got burned by this for a day!)
The realserver must use the load balancer as it's default gateway.
The realserver must not be on both the internal and external network, they must be on separate segements/LANs/VLANs.
You *must* have a vrrp_instance section defined for keepalived to setup the VIPs.
In VRRP, a lower number for the priority means that VRRP instance has less say - in other words, a lower priority is a lower number.
VRRP instances can not have the same router_id - keepalived won't give any errors, but you won't see that VRRP instance's IP addresses.
While keepalived won't complain if you leave out some options in the config file, it's not recommended. Just because you put a TCP_CHECK on a realserver that listens on port 80, don't leave out the connect_port 80 directive!

These may seem obvious, but they are all things that happened to me during my setup of keepalived.

If you have further questions, and have read this and all other keepalive documentation, please subscribe to the the keepalived-devel mailing list, available at http://www.keepalived.org/mailinglist.html

11. Credits

Keepalived is written and maintained by Alexandre Cassen. It is primarily through his effort that it exists at all, and I thank him for that efffort.

This HOWTO is written by Adam Fletcher, adamf+keepalived@csh.rit.edu.

This document is Copyright 2002 by Adam Fletcher Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is available athttp://www.gnu.org/copyleft/fdl.html

From:http://www.keepalived.org/LVS-NAT-Keepalived-HOWTO.html

集群的可扩展性及其分布式体系结构（一）--介绍

shanyiwan@live.com() — Fri, 09 Oct 2009 10:04:27 GMT

这是一个新的关于集群技术的专栏。作者将侧重就集群的可扩展性及体系结构分析、原理论、集群的考量、具体的分析案例（LVS、 beowulf、MOSIX）、集群高可用技术、分布式文件系统等等各个方面为您更加深入的介绍集群系统。本文是第一篇。主要阐述集群起源的概念，分布式系统的定义和目标体现。作者通过一个故事讲述了考虑集群解决方案时需要注意的是：如何正确看待集群，应该从什么角度来考察一个集群系统。解决了集群的这一根本性立场和观点角度，才能利用集群的技术解决现实问题。

前言

"话说天下大势，分久必合，合久必分"高端计算的体系结构发展正是遵循着这样一个规律。从金融、电信等大行业的数据集中处理到分布计算处理，再到二次集中和海量计算，趋势和应用显然是在变化，作为技术本身，分区和群集技术正是为适应不断变化的需要而产生的。

分而治之

分区技术初始于大型主机时代。企业级用户在购买大机后，许多应用都只使用大机的一小部分资源。这时，如果一个企业的生产部门只需要使用这台机器的10%的资源，而且需要比较好的安全性，那么分区技术就能够满足它的要求，将主机的部分处理器和系统资源隔离出来，只利用这些资源组织计算系统，供生产部门使用，其他的计算资源不受任何影响。这是早期的物理分区的概念。分区技术发展到今天，并不只为大型主机服务，因为分布式计算的应用在不断地扩展，在许多方面，高性能的UNIX服务器和IA架构服务器已经在慢慢地取代大机的位置，分区这项技术在这些领域中也有着重要的作用。

分区带给人们的是更高的投资回报率和使用率，更加灵活的应用水平，以及动态地使用和分配资源。这些特点通过不同的分区种类来实现，目前分区的种类越来越多，而且越来越细致，更倾向于客户的应用。最典型的分区技术就数我们在PC上使用的多操作系统了，这属于本地物理分区的范畴。而在多机环境中，通过分区可以将不同"种群"的计算机分割开来（即使几台计算机位于同一个物理位置也可能服务于不同的"种群"），被分区的计算机还可以在不同的层面上复用，以提高计算机系统的利用率。最常见的情况莫如VPN了，借助安全协议，VPN可以把分散的远程的计算机系统集合成一个逻辑上的"局域网"，协同完成复杂任务并保证远程通信的安全；而这些计算机可能同那些物理上相近的计算机（比如同一个办公室）再次合并为一个机群，利用空闲时间完成另外一些实时性不高的计算任务。

现在来讲讲"合"

集群技术是一种计算机系统之间连接的方式，运用它可以将分散的计算系统连接起来完成原来单独节点的计算系统无法完成的任务，最早的集群系统就是出于这样的并行处理的目的出现的。但是随着计算机性能的发展和网络不安全因素的出现，系统的稳定性和可靠性是人们主要面临解决的问题。人们开始使用集群方式将两台以上的设备连接起来，当整个集群系统中发生单点或者局部多点故障时，集群中其他的计算机将自动来接替出故障设备。最典型的例子就是实现双机热备份。两台一样的计算机系统，使用集群软件连接起来，其中一台作为另一台的备份，当主机系统崩溃时，另一台来接替它的任务。除此以外，利用集群系统的高并行性能进行复杂的科学、工程计算，也是非常有经济价值的做法。

对于集群系统来说，计算机的架构是一个非常重要的问题。同种架构的计算机之间，可以比较容易实现集群连接，一般来讲都是由厂商提供的集群软件。对于异构的计算机系统，一般没有特别多的选择，目前类似JAVA的中间件技术可以解决部分跨平台的问题。集群技术可以有效地解决开放系统的稳定、抗压和负载平衡问题。

为什么需要集群

以往，企业用户在构建信息应用系统时只能在两种计算体系中进行选择，一是基于主机的计算体系，大体上采用的就是我们上面描述的分区方式；另一就是基于客户机/服务器的群集型计算体系。基于主机的计算体系具有良好的系统可伸缩性、可靠性和高性能，但它昂贵的价格迫使用户在系统建设之初就在硬件系统上投入巨大资源，而用户花费巨额资金购买的主机系统也许包含了不少并不需要的功能，造成资源浪费。基于客户机/服务器的群集型计算体系允许用户根据实际需要逐步增加硬件系统，但这种体系的系统毕竟不是真正意义上的集群，缺乏必要的可用性和可管理性，在应用升级和管理方面都需要用户付出巨大的代价，每增加一个服务器/ 客户端的连接就要同时增加两端的管理负担。

因此，未来对计算速度、系统可靠性和成本实效性的要求必将促使发展另外的计算机模型来替代上述的计算模型。随着计算机网络的出现，一个新的具有更高性能/ 价格比的体系逐渐成为应用主流--分布式集群计算体系。当用户需要完成任何任务时，分布式集群计算提供了尽可能多的计算机处理能力和数据的透明访问能力，同时实现高性能与高可靠性的目标。

集群计算模式是最经济的计算模式。集群系统允许用户把普通商用硬件系统组成机群，并根据需要随时在机群中增加新的硬件，提高系统的伸缩性和可用性，从而能够在价格相对低廉的中低端平台上享用过去只有高端系统才具备的高可伸缩性和高可用性，既提高了系统的性能，同时也降低了成本,实现了更多的计算机=更快速度的目标。

人们对集群计算的兴趣日益增加。集群计算的主题是多种多样的，许多研究人员正在研究有关分布式硬件体系结构和分布式系统软件设计的各方面问题以开发利用潜在的集群并行性和集群可用性。

集群计算系统（或叫做分布式系统）多种多样并涉及不同的系统体系结构。对一些用户来说，一个集群系统是为解决单个问题而紧密结合在一起工作的多处理机的集合。对另一些用户来说，一个集群系统可能意味着一个由地理上分散的各自独立的处理机组成的计算机网络，这些处理机连接在一起以实现对不同资源的共享。

然而，集群这个词在计算机系统中被如此广泛应用以至于它的使用变得有点贬值。许多这方面的混乱来源于缺乏对物理的分布和逻辑的分布的区分。通过区分这两个概念，就可以更准确地描述一个分布式系统的属性。

对于分布式集群系统，我们使用以下定义：在用户看来，一个集群系统是一个单一的普通系统，然而运行在一系列自治处理单元（PE也叫节点）上的系统，每个处理单元（结点）有各自的物理内存空间并且并且通过高速链路或者标准商品化网络连接。结点间通过紧密的合作实现对同一任务的协同计算。系统必须支持任意数量的进程和结点的动态扩展。

建立一个集群系统的主要目的在于：

固有的应用得以保障。集群系统以一种很自然的方式开始存在，例如，在我们的社会中，人们常常以群体的形式出现并彼此共享信息，公司、社团、班级等等都是这样的概念。在从个人计算向集群分布式计算迁移的时候，往往可以保留原有在个人计算系统上的应用，直接将原有的应用重新在新的集群系统中运行，并获得性能的提升。这也是集群出现的一大原因。
性能/成本。集群系统的并行性降低了处理的瓶颈，提供了全面改进的性能，也就是说，集群系统提供了更好的性能价格比。
资源共享。集群系统能有效地支持不同位置的用户对信息和资源（硬件和软件）的共享。
灵活性和可扩展性。集群系统可以增量扩展，并能方便地修改或扩展系统以适应变化的环境而无需中断其运行。
实用性和容错性。依靠存储单元和处理单元的多重性，集群系统具有在系统出现故障的情况下继续运行的潜力。
可伸缩性。集群系统能容易地扩大以包括更多的资源（硬件和软件）。

最近，我们注意到，除了原有的以生产商用专有集群系统为主的厂商仍在发展自己的专有平台集群系统外，几个主要的硬件厂商都在研制并有计划推出基于 Linux的开放源码的集群产品，比如IBM、HP、SGI等等。开放源码界的迅猛发展为集群的出现提供良好的技术舞台，而向TurboLinux、 VALinux等老牌的Linux厂商更是把Linux集群这一高端应用领域作为自己的战略发展方向，不遗余力的加入激烈的市场产品竞争中。涌现在我们面前的则是多种多样的可供选择的集群解决方案，有硬件的也有软件的。那么，如何借助开放源码届丰富的技术资源，实现一个优秀的集群系统呢？首先可以肯定的是：一个优秀的集群系统不可能是一蹴而就的，需要仔细考虑用户的应用环境、业务需求、所能投入的成本还有技术的后继支援。下面，我们从集群技术方面一些比较重要的功能来探讨一个优秀集群系统所应该具备的一些根本性的因素。而在此之前，我们先来看一个可能是最常见的小故事。

管理员的故事

对于用户来说，怎样看待计算机系统和计算资源是因人而异的。特别是在群集应用环境中，不同的用户需求对资源的要求和侧重点也有所不同。

这里有一个故事，发生在一个被频繁访问的小门户站点。网站刚开始运行了一段时间后，情况良好，但是不久后管理员发现用户抱怨响应速度很低。于是，他升级了服务器的CPU和磁盘系统，又加了512M的内存。心想，这会总不会有人再说什么了吧。可是，过了没多久，（当然，我们的这位站点很幸运的又吸引了大量的用户，现在用户数量变成了原来的5倍），情况更糟糕了，在高峰期间有的访问甚至被站点拒绝。这是怎么回事呢？再升级吧，可硬件的升级似乎走到了尽头，我们的管理员再次陷入困境。

这里，不幸的管理员遇到了系统瓶颈问题。通过查看日志，我们发现服务器CPU的负荷基本保持在10％左右（占用率），但是却有很多请求等待处理。是CPU 不够快吗，显然不是；是磁盘太慢吗，已经用了最快的RAID阵列，平均寻道时间到达5ms，不能再快了；内存不够大吗，主板所能支持的只有4G，已经到了上限。所有的一切已经是极限，再升级下去只好更换到大型机了（那又会是一个无休止的投入）。站点每天仍要接受上百万次甚至上千万次的页面访问，怎么样才能既控制服务器硬件成本又能大幅度提高性能呢？

固然，添加高速磁盘阵列可以减少寻道时间，提高响应的速度，或者，使用Cache技术能够加快页面访问效率。但是，一味地依靠扩展单一系统的硬件终究会有上限，磁盘、内存、网络代宽的发展速度远远低于CPU的摩尔定律速度，造成彼此性能的互相牵制。而且，在单机系统种，越到高端的硬件，成本与性能比越是成指数下降（见下图）。解决这样的问题，就有必要从体系结构入手，而不是简单的升级机器。

单机系统性能/价格曲线

在上面的故事里。对于用户而言（指的是系统管理员和网站用户），服务器是提供web服务的资源，是若干计算、存储、数据的资源集合（查找数据、提供空间）。用户关心的是你的站点的响应速度而非你用了多快的CPU多大的内存。因此，我们从用户的角度出发，不难发现，如何看待你的计算资源，是紧紧和使用者的需求联系在一起的。那么，解决问题的关键就不是一些简单的数字层面上的思考了。沿着用户的需求--提高站点的响应速度，更确切的说缩短web站点响应每个HTTP请求的时间�D�D是管理员首要解决的问题。

我们并不否认升级硬件的作用，但是我们知道，这样的情况下，仅仅依赖硬件的局部更换是不经济的做法。CPU由500MHz变成1GHz，web的性能不会就提升两倍，甚至毫不夸张的说，这种升级只能提高web性能10％左右。喔，太令人沮丧了～～。

那我们能不能反过来想呢，既然量变没有实质作用，那干脆就来个大改革。彻底改变服务器的体系结构。当然，要把花费的钞票控制在BOSS可以承受的范围内，钱是用来花的不是用来烧的。方案可能有：

替换原有的系统，购买64位元服务器，比如SUN的Enterprise2000 ，不但CPU够劲，总线带宽也足够支撑严酷的应用环境考研，而且，千百万用户证实了：SUN就是我们".COM"的那个至关重要的"点"。软硬件都没话说，那我们就买吧！
呃～，等等，什么，你再说一遍，一台Enterprise要我XX万大洋！？原有的ASP要重写！！？？请一个管理员维护费要~~！！！
我已经听不下去了，更何况掏腰包的人。更换系统就意味着投资、投资、再投资，更要命的是，原来的东西怎么办？扔掉？留着也没用。
更换系统的硬件平台显然不是解决问题的好办法，那么，在原有的基础上扩展系统呢？于是，有了下面的这种方法：
利用免费的集群软件，在保留原有的硬件投资基础上，添加几台新的PC设备，组建负载均衡集群系统。既然一台机器解决不了问题，那就让多台机器一起分担吧。怎么做呢？
第一步：安装Linux或者BSD操作系统，由于拥有广大的开放源码届的大力支持，你无须担心缺少优秀应用程序。无论你是在怎样应用环境中，开放源码团体总能为你提供合适的稳定的软件。
第二步：安装相应的Kernel和有关的系统补丁。并且安装和你的kernel匹配的集群软件包。比较常见的有LVS、LSF、MOSIX等。都可以很好满足负载均衡的要求。其中LVS由于其基于Netfilter技术，具有很好的可扩展性和性能表现，受到广大开发者的青睐。
第三步：配置你的负载均衡集群系统。无非就是配置一些脚本文件。大多数的集群配置文件都是直接可编辑的，文法简单，编好即用，非常方便。
第四步：配置你的集群服务软件。这里的服务软件特指那些提供实际网络服务的应用程序。其实一般都是WEB SERVER或者Mail Server。故事中是一个Web站点，我们就弄个Apache吧。不管你是静态文本还是CGI，Apache都能够帮助你实现平滑的Web移植。如果你不幸使用了ASP这样的专有技术，你还可以考虑诸iASP这样的第三方插件来实现移植。一般来说，对于PHP、JSP和CGI，Apache都能支持的很好。
第五步：适当设定你的数据中心。其实这一步对于集群系统来说非常重要。因为由单台服务器转向分布式的集群结构，数据的存储一致问题就凸现而出。如果WEB 应用集中于数据库访问，那么可以使用集中的数据库服务器，Progresql或者Mysql甚至是Oracle都行，其本身对于数据的一致、安全、性能都有保障。或者，可以考虑使用NFS、AFS等分布式文件系统来共享集群各结点的存储空间和应用数据。
最后一步：写一份报告。告诉你的BOSS你不但顺利解决了站点的瓶颈问题，而且大大大减轻了他花钱升级的痛苦（实际上，除了购买廉价服务器的费用，其他基本上没有什么花费）。说不定吝啬的BOSS还会涨你的工资呢～嘿嘿。

小结

故事讲完，不知各位看官有何感想。其实集群技术并不是实验室的阳春白雪，也不是老教授闭门造车的产物。集群技术"和我们的国计民生，老百姓的生活有密切的关系" ；）。我们看到，适当的应用集群技术，不但具有明显的经济效益（很多用户就是冲着省钱而来），而且在性能、可用性、稳定性等诸多方面，具有很强的可扩展性。是计算体系近年来发展的主要方向。

研究集群技术，需要从多个方面着手，多角度的考虑集群系统。如果从上述的资源的角度考虑集群，就要注意用户是需要什么样的计算资源，是计算能力呢还是响应速度，或者是可用性；如果从硬件构架看，就有工作站集群（COW）、大规模并行处理机（MPP）、对称多处理机（SMP）、分布式异构计算集群（典型是 GRID）；如果从应用领域看，既有侧重于科学和工程计算的并行计算集群，也有侧重于作业负载的负载平衡集群，或者是广泛应用于严格商用环境的高可用容错集群；还有考虑集群的时候重点研究的几个方面：可用性、单一系统映像（SSI）、作业管理、调度、通信等等。可以说，孤立的从某一个角度或者某一个方面看集群，是无法真正了解集群的原理，准确的评估集群的各方面能力、性能，更别说为应用需求选择合适的集群环境了。了解集群，就得先从体系结构的设计和可扩展性开始。

利用keepalived 实现HA&LB

shanyiwan@live.com() — Fri, 25 Sep 2009 02:07:26 GMT

利用keepalived 实现HA&LB
装包
下载地址 wget http://www.keepalived.org/software/keepalived-1.1.17.tar.gz

./configure --prefix=/ --mandir=/usr/local/share/man  
--with-kernel-dir=/usr/src/kernels/2.6.18-128.el5-i686/

以上这步是设定安装路径以及帮助文件的路径可以自行定义后面那段具体作用大概是
with-kernel-dir 这个参数很重要，这个参数并不表示我们要把keepalived编进内核，
而是指使用内核源码里面的头文件，也就是include目录（如果需要用到LVS才需要这么做，
否则是不需要的，而且如果要使用netlink，还需要link_watch.cz这个文件）

make  
make install

然后修改 keepalived 的配置文件vim /etc/keepalived/keepalived.conf

{  
   notification_email_from hongzhou@sina.com #指定 keepalived 在发生事件（如切换）时，需要发送email到对象，可以填多个，每行一个  
 
   smtp_server 192.168.1.1 #这两项是邮件服务器的配置  
   smtp_connect_timeout 30  
   router_id LVS_DEVEL       # 运行keepalived的机器的一个标识  
}  
#VRRPD 配置  
vrrp_sync_group VG1 {  
 group {  
  VI_1 #(VRRP实例名)  
  }  
       }  
 
#VRRP 实例配置  
vrrp_instance VI_1 {   
    state BACKUP # 无论主与备都设置为BACKUP是为了不抢占主director，通过priority来确认谁先起作用，主的优先级至少高于从50  
    interface eth0 # （inside_network）实例绑定的网卡  
    macast_src_ip #发送多播包的地址，如果不设置，默认使用绑定网卡的 primary IP  
   建议使用专门网卡并使用这项，问题：如果这块儿网卡出了问题如何处理？  
   
    virtual_router_id 51   #VRID 标记 (0-255)主备服务器必须一致  
    priority 101 #优先级  
    advert_int 1 #检查间隔，默认为1秒  
    nopreempt # 不抢占 这个配置只能设置在 state为 BACKUP的主机上，而且这台主机的priority必须比另一台要高  
    authentication { #这是认证的相关设置  
        auth_type PASS #认证方式 PASS | AH  
        auth_pass 33210 #认证密码  
    }  
    virtual_ipaddress {  
        192.168.0.16    # VIP 地址  
    }  
}  
 
virtual_server 192.168.0.16 80 {  
    delay_loop 6 # 每六秒查询realserver状态  
    lb_algo rr   #lvs采用轮询算法 rr|wrr|lc|wlc|lblc|sh|dh  
    lb_kind DR   #lvs使用Direct Route NAT|DR|TUN  
    protocol TCP #使用TCP 协议来检查  
    persistence_timeout 600 # 会话保持时间，就是同一个ip在600秒内会被分到同一个realserver上  
   
    real_server 192.168.0.3 80 {  
        weight 1  #权重  
 inhibit_on_failure # 在检查失败的时候 将权重设为0 并继续监控而不是直接把他踢掉  
        TCP_CHECK { #以什么方式来检查 健康状况 HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK  
             
            connect_timeout 3 #连接超时时间  
            nb_get_retry 3    #重试连接次数  
            delay_before_retry 3 #每次重试连接间隔 （单位秒）  
     bindto 192.168.0.16 #以此地址来 向服务器发送请求来检查其健康状态  
            #注意HTTP_GET|SSL_GET|SMTP_CHECK|MISC_CHECK 这几种检查方式请参考《keepalived权威指南》第十四页     
                  }  
                                }  
    real_server 192.168.0.4 80 {  
        weight 1  
        TCP_CHECK {  
            connect_timeout 3  
            nb_get_retry 3  
            delay_before_retry 3  
                  }  
                                }  
}

realservice 配置脚本

ifconfig lo:1 192.168.0.16 netmask 255.255.255.255  
echo 2 > /proc/sys/net/ipv4/conf/eth1/arp_announce  
echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce  
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore  
echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_ignore

HAProxy Architecture Guide

shanyiwan@live.com() — Wed, 23 Sep 2009 05:54:58 GMT

                           -------------------
                                 HAProxy
                           Architecture  Guide
                           -------------------
                             version 1.1.34
                              willy tarreau
                               2006/01/29
This document provides real world examples with working configurations.
Please note that except stated otherwise, global configuration parameters
such as logging, chrooting, limits and time-outs are not described here.
===================================================
1. Simple HTTP load-balancing with cookie insertion
===================================================
A web application often saturates the front-end server with high CPU loads,
due to the scripting language involved. It also relies on a back-end database
which is not much loaded. User contexts are stored on the server itself, and
not in the database, so that simply adding another server with simple IP/TCP
load-balancing would not work.
                +-------+
                |clients|  clients and/or reverse-proxy
                +---+---+
                    |
                   -+-----+--------+----
                          |       _|_db
                       +--+--+   (___)
                       | web |   (___)
                       +-----+   (___)
                   192.168.1.1   192.168.1.2
Replacing the web server with a bigger SMP system would cost much more than
adding low-cost pizza boxes. The solution is to buy N cheap boxes and install
the application on them. Install haproxy on the old one which will spread the
load across the new boxes.
  192.168.1.1    192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----------+-----+-----+-----+--------+----
        |           |     |     |     |       _|_db
     +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | A | | B | | C | | D |    (___)
     +-----+      +---+ +---+ +---+ +---+    (___)
     haproxy        4 cheap web servers
Config on haproxy (LB1) :
-------------------------
       
    listen webfarm 192.168.1.1:80
       mode http
       balance roundrobin
       cookie SERVERID insert indirect
       option httpchk HEAD /index.html HTTP/1.0
       server webA 192.168.1.11:80 cookie A check
       server webB 192.168.1.12:80 cookie B check
       server webC 192.168.1.13:80 cookie C check
       server webD 192.168.1.14:80 cookie D check
       
Description :
-------------
 - LB1 will receive clients requests.
 - if a request does not contain a cookie, it will be forwarded to a valid
   server
 - in return, a cookie "SERVERID" will be inserted in the response holding the
   server name (eg: "A").
 - when the client comes again with the cookie "SERVERID=A", LB1 will know that
   it must be forwarded to server A. The cookie will be removed so that the
   server does not see it.
 - if server "webA" dies, the requests will be sent to another valid server
   and a cookie will be reassigned.
Flows :
-------
(client)                           (haproxy)                         (server A)
  >-- GET /URI1 HTTP/1.0 ------------> |
               ( no cookie, haproxy forwards in load-balancing mode. )
                                       | >-- GET /URI1 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
               ( the proxy now adds the server cookie in return )
  <-- HTTP/1.0 200 OK ---------------< |
      Set-Cookie: SERVERID=A           |
  >-- GET /URI2 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
      ( the proxy sees the cookie. it forwards to server A and deletes it )
                                       | >-- GET /URI2 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
   ( the proxy does not add the cookie in return because the client knows it )
  <-- HTTP/1.0 200 OK ---------------< |
  >-- GET /URI3 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
                                    ( ... )
Limits :
--------
 - if clients use keep-alive (HTTP/1.1), only the first response will have
   a cookie inserted, and only the first request of each session will be
   analyzed. This does not cause trouble in insertion mode because the cookie
   is put immediately in the first response, and the session is maintained to
   the same server for all subsequent requests in the same session. However,
   the cookie will not be removed from the requests forwarded to the servers,
   so the server must not be sensitive to unknown cookies. If this causes
   trouble, you can disable keep-alive by adding the following option :
        option httpclose
 - if for some reason the clients cannot learn more than one cookie (eg: the
   clients are indeed some home-made applications or gateways), and the
   application already produces a cookie, you can use the "prefix" mode (see
   below).
 - LB1 becomes a very sensible server. If LB1 dies, nothing works anymore.
   => you can back it up using keepalived (see below)
 - if the application needs to log the original client's IP, use the
   "forwardfor" option which will add an "X-Forwarded-For" header with the
   original client's IP address. You must also use "httpclose" to ensure
   that you will rewrite every requests and not only the first one of each
   session :
        option httpclose
        option forwardfor
   The web server will have to be configured to use this header instead.
   For example, on apache, you can use LogFormat for this :
        LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b " combined
        CustomLog /var/log/httpd/access_log combined
Hints :
-------
Sometimes on the internet, you will find a few percent of the clients which
disable cookies on their browser. Obviously they have troubles everywhere on
the web, but you can still help them access your site by using the "source"
balancing algorithm instead of the "roundrobin". It ensures that a given IP
address always reaches the same server as long as the number of servers remains
unchanged. Never use this behind a proxy or in a small network, because the
distribution will be unfair. However, in large internal networks, and on the
internet, it works quite well. Clients which have a dynamic address will not
be affected as long as they accept the cookie, because the cookie always has
precedence over load balancing :
    listen webfarm 192.168.1.1:80
       mode http
       balance source
       cookie SERVERID insert indirect
       option httpchk HEAD /index.html HTTP/1.0
       server webA 192.168.1.11:80 cookie A check
       server webB 192.168.1.12:80 cookie B check
       server webC 192.168.1.13:80 cookie C check
       server webD 192.168.1.14:80 cookie D check
       
==================================================================
2. HTTP load-balancing with cookie prefixing and high availability
==================================================================
Now you don't want to add more cookies, but rather use existing ones. The
application already generates a "JSESSIONID" cookie which is enough to track
sessions, so we'll prefix this cookie with the server name when we see it.
Since the load-balancer becomes critical, it will be backed up with a second
one in VRRP mode using keepalived under Linux.
Download the latest version of keepalived from this site and install it
on each load-balancer LB1 and LB2 :
       http://www.keepalived.org/
You then have a shared IP between the two load-balancers (we will still use the
original IP). It is active only on one of them at any moment. To allow the
proxy to bind to the shared IP on Linux 2.4, you must enable it in /proc :
# echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind
    shared IP=192.168.1.1
  192.168.1.3  192.168.1.4    192.168.1.11-192.168.1.14   192.168.1.2
 -------+------------+-----------+-----+-----+-----+--------+----
        |            |           |     |     |     |       _|_db
     +--+--+      +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | LB2 |      | A | | B | | C | | D |    (___)
     +-----+      +-----+      +---+ +---+ +---+ +---+    (___)
     haproxy      haproxy        4 cheap web servers
     keepalived   keepalived
Config on both proxies (LB1 and LB2) :
--------------------------------------       
    listen webfarm 192.168.1.1:80
       mode http
       balance roundrobin
       cookie JSESSIONID prefix
       option httpclose
       option forwardfor
       option httpchk HEAD /index.html HTTP/1.0
       server webA 192.168.1.11:80 cookie A check
       server webB 192.168.1.12:80 cookie B check
       server webC 192.168.1.13:80 cookie C check
       server webD 192.168.1.14:80 cookie D check
       
Notes: the proxy will modify EVERY cookie sent by the client and the server,
so it is important that it can access to ALL cookies in ALL requests for
each session. This implies that there is no keep-alive (HTTP/1.1), thus the
"httpclose" option. Only if you know for sure that the client(s) will never
use keep-alive (eg: Apache 1.3 in reverse-proxy mode), you can remove this
option.
Configuration for keepalived on LB1/LB2 :
-----------------------------------------
    vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
        script "killall -0 haproxy"     # cheaper than pidof
        interval 2                      # check every 2 seconds
	weight 2                        # add 2 points of prio if OK
    }
    vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101                    # 101 on master, 100 on backup
        virtual_ipaddress {
            192.168.1.1
        }
        track_script {
            chk_haproxy
        }
    }
Description :
-------------
 - LB1 is VRRP master (keepalived), LB2 is backup. Both monitor the haproxy
   process, and lower their prio if it fails, leading to a failover to the
   other node.
 - LB1 will receive clients requests on IP 192.168.1.1.
 - both load-balancers send their checks from their native IP.
 - if a request does not contain a cookie, it will be forwarded to a valid
   server
 - in return, if a JESSIONID cookie is seen, the server name will be prefixed
   into it, followed by a delimitor ('~')
 - when the client comes again with the cookie "JSESSIONID=A~xxx", LB1 will
   know that it must be forwarded to server A. The server name will then be
   extracted from cookie before it is sent to the server.
 - if server "webA" dies, the requests will be sent to another valid server
   and a cookie will be reassigned.
Flows :
-------
(client)                           (haproxy)                         (server A)
  >-- GET /URI1 HTTP/1.0 ------------> |
               ( no cookie, haproxy forwards in load-balancing mode. )
                                       | >-- GET /URI1 HTTP/1.0 ---------->
                                       |     X-Forwarded-For: 10.1.2.3
                                       | <-- HTTP/1.0 200 OK -------------<
                        ( no cookie, nothing changed )
  <-- HTTP/1.0 200 OK ---------------< |
  >-- GET /URI2 HTTP/1.0 ------------> |
    ( no cookie, haproxy forwards in lb mode, possibly to another server. )
                                       | >-- GET /URI2 HTTP/1.0 ---------->
                                       |     X-Forwarded-For: 10.1.2.3
                                       | <-- HTTP/1.0 200 OK -------------<
                                       |     Set-Cookie: JSESSIONID=123
    ( the cookie is identified, it will be prefixed with the server name )
  <-- HTTP/1.0 200 OK ---------------< |
      Set-Cookie: JSESSIONID=A~123     |
  >-- GET /URI3 HTTP/1.0 ------------> |
      Cookie: JSESSIONID=A~123         |
       ( the proxy sees the cookie, removes the server name and forwards
          to server A which sees the same cookie as it previously sent )
                                       | >-- GET /URI3 HTTP/1.0 ---------->
                                       |     Cookie: JSESSIONID=123
                                       |     X-Forwarded-For: 10.1.2.3
                                       | <-- HTTP/1.0 200 OK -------------<
                        ( no cookie, nothing changed )
  <-- HTTP/1.0 200 OK ---------------< |
                                    ( ... )
Hints :
-------
Sometimes, there will be some powerful servers in the farm, and some smaller
ones. In this situation, it may be desirable to tell haproxy to respect the
difference in performance. Let's consider that WebA and WebB are two old
P3-1.2 GHz while WebC and WebD are shiny new Opteron-2.6 GHz. If your
application scales with CPU, you may assume a very rough 2.6/1.2 performance
ratio between the servers. You can inform haproxy about this using the "weight"
keyword, with values between 1 and 256. It will then spread the load the most
smoothly possible respecting those ratios :
       server webA 192.168.1.11:80 cookie A weight 12 check
       server webB 192.168.1.12:80 cookie B weight 12 check
       server webC 192.168.1.13:80 cookie C weight 26 check
       server webD 192.168.1.14:80 cookie D weight 26 check
========================================================
2.1 Variations involving external layer 4 load-balancers
========================================================
Instead of using a VRRP-based active/backup solution for the proxies,
they can also be load-balanced by a layer4 load-balancer (eg: Alteon)
which will also check that the services run fine on both proxies :
              | VIP=192.168.1.1
         +----+----+
         | Alteon  |
         +----+----+
              |
 192.168.1.3  |  192.168.1.4  192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----+------+-----------+-----+-----+-----+--------+----
        |            |           |     |     |     |       _|_db
     +--+--+      +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | LB2 |      | A | | B | | C | | D |    (___)
     +-----+      +-----+      +---+ +---+ +---+ +---+    (___)
     haproxy      haproxy        4 cheap web servers
Config on both proxies (LB1 and LB2) :
--------------------------------------
       
    listen webfarm 0.0.0.0:80
       mode http
       balance roundrobin
       cookie JSESSIONID prefix
       option httpclose
       option forwardfor
       option httplog
       option dontlognull
       option httpchk HEAD /index.html HTTP/1.0
       server webA 192.168.1.11:80 cookie A check
       server webB 192.168.1.12:80 cookie B check
       server webC 192.168.1.13:80 cookie C check
       server webD 192.168.1.14:80 cookie D check
The "dontlognull" option is used to prevent the proxy from logging the health
checks from the Alteon. If a session exchanges no data, then it will not be
logged.
       
Config on the Alteon :
----------------------
    /c/slb/real  11
           ena
           name "LB1"
           rip 192.168.1.3
    /c/slb/real  12
           ena
           name "LB2"
           rip 192.168.1.4
    /c/slb/group 10
           name "LB1-2"
           metric roundrobin
           health tcp
           add 11
           add 12
    /c/slb/virt 10
           ena
           vip 192.168.1.1
    /c/slb/virt 10/service http
           group 10
Note: the health-check on the Alteon is set to "tcp" to prevent the proxy from
forwarding the connections. It can also be set to "http", but for this the
proxy must specify a "monitor-net" with the Alteons' addresses, so that the
Alteon can really check that the proxies can talk HTTP but without forwarding
the connections to the end servers. Check next section for an example on how to
use monitor-net.
============================================================
2.2 Generic TCP relaying and external layer 4 load-balancers
============================================================
Sometimes it's useful to be able to relay generic TCP protocols (SMTP, TSE,
VNC, etc...), for example to interconnect private networks. The problem comes
when you use external load-balancers which need to send periodic health-checks
to the proxies, because these health-checks get forwarded to the end servers.
The solution is to specify a network which will be dedicated to monitoring
systems and must not lead to a forwarding connection nor to any log, using the
"monitor-net" keyword. Note: this feature expects a version of haproxy greater
than or equal to 1.1.32 or 1.2.6.
                |  VIP=172.16.1.1   |
           +----+----+         +----+----+
           | Alteon1 |         | Alteon2 |
           +----+----+         +----+----+
 192.168.1.252  |  GW=192.168.1.254 |  192.168.1.253
                |                   |
          ------+---+------------+--+-----------------> TSE farm : 192.168.1.10
       192.168.1.1  |            | 192.168.1.2
                 +--+--+      +--+--+
                 | LB1 |      | LB2 |
                 +-----+      +-----+
                 haproxy      haproxy
Config on both proxies (LB1 and LB2) :
--------------------------------------
       
    listen tse-proxy
       bind :3389,:1494,:5900  # TSE, ICA and VNC at once.
       mode tcp
       balance roundrobin
       server tse-farm 192.168.1.10
       monitor-net 192.168.1.252/31
The "monitor-net" option instructs the proxies that any connection coming from
192.168.1.252 or 192.168.1.253 will not be logged nor forwarded and will be
closed immediately. The Alteon load-balancers will then see the proxies alive
without perturbating the service.
Config on the Alteon :
----------------------
    /c/l3/if 1
           ena
           addr 192.168.1.252
           mask 255.255.255.0
    /c/slb/real  11
           ena
           name "LB1"
           rip 192.168.1.1
    /c/slb/real  12
           ena
           name "LB2"
           rip 192.168.1.2
    /c/slb/group 10
           name "LB1-2"
           metric roundrobin
           health tcp
           add 11
           add 12
    /c/slb/virt 10
           ena
           vip 172.16.1.1
    /c/slb/virt 10/service 1494
           group 10
    /c/slb/virt 10/service 3389
           group 10
    /c/slb/virt 10/service 5900
           group 10
Special handling of SSL :
-------------------------
Sometimes, you want to send health-checks to remote systems, even in TCP mode,
in order to be able to failover to a backup server in case the first one is
dead. Of course, you can simply enable TCP health-checks, but it sometimes
happens that intermediate firewalls between the proxies and the remote servers
acknowledge the TCP connection themselves, showing an always-up server. Since
this is generally encountered on long-distance communications, which often
involve SSL, an SSL health-check has been implemented to workaround this issue.
It sends SSL Hello messages to the remote server, which in turns replies with
SSL Hello messages. Setting it up is very easy :
    listen tcp-syslog-proxy
       bind :1514      # listen to TCP syslog traffic on this port (SSL)
       mode tcp
       balance roundrobin
       option ssl-hello-chk
       server syslog-prod-site 192.168.1.10 check
       server syslog-back-site 192.168.2.10 check backup
=========================================================
3. Simple HTTP/HTTPS load-balancing with cookie insertion
=========================================================
This is the same context as in example 1 above, but the web
server uses HTTPS.
                +-------+
                |clients|  clients
                +---+---+
                    |
                   -+-----+--------+----
                          |       _|_db
                       +--+--+   (___)
                       | SSL |   (___)
                       | web |   (___)
                       +-----+
                   192.168.1.1   192.168.1.2
Since haproxy does not handle SSL, this part will have to be extracted from the
servers (freeing even more ressources) and installed on the load-balancer
itself. Install haproxy and apache+mod_ssl on the old box which will spread the
load between the new boxes. Apache will work in SSL reverse-proxy-cache. If the
application is correctly developped, it might even lower its load. However,
since there now is a cache between the clients and haproxy, some security
measures must be taken to ensure that inserted cookies will not be cached.
  192.168.1.1    192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----------+-----+-----+-----+--------+----
        |           |     |     |     |       _|_db
     +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | A | | B | | C | | D |    (___)
     +-----+      +---+ +---+ +---+ +---+    (___)
     apache         4 cheap web servers
     mod_ssl
     haproxy 
Config on haproxy (LB1) :
-------------------------
       
    listen 127.0.0.1:8000
       mode http
       balance roundrobin
       cookie SERVERID insert indirect nocache
       option httpchk HEAD /index.html HTTP/1.0
       server webA 192.168.1.11:80 cookie A check
       server webB 192.168.1.12:80 cookie B check
       server webC 192.168.1.13:80 cookie C check
       server webD 192.168.1.14:80 cookie D check
       
Description :
-------------
 - apache on LB1 will receive clients requests on port 443
 - it forwards it to haproxy bound to 127.0.0.1:8000
 - if a request does not contain a cookie, it will be forwarded to a valid
   server
 - in return, a cookie "SERVERID" will be inserted in the response holding the
   server name (eg: "A"), and a "Cache-control: private" header will be added
   so that the apache does not cache any page containing such cookie.
 - when the client comes again with the cookie "SERVERID=A", LB1 will know that
   it must be forwarded to server A. The cookie will be removed so that the
   server does not see it.
 - if server "webA" dies, the requests will be sent to another valid server
   and a cookie will be reassigned.
Notes :
-------
 - if the cookie works in "prefix" mode, there is no need to add the "nocache"
   option because it is an application cookie which will be modified, and the
   application flags will be preserved.
 - if apache 1.3 is used as a front-end before haproxy, it always disables
   HTTP keep-alive on the back-end, so there is no need for the "httpclose"
   option on haproxy.
 - configure apache to set the X-Forwarded-For header itself, and do not do
   it on haproxy if you need the application to know about the client's IP.
Flows :
-------
(apache)                           (haproxy)                         (server A)
  >-- GET /URI1 HTTP/1.0 ------------> |
               ( no cookie, haproxy forwards in load-balancing mode. )
                                       | >-- GET /URI1 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
               ( the proxy now adds the server cookie in return )
  <-- HTTP/1.0 200 OK ---------------< |
      Set-Cookie: SERVERID=A           |
      Cache-Control: private           |
  >-- GET /URI2 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
      ( the proxy sees the cookie. it forwards to server A and deletes it )
                                       | >-- GET /URI2 HTTP/1.0 ---------->
                                       | <-- HTTP/1.0 200 OK -------------<
   ( the proxy does not add the cookie in return because the client knows it )
  <-- HTTP/1.0 200 OK ---------------< |
  >-- GET /URI3 HTTP/1.0 ------------> |
      Cookie: SERVERID=A               |
                                    ( ... )
========================================
3.1. Alternate solution using Stunnel
========================================
When only SSL is required and cache is not needed, stunnel is a cheaper
solution than Apache+mod_ssl. By default, stunnel does not process HTTP and
does not add any X-Forwarded-For header, but there is a patch on the official
haproxy site to provide this feature to recent stunnel versions.
This time, stunnel will only process HTTPS and not HTTP. This means that
haproxy will get all HTTP traffic, so haproxy will have to add the
X-Forwarded-For header for HTTP traffic, but not for HTTPS traffic since
stunnel will already have done it. We will use the "except" keyword to tell
haproxy that connections from local host already have a valid header.
  192.168.1.1    192.168.1.11-192.168.1.14   192.168.1.2
 -------+-----------+-----+-----+-----+--------+----
        |           |     |     |     |       _|_db
     +--+--+      +-+-+ +-+-+ +-+-+ +-+-+    (___)
     | LB1 |      | A | | B | | C | | D |    (___)
     +-----+      +---+ +---+ +---+ +---+    (___)
     stunnel        4 cheap web servers
     haproxy 
Config on stunnel (LB1) :
-------------------------
    cert=/etc/stunnel/stunnel.pem
    setuid=stunnel
    setgid=proxy
    socket=l:TCP_NODELAY=1
    socket=r:TCP_NODELAY=1
    [https]
    accept=192.168.1.1:443
    connect=192.168.1.1:80
    xforwardedfor=yes
Config on haproxy (LB1) :
-------------------------
       
    listen 192.168.1.1:80
       mode http
       balance roundrobin
       option forwardfor except 192.168.1.1
       cookie SERVERID insert indirect nocache
       option httpchk HEAD /index.html HTTP/1.0
       server webA 192.168.1.11:80 cookie A check
       server webB 192.168.1.12:80 cookie B check
       server webC 192.168.1.13:80 cookie C check
       server webD 192.168.1.14:80 cookie D check
Description :
-------------
 - stunnel on LB1 will receive clients requests on port 443
 - it forwards them to haproxy bound to port 80
 - haproxy will receive HTTP client requests on port 80 and decrypted SSL
   requests from Stunnel on the same port.
 - stunnel will add the X-Forwarded-For header
 - haproxy will add the X-Forwarded-For header for everyone except the local
   address (stunnel).
========================================
4. Soft-stop for application maintenance
========================================
When an application is spread across several servers, the time to update all
instances increases, so the application seems jerky for a longer period.
HAproxy offers several solutions for this. Although it cannot be reconfigured
without being stopped, nor does it offer any external command, there are other
working solutions.
=========================================
4.1 Soft-stop using a file on the servers
=========================================
This trick is quite common and very simple: put a file on the server which will
be checked by the proxy. When you want to stop the server, first remove this
file. The proxy will see the server as failed, and will not send it any new
session, only the old ones if the "persist" option is used. Wait a bit then
stop the server when it does not receive anymore connections.
       
    listen 192.168.1.1:80
       mode http
       balance roundrobin
       cookie SERVERID insert indirect
       option httpchk HEAD /running HTTP/1.0
       server webA 192.168.1.11:80 cookie A check inter 2000 rise 2 fall 2
       server webB 192.168.1.12:80 cookie B check inter 2000 rise 2 fall 2
       server webC 192.168.1.13:80 cookie C check inter 2000 rise 2 fall 2
       server webD 192.168.1.14:80 cookie D check inter 2000 rise 2 fall 2
       option persist
       redispatch
       contimeout 5000
Description :
-------------
 - every 2 seconds, haproxy will try to access the file "/running" on the
   servers, and declare the server as down after 2 attempts (4 seconds).
 - only the servers which respond with a 200 or 3XX response will be used.
 - if a request does not contain a cookie, it will be forwarded to a valid
   server
 - if a request contains a cookie for a failed server, haproxy will insist
   on trying to reach the server anyway, to let the user finish what he was
   doing. ("persist" option)
 - if the server is totally stopped, the connection will fail and the proxy
   will rebalance the client to another server ("redispatch")
Usage on the web servers :
--------------------------
- to start the server :
    # /etc/init.d/httpd start
    # touch /home/httpd/www/running
- to soft-stop the server
    # rm -f /home/httpd/www/running
- to completely stop the server :
    # /etc/init.d/httpd stop
Limits
------
If the server is totally powered down, the proxy will still try to reach it
for those clients who still have a cookie referencing it, and the connection
attempt will expire after 5 seconds ("contimeout"), and only after that, the
client will be redispatched to another server. So this mode is only useful
for software updates where the server will suddenly refuse the connection
because the process is stopped. The problem is the same if the server suddenly
crashes. All of its users will be fairly perturbated.
==================================
4.2 Soft-stop using backup servers
==================================
A better solution which covers every situation is to use backup servers.
Version 1.1.30 fixed a bug which prevented a backup server from sharing
the same cookie as a standard server.
       
    listen 192.168.1.1:80
       mode http
       balance roundrobin
       redispatch
       cookie SERVERID insert indirect
       option httpchk HEAD / HTTP/1.0
       server webA 192.168.1.11:80 cookie A check port 81 inter 2000
       server webB 192.168.1.12:80 cookie B check port 81 inter 2000
       server webC 192.168.1.13:80 cookie C check port 81 inter 2000
       server webD 192.168.1.14:80 cookie D check port 81 inter 2000
       server bkpA 192.168.1.11:80 cookie A check port 80 inter 2000 backup
       server bkpB 192.168.1.12:80 cookie B check port 80 inter 2000 backup
       server bkpC 192.168.1.13:80 cookie C check port 80 inter 2000 backup
       server bkpD 192.168.1.14:80 cookie D check port 80 inter 2000 backup
Description
-----------
Four servers webA..D are checked on their port 81 every 2 seconds. The same
servers named bkpA..D are checked on the port 80, and share the exact same
cookies. Those servers will only be used when no other server is available
for the same cookie.
When the web servers are started, only the backup servers are seen as
available. On the web servers, you need to redirect port 81 to local
port 80, either with a local proxy (eg: a simple haproxy tcp instance),
or with iptables (linux) or pf (openbsd). This is because we want the
real web server to reply on this port, and not a fake one. Eg, with
iptables :
  # /etc/init.d/httpd start
  # iptables -t nat -A PREROUTING -p tcp --dport 81 -j REDIRECT --to-port 80
A few seconds later, the standard server is seen up and haproxy starts to send
it new requests on its real port 80 (only new users with no cookie, of course).
If a server completely crashes (even if it does not respond at the IP level),
both the standard and backup servers will fail, so clients associated to this
server will be redispatched to other live servers and will lose their sessions.
Now if you want to enter a server into maintenance, simply stop it from
responding on port 81 so that its standard instance will be seen as failed,
but the backup will still work. Users will not notice anything since the
service is still operational :
  # iptables -t nat -D PREROUTING -p tcp --dport 81 -j REDIRECT --to-port 80
The health checks on port 81 for this server will quickly fail, and the
standard server will be seen as failed. No new session will be sent to this
server, and existing clients with a valid cookie will still reach it because
the backup server will still be up.
Now wait as long as you want for the old users to stop using the service, and
once you see that the server does not receive any traffic, simply stop it :
  # /etc/init.d/httpd stop
The associated backup server will in turn fail, and if any client still tries
to access this particular server, he will be redispatched to any other valid
server because of the "redispatch" option.
This method has an advantage : you never touch the proxy when doing server
maintenance. The people managing the servers can make them disappear smoothly.
4.2.1 Variations for operating systems without any firewall software
--------------------------------------------------------------------
The downside is that you need a redirection solution on the server just for
the health-checks. If the server OS does not support any firewall software,
this redirection can also be handled by a simple haproxy in tcp mode :
    global
        daemon
        quiet
        pidfile /var/run/haproxy-checks.pid
    listen 0.0.0.0:81
        mode tcp
        dispatch 127.0.0.1:80
        contimeout 1000
        clitimeout 10000
        srvtimeout 10000
To start the web service :
  # /etc/init.d/httpd start
  # haproxy -f /etc/haproxy/haproxy-checks.cfg
To soft-stop the service :
  # kill $(</var/run/haproxy-checks.pid)
The port 81 will stop responding and the load-balancer will notice the failure.
4.2.2 Centralizing the server management
----------------------------------------
If one finds it preferable to manage the servers from the load-balancer itself,
the port redirector can be installed on the load-balancer itself. See the
example with iptables below.
Make the servers appear as operational :
  # iptables -t nat -A OUTPUT -d 192.168.1.11 -p tcp --dport 81 -j DNAT --to-dest :80
  # iptables -t nat -A OUTPUT -d 192.168.1.12 -p tcp --dport 81 -j DNAT --to-dest :80
  # iptables -t nat -A OUTPUT -d 192.168.1.13 -p tcp --dport 81 -j DNAT --to-dest :80
  # iptables -t nat -A OUTPUT -d 192.168.1.14 -p tcp --dport 81 -j DNAT --to-dest :80
Soft stop one server :
  # iptables -t nat -D OUTPUT -d 192.168.1.12 -p tcp --dport 81 -j DNAT --to-dest :80
Another solution is to use the "COMAFILE" patch provided by Alexander Lazic,
which is available for download here :
   http://w.ods.org/tools/haproxy/contrib/
4.2.3 Notes :
-------------
  - Never, ever, start a fake service on port 81 for the health-checks, because
    a real web service failure will not be detected as long as the fake service
    runs. You must really forward the check port to the real application.
  - health-checks will be sent twice as often, once for each standard server,
    and once for each backup server. All this will be multiplicated by the
    number of processes if you use multi-process mode. You will have to ensure
    that all the checks sent to the server do not overload it.
=======================
4.3 Hot reconfiguration
=======================
There are two types of haproxy users :
  - those who can never do anything in production out of maintenance periods ;
  - those who can do anything at any time provided that the consequences are
    limited.
The first ones have no problem stopping the server to change configuration
because they got some maintenance periods during which they can break anything.
So they will even prefer doing a clean stop/start sequence to ensure everything
will work fine upon next reload. Since those have represented the majority of
haproxy uses, there has been little effort trying to improve this.
However, the second category is a bit different. They like to be able to fix an
error in a configuration file without anyone noticing. This can sometimes also
be the case for the first category because humans are not failsafe.
For this reason, a new hot reconfiguration mechanism has been introduced in
version 1.1.34. Its usage is very simple and works even in chrooted
environments with lowered privileges. The principle is very simple : upon
reception of a SIGTTOU signal, the proxy will stop listening to all the ports.
This will release the ports so that a new instance can be started. Existing
connections will not be broken at all. If the new instance fails to start,
then sending a SIGTTIN signal back to the original processes will restore
the listening ports. This is possible without any special privileges because
the sockets will not have been closed, so the bind() is still valid. Otherwise,
if the new process starts successfully, then sending a SIGUSR1 signal to the
old one ensures that it will exit as soon as its last session ends.
A hot reconfiguration script would look like this :
  # save previous state
  mv /etc/haproxy/config /etc/haproxy/config.old
  mv /var/run/haproxy.pid /var/run/haproxy.pid.old
  mv /etc/haproxy/config.new /etc/haproxy/config
  kill -TTOU $(cat /var/run/haproxy.pid.old)
  if haproxy -p /var/run/haproxy.pid -f /etc/haproxy/config; then
    echo "New instance successfully loaded, stopping previous one."
    kill -USR1 $(cat /var/run/haproxy.pid.old)
    rm -f /var/run/haproxy.pid.old
    exit 1
  else
    echo "New instance failed to start, resuming previous one."
    kill -TTIN $(cat /var/run/haproxy.pid.old)
    rm -f /var/run/haproxy.pid
    mv /var/run/haproxy.pid.old /var/run/haproxy.pid
    mv /etc/haproxy/config /etc/haproxy/config.new
    mv /etc/haproxy/config.old /etc/haproxy/config
    exit 0
  fi
After this, you can still force old connections to end by sending
a SIGTERM to the old process if it still exists :
    kill $(cat /var/run/haproxy.pid.old)
    rm -f /var/run/haproxy.pid.old
Be careful with this as in multi-process mode, some pids might already
have been reallocated to completely different processes.
==================================================
5. Multi-site load-balancing with local preference
==================================================
5.1 Description of the problem
==============================
Consider a world-wide company with sites on several continents. There are two
production sites SITE1 and SITE2 which host identical applications. There are
many offices around the world. For speed and communication cost reasons, each
office uses the nearest site by default, but can switch to the backup site in
the event of a site or application failure. There also are users on the
production sites, which use their local sites by default, but can switch to the
other site in case of a local application failure.
The main constraints are :
  - application persistence : although the application is the same on both
    sites, there is no session synchronisation between the sites. A failure
    of one server or one site can cause a user to switch to another server
    or site, but when the server or site comes back, the user must not switch
    again.
  - communication costs : inter-site communication should be reduced to the
    minimum. Specifically, in case of a local application failure, every
    office should be able to switch to the other site without continuing to
    use the default site.
5.2 Solution
============
  - Each production site will have two haproxy load-balancers in front of its
    application servers to balance the load across them and provide local HA.
    We will call them "S1L1" and "S1L2" on site 1, and "S2L1" and "S2L2" on
    site 2. These proxies will extend the application's JSESSIONID cookie to
    put the server name as a prefix.
  - Each production site will have one front-end haproxy director to provide
    the service to local users and to remote offices. It will load-balance
    across the two local load-balancers, and will use the other site's
    load-balancers as backup servers. It will insert the local site identifier
    in a SITE cookie for the local load-balancers, and the remote site
    identifier for the remote load-balancers. These front-end directors will
    be called "SD1" and "SD2" for "Site Director".
  - Each office will have one haproxy near the border gateway which will direct
    local users to their preference site by default, or to the backup site in
    the event of a previous failure. It will also analyze the SITE cookie, and
    direct the users to the site referenced in the cookie. Thus, the preferred
    site will be declared as a normal server, and the backup site will be
    declared as a backup server only, which will only be used when the primary
    site is unreachable, or when the primary site's director has forwarded
    traffic to the second site. These proxies will be called "OP1".."OPXX"
    for "Office Proxy #XX".
  
5.3 Network diagram
===================
Note : offices 1 and 2 are on the same continent as site 1, while
       office 3 is on the same continent as site 3. Each production
       site can reach the second one either through the WAN or through
       a dedicated link.
        Office1         Office2                          Office3
         users           users                            users
192.168  # # #   192.168 # # #                            # # #
.1.0/24  | | |   .2.0/24 | | |             192.168.3.0/24 | | |	  
  --+----+-+-+-   --+----+-+-+-                   ---+----+-+-+- 
    |      | .1     |      | .1                      |      | .1
    |    +-+-+      |    +-+-+                       |    +-+-+
    |    |OP1|      |    |OP2|                       |    |OP3|  ...
  ,-:-.  +---+    ,-:-.  +---+                     ,-:-.  +---+
 (  X  )         (  X  )                          (  X  )        
  `-:-'           `-:-'             ,---.          `-:-'         
  --+---------------+------+----~~~(  X  )~~~~-------+---------+-
                           |        `---'                      |    
                           |                                   |    
                 +---+   ,-:-.                       +---+   ,-:-.  
                 |SD1|  (  X  )                      |SD2|  (  X  )
   ( SITE 1 )    +-+-+   `-:-'         ( SITE 2 )    +-+-+   `-:-'
                   |.1     |                           |.1     |    
   10.1.1.0/24     |       |     ,---. 10.2.1.0/24     |       |    
        -+-+-+-+-+-+-+-----+-+--(  X  )------+-+-+-+-+-+-+-----+-+--
         | | | | |   |       |   `---'       | | | | |   |       |
      ...# # # # #   |.11    |.12         ...# # # # #   |.11    |.12
          Site 1   +-+--+  +-+--+              Site 2  +-+--+  +-+--+   
          Local    |S1L1|  |S1L2|              Local   |S2L1|  |S2L2|   
          users    +-+--+  +--+-+              users   +-+--+  +--+-+   
                     |        |	                         |        |     
   10.1.2.0/24    -+-+-+--+--++--      10.2.2.0/24    -+-+-+--+--++--   
                   |.1       |.4                       |.1       |.4
                 +-+-+     +-+-+                     +-+-+     +-+-+    
                 |W11| ~~~ |W14|                     |W21| ~~~ |W24|    
                 +---+     +---+                     +---+     +---+    
              4 application servers               4 application servers
                    on site 1                           on site 2
5.4 Description
===============
5.4.1 Local users
-----------------
 - Office 1 users connect to OP1 = 192.168.1.1
 - Office 2 users connect to OP2 = 192.168.2.1
 - Office 3 users connect to OP3 = 192.168.3.1
 - Site 1 users connect to SD1 = 10.1.1.1
 - Site 2 users connect to SD2 = 10.2.1.1
5.4.2 Office proxies
--------------------
 - Office 1 connects to site 1 by default and uses site 2 as a backup.
 - Office 2 connects to site 1 by default and uses site 2 as a backup.
 - Office 3 connects to site 2 by default and uses site 1 as a backup.
The offices check the local site's SD proxy every 30 seconds, and the
remote one every 60 seconds.
Configuration for Office Proxy OP1
----------------------------------
    listen 192.168.1.1:80
       mode http
       balance roundrobin
       redispatch
       cookie SITE
       option httpchk HEAD / HTTP/1.0
       server SD1 10.1.1.1:80 cookie SITE1 check inter 30000
       server SD2 10.2.1.1:80 cookie SITE2 check inter 60000 backup
Configuration for Office Proxy OP2
----------------------------------
    listen 192.168.2.1:80
       mode http
       balance roundrobin
       redispatch
       cookie SITE
       option httpchk HEAD / HTTP/1.0
       server SD1 10.1.1.1:80 cookie SITE1 check inter 30000
       server SD2 10.2.1.1:80 cookie SITE2 check inter 60000 backup
Configuration for Office Proxy OP3
----------------------------------
    listen 192.168.3.1:80
       mode http
       balance roundrobin
       redispatch
       cookie SITE
       option httpchk HEAD / HTTP/1.0
       server SD2 10.2.1.1:80 cookie SITE2 check inter 30000
       server SD1 10.1.1.1:80 cookie SITE1 check inter 60000 backup
5.4.3 Site directors ( SD1 and SD2 )
------------------------------------
The site directors forward traffic to the local load-balancers, and set a
cookie to identify the site. If no local load-balancer is available, or if
the local application servers are all down, it will redirect traffic to the
remote site, and report this in the SITE cookie. In order not to uselessly
load each site's WAN link, each SD will check the other site at a lower
rate. The site directors will also insert their client's address so that
the application server knows which local user or remote site accesses it.
The SITE cookie which is set by these directors will also be understood
by the office proxies. This is important because if SD1 decides to forward
traffic to site 2, it will write "SITE2" in the "SITE" cookie, and on next
request, the office proxy will automatically and directly talk to SITE2 if
it can reach it. If it cannot, it will still send the traffic to SITE1
where SD1 will in turn try to reach SITE2.
The load-balancers checks are performed on port 81. As we'll see further,
the load-balancers provide a health monitoring port 81 which reroutes to
port 80 but which allows them to tell the SD that they are going down soon
and that the SD must not use them anymore.
Configuration for SD1
---------------------
    listen 10.1.1.1:80
       mode http
       balance roundrobin
       redispatch
       cookie SITE insert indirect
       option httpchk HEAD / HTTP/1.0
       option forwardfor
       server S1L1 10.1.1.11:80 cookie SITE1 check port 81 inter 4000
       server S1L2 10.1.1.12:80 cookie SITE1 check port 81 inter 4000
       server S2L1 10.2.1.11:80 cookie SITE2 check port 81 inter 8000 backup
       server S2L2 10.2.1.12:80 cookie SITE2 check port 81 inter 8000 backup
Configuration for SD2
---------------------
    listen 10.2.1.1:80
       mode http
       balance roundrobin
       redispatch
       cookie SITE insert indirect
       option httpchk HEAD / HTTP/1.0
       option forwardfor
       server S2L1 10.2.1.11:80 cookie SITE2 check port 81 inter 4000
       server S2L2 10.2.1.12:80 cookie SITE2 check port 81 inter 4000
       server S1L1 10.1.1.11:80 cookie SITE1 check port 81 inter 8000 backup
       server S1L2 10.1.1.12:80 cookie SITE1 check port 81 inter 8000 backup
5.4.4 Local load-balancers S1L1, S1L2, S2L1, S2L2
-------------------------------------------------
Please first note that because SD1 and SD2 use the same cookie for both
servers on a same site, the second load-balancer of each site will only
receive load-balanced requests, but as soon as the SITE cookie will be
set, only the first LB will receive the requests because it will be the
first one to match the cookie.
The load-balancers will spread the load across 4 local web servers, and
use the JSESSIONID provided by the application to provide server persistence
using the new 'prefix' method. Soft-stop will also be implemented as described
in section 4 above. Moreover, these proxies will provide their own maintenance
soft-stop. Port 80 will be used for application traffic, while port 81 will
only be used for health-checks and locally rerouted to port 80. A grace time
will be specified to service on port 80, but not on port 81. This way, a soft
kill (kill -USR1) on the proxy will only kill the health-check forwarder so
that the site director knows it must not use this load-balancer anymore. But
the service will still work for 20 seconds and as long as there are established
sessions.
These proxies will also be the only ones to disable HTTP keep-alive in the
chain, because it is enough to do it at one place, and it's necessary to do
it with 'prefix' cookies.
Configuration for S1L1/S1L2
---------------------------
    listen 10.1.1.11:80 # 10.1.1.12:80 for S1L2
       grace 20000  # don't kill us until 20 seconds have elapsed
       mode http
       balance roundrobin
       cookie JSESSIONID prefix
       option httpclose
       option forwardfor
       option httpchk HEAD / HTTP/1.0
       server W11 10.1.2.1:80 cookie W11 check port 81 inter 2000
       server W12 10.1.2.2:80 cookie W12 check port 81 inter 2000
       server W13 10.1.2.3:80 cookie W13 check port 81 inter 2000
       server W14 10.1.2.4:80 cookie W14 check port 81 inter 2000
       server B11 10.1.2.1:80 cookie W11 check port 80 inter 4000 backup
       server B12 10.1.2.2:80 cookie W12 check port 80 inter 4000 backup
       server B13 10.1.2.3:80 cookie W13 check port 80 inter 4000 backup
       server B14 10.1.2.4:80 cookie W14 check port 80 inter 4000 backup
    listen 10.1.1.11:81 # 10.1.1.12:81 for S1L2
       mode tcp
       dispatch 10.1.1.11:80  # 10.1.1.12:80 for S1L2
Configuration for S2L1/S2L2
---------------------------
    listen 10.2.1.11:80 # 10.2.1.12:80 for S2L2
       grace 20000  # don't kill us until 20 seconds have elapsed
       mode http
       balance roundrobin
       cookie JSESSIONID prefix
       option httpclose
       option forwardfor
       option httpchk HEAD / HTTP/1.0
       server W21 10.2.2.1:80 cookie W21 check port 81 inter 2000
       server W22 10.2.2.2:80 cookie W22 check port 81 inter 2000
       server W23 10.2.2.3:80 cookie W23 check port 81 inter 2000
       server W24 10.2.2.4:80 cookie W24 check port 81 inter 2000
       server B21 10.2.2.1:80 cookie W21 check port 80 inter 4000 backup
       server B22 10.2.2.2:80 cookie W22 check port 80 inter 4000 backup
       server B23 10.2.2.3:80 cookie W23 check port 80 inter 4000 backup
       server B24 10.2.2.4:80 cookie W24 check port 80 inter 4000 backup
    listen 10.2.1.11:81 # 10.2.1.12:81 for S2L2
       mode tcp
       dispatch 10.2.1.11:80  # 10.2.1.12:80 for S2L2
5.5 Comments
------------
Since each site director sets a cookie identifying the site, remote office
users will have their office proxies direct them to the right site and stick
to this site as long as the user still uses the application and the site is
available. Users on production sites will be directed to the right site by the
site directors depending on the SITE cookie.
If the WAN link dies on a production site, the remote office users will not
see their site anymore, so they will redirect the traffic to the second site.
If there are dedicated inter-site links as on the diagram above, the second
SD will see the cookie and still be able to reach the original site. For
example :
Office 1 user sends the following to OP1 :
  GET / HTTP/1.0
  Cookie: SITE=SITE1; JSESSIONID=W14~123;
OP1 cannot reach site 1 because its external router is dead. So the SD1 server
is seen as dead, and OP1 will then forward the request to SD2 on site 2,
regardless of the SITE cookie.
SD2 on site 2 receives a SITE cookie containing "SITE1". Fortunately, it
can reach Site 1's load balancers S1L1 and S1L2. So it forwards the request
so S1L1 (the first one with the same cookie).
S1L1 (on site 1) finds "W14" in the JSESSIONID cookie, so it can forward the
request to the right server, and the user session will continue to work. Once
the Site 1's WAN link comes back, OP1 will see SD1 again, and will not route
through SITE 2 anymore.
However, when a new user on Office 1 connects to the application during a
site 1 failure, it does not contain any cookie. Since OP1 does not see SD1
because of the network failure, it will direct the request to SD2 on site 2,
which will by default direct the traffic to the local load-balancers, S2L1 and
S2L2. So only initial users will load the inter-site link, not the new ones.
===================
6. Source balancing
===================
Sometimes it may reveal useful to access servers from a pool of IP addresses
instead of only one or two. Some equipments (NAT firewalls, load-balancers)
are sensible to source address, and often need many sources to distribute the
load evenly amongst their internal hash buckets.
To do this, you simply have to use several times the same server with a
different source. Example :
    listen 0.0.0.0:80
       mode tcp
       balance roundrobin
       server from1to1 10.1.1.1:80 source 10.1.2.1
       server from2to1 10.1.1.1:80 source 10.1.2.2
       server from3to1 10.1.1.1:80 source 10.1.2.3
       server from4to1 10.1.1.1:80 source 10.1.2.4
       server from5to1 10.1.1.1:80 source 10.1.2.5
       server from6to1 10.1.1.1:80 source 10.1.2.6
       server from7to1 10.1.1.1:80 source 10.1.2.7
       server from8to1 10.1.1.1:80 source 10.1.2.8
=============================================
7. Managing high loads on application servers
=============================================
One of the roles often expected from a load balancer is to mitigate the load on
the servers during traffic peaks. More and more often, we see heavy frameworks
used to deliver flexible and evolutive web designs, at the cost of high loads
on the servers, or very low concurrency. Sometimes, response times are also
rather high. People developing web sites relying on such frameworks very often
look for a load balancer which is able to distribute the load in the most
evenly fashion and which will be nice with the servers.
There is a powerful feature in haproxy which achieves exactly this : request
queueing associated with concurrent connections limit.
Let's say you have an application server which supports at most 20 concurrent
requests. You have 3 servers, so you can accept up to 60 concurrent HTTP
connections, which often means 30 concurrent users in case of keep-alive (2
persistent connections per user).
Even if you disable keep-alive, if the server takes a long time to respond,
you still have a high risk of multiple users clicking at the same time and
having their requests unserved because of server saturation. To workaround
the problem, you increase the concurrent connection limit on the servers,
but their performance stalls under higher loads.
The solution is to limit the number of connections between the clients and the
servers. You set haproxy to limit the number of connections on a per-server
basis, and you let all the users you want connect to it. It will then fill all
the servers up to the configured connection limit, and will put the remaining
connections in a queue, waiting for a connection to be released on a server.
This ensures five essential principles :
  - all clients can be served whatever their number without crashing the
    servers, the only impact it that the response time can be delayed.
  - the servers can be used at full throttle without the risk of stalling,
    and fine tuning can lead to optimal performance.
  - response times can be reduced by making the servers work below the
    congestion point, effectively leading to shorter response times even
    under moderate loads.
  - no domino effect when a server goes down or starts up. Requests will be
    queued more or less, always respecting servers limits.
  - it's easy to achieve high performance even on memory-limited hardware.
    Indeed, heavy frameworks often consume huge amounts of RAM and not always
    all the CPU available. In case of wrong sizing, reducing the number of
    concurrent connections will protect against memory shortages while still
    ensuring optimal CPU usage.
Example :
---------
Haproxy is installed in front of an application servers farm. It will limit
the concurrent connections to 4 per server (one thread per CPU), thus ensuring
very fast response times.
  192.168.1.1   192.168.1.11-192.168.1.13   192.168.1.2
 -------+-------------+-----+-----+------------+----
        |             |     |     |           _|_db
     +--+--+        +-+-+ +-+-+ +-+-+        (___)
     | LB1 |        | A | | B | | C |        (___)
     +-----+        +---+ +---+ +---+        (___)
     haproxy       3 application servers
                   with heavy frameworks
Config on haproxy (LB1) :
-------------------------
       
    listen appfarm 192.168.1.1:80
       mode http
       maxconn 10000
       option httpclose
       option forwardfor
       balance roundrobin
       cookie SERVERID insert indirect
       option httpchk HEAD /index.html HTTP/1.0
       server railsA 192.168.1.11:80 cookie A maxconn 4 check
       server railsB 192.168.1.12:80 cookie B maxconn 4 check
       server railsC 192.168.1.13:80 cookie C maxconn 4 check
       contimeout 60000
Description :
-------------
The proxy listens on IP 192.168.1.1, port 80, and expects HTTP requests. It
can accept up to 10000 concurrent connections on this socket. It follows the
roundrobin algorithm to assign servers to connections as long as servers are
not saturated.
It allows up to 4 concurrent connections per server, and will queue the
requests above this value. The "contimeout" parameter is used to set the
maximum time a connection may take to establish on a server, but here it
is also used to set the maximum time a connection may stay unserved in the
queue (1 minute here).
If the servers can each process 4 requests in 10 ms on average, then at 3000
connections, response times will be delayed by at most :
   3000 / 3 servers / 4 conns * 10 ms = 2.5 seconds
Which is not that dramatic considering the huge number of users for such a low
number of servers.
When connection queues fill up and application servers are starving, response
times will grow and users might abort by clicking on the "Stop" button. It is
very undesirable to send aborted requests to servers, because they will eat
CPU cycles for nothing.
An option has been added to handle this specific case : "option abortonclose".
By specifying it, you tell haproxy that if an input channel is closed on the
client side AND the request is still waiting in the queue, then it is highly
likely that the user has stopped, so we remove the request from the queue
before it will get served.
Managing unfair response times
------------------------------
Sometimes, the application server will be very slow for some requests (eg:
login page) and faster for other requests. This may cause excessive queueing
of expectedly fast requests when all threads on the server are blocked on a
request to the database. Then the only solution is to increase the number of
concurrent connections, so that the server can handle a large average number
of slow connections with threads left to handle faster connections.
But as we have seen, increasing the number of connections on the servers can
be detrimental to performance (eg: Apache processes fighting for the accept()
lock). To improve this situation, the "minconn" parameter has been introduced.
When it is set, the maximum connection concurrency on the server will be bound
by this value, and the limit will increase with the number of clients waiting
in queue, till the clients connected to haproxy reach the proxy's maxconn, in
which case the connections per server will reach the server's maxconn. It means
that during low-to-medium loads, the minconn will be applied, and during surges
the maxconn will be applied. It ensures both optimal response times under
normal loads, and availability under very high loads.
Example :
---------
       
    listen appfarm 192.168.1.1:80
       mode http
       maxconn 10000
       option httpclose
       option abortonclose
       option forwardfor
       balance roundrobin
       # The servers will get 4 concurrent connections under low
       # loads, and 12 when there will be 10000 clients.
       server railsA 192.168.1.11:80 minconn 4 maxconn 12 check
       server railsB 192.168.1.12:80 minconn 4 maxconn 12 check
       server railsC 192.168.1.13:80 minconn 4 maxconn 12 check
       contimeout 60000