flyinweb's blog

How do I configure nginx as failover reverse proxy load balancer in front of two Apache web servers under CentOS / RHEL 5.x?

nginx is a Web and Reverse proxy server. Nginx used in front of Apache Web servers. All connections coming from the Internet addressed to one of the Web servers are routed through the nginx proxy server, which may either deal with the request itself or pass the request wholly or partially to the main web servers.

Our Sample Setup

Internet--
         |
    =============                               |---- apache1 (192.168.1.15)
    | ISP Router|                               |
    =============                               |---- apache2 (192.168.1.16)
         |                                      |
         |                                      |---- db1 (192.168.1.17)
         |      |eth0 -> 192.168.1.11 ----------/
         |-lb0==|                        /
         |      |eth1 -> 202.54.1.1 ----/
         |
         |      |eth0 -> 192.168.1.10 ----------\
         |-lb1==|                        /      |---- apache1 (192.168.1.15)
                |eth1 -> 202.54.1.1 ----/       |
                                                |---- apache2 (192.168.1.16)
                                                |
                                                |---- db1 (192.168.1.17)

Where,

• lb0 - Linux box directly connected to the Internet via eth1. This is master load balancer.
• lb1 - Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.
• 202.54.1.1 - This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
• eth0 is connected to LAN and all other backend software servers are connected via eth0.
• nginx is installed on both lb0 and lb1. It will listen on 202.54.1.1. You need to configure nginx as reverse proxy server. It will connects to Apache1 and Apache2.
• Install httpd server on Apache#1 and Apache#2 server. Configure them to listen on 192.168.1.15:80 and 192.168.1.16:80. Do not assign public IP to this box. Only activate eth0 via LAN.
• Install MySQL / Oracle / PgSQL server on Db#1. Configure db server to listen on 192.168.1.17:$db_server_port. Do not assign public IP to this box. Only activate eth0 via LAN.

In short you need the following hardware:

• 2 load balancer reverse proxy servers (250GB SATA, 2GB RAM, Single Intel P-D930 or AMD 170s with RHEL 64 bit+keepalived+nginx)
• 2 Apache web servers (Software RAID-1, SCSI-73GBx2 15k disk, 6GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+Apache 2)
• 1 backup Apache web servers (Software RAID-1, SCSI-73GBx2 15k disk, 6GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+Apache 2)
• 1 database server (RAID-10, SCSI-73GBx4 15k disk, 16GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+MySQL 5)
• 1 Caching server (RAID-1, SCSI-73GBx2 15k disk, 8GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit)
• 1 offsite backup server (RAID-6, 1TB SATAx4, 4GB RAM, Single Intel/AMD CPU with RHEL 64bit)
• Slave database, storage, pop3 and SMTP server as per requirements.
• Internet uplink 100Mbps+ or as per requirements.

Remove Unwanted Software From lb0 and lb1

Type the following commands:
# yum -y groupremove "X Window System"
# x=$(yum list installed | egrep -i 'php|httpd|mysql|bind|dhclient|tftp|inetd|xinetd|ypserv|telnet-server|rsh-server|vsftpd|tcsh' | awk '{ print $1}')
# yum -y remove $x
# yum -y install bind-utils sysstat openssl-devel.x86_64 pcre-devel.x86_64 openssl097a.x86_64
# /usr/sbin/authconfig --passalgo=sha512 --update
# passwd root
The above will remove X windows and other unwanted software from both lb0 and lb1.

Install Nginx On Both lb0 and lb1

Type the following commands to download nginx, enter:
# cd /opt
# wget http://sysoev.ru/nginx/nginx-0.8.33.tar.gz
Untar nginx, enter:
# tar -zxvf nginx-0.8.33.tar.gz
# cd nginx-0.8.33
Configure nginx for 64 bit RHEL / CentOS Linux:
# ./configure --without-http_autoindex_module --without-http_ssi_module --without-http_userid_module --without-http_auth_basic_module --without-http_geo_module --without-http_fastcgi_module --without-http_empty_gif_module --with-openssl=/lib64

Sample outputs:

....
  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
...

Install the same:
# make
# make install

Create nginx User Account

Type the following commands to create a user account:
# useradd -s /sbin/nologin -d /usr/local/nginx/html -M nginx
# passwd -l nginx

Configure nginx As Reverse Proxy Load Balancer On Both lb0 and lb1

Edit /usr/local/nginx/conf/nginx.conf, enter:
# vi /usr/local/nginx/conf/nginx.conf
Update it as follows:

 
pid               logs/nginx.pid;
user              nginx nginx;
worker_processes  10;
 
events {
    worker_connections  1024;}
 
http {
  default_type       application/octet-stream;
 
 ## Common options ##
 include options.conf;
 
 ## Proxy settings ##
 include proxy.conf;
 
 ## lb domains ##
 include nixcraft.in.conf;}

Edit /usr/local/nginx/conf/options.conf, enter:
# vi /usr/local/nginx/conf/options.conf
Update it as follows:

 
 ## Size Limits
  client_body_buffer_size     128K;
  client_header_buffer_size   1M;
  client_max_body_size          1M;
  large_client_header_buffers 8 8k;
 
 ## Timeouts
  client_body_timeout   60;
  client_header_timeout 60;
  expires               24h;
  keepalive_timeout     6060;
  send_timeout          60;
 
 ## General Options
  ignore_invalid_headers   on;
  keepalive_requests      100;
  limit_zone gulag $binary_remote_addr 5m;
  recursive_error_pages    on;
  sendfile                 on;
  server_name_in_redirect off;
  server_tokens           off;
 
 ## TCP options
  tcp_nodelay on;
  tcp_nopush  on;
 
 ## Compression
  gzip              on;
  gzip_buffers      16 8k;
  gzip_comp_level   6;
  gzip_http_version 1.0;
  gzip_min_length   0;
  gzip_types        text/plain text/css image/x-icon application/x-perl application/x-httpd-cgi;
  gzip_vary         on;
 
 ## Log Format
  log_format  main  '$remote_addr $host $remote_user [$time_local]"$request" '
                    '$status $body_bytes_sent "$http_referer""$http_user_agent" '
                    '"$gzip_ratio"';

Edit /usr/local/nginx/conf/proxy.conf, enter:

 
 ## Proxy caching options
  proxy_buffering           on;
  proxy_cache_min_uses       3;
  proxy_cache_path          /usr/local/nginx/proxy_temp/ levels=1:2 keys_zone=cache:10m inactive=10m max_size=1000M;
  proxy_cache_valid         any 10m;
  proxy_ignore_client_abort off;
  proxy_intercept_errors    on;
  proxy_next_upstream       error timeout invalid_header;
  proxy_redirect            off;
  proxy_set_header          X-Forwarded-For $remote_addr;
  proxy_connect_timeout     60;
  proxy_send_timeout        60;
  proxy_read_timeout        60;

Edit /usr/local/nginx/conf/nixcraft.in.conf, enter:

 
## Connect to backend servers via LAN ##
## Reverse Proxy Load Balancer Logic ##
upstream nixcraft  {
      server 192.168.1.15weight=10 max_fails=3 fail_timeout=30s;
      server 192.168.1.16weight=10 max_fails=3 fail_timeout=30s;
      # only comes alive when above two fails
      server 192.168.1.23weight=1 backup;}
 
server {
      access_log  logs/access.log main;
      error_log   logs/error.log;
      index       index.html;
      root        /usr/local/nginx/html;
      server_name nixcraft.in www.nixcraft.in subdomain.nixcraft.in;
 
     ## Only requests to our Host are allowed
      if ($host !~ ^(nixcraft.in|www.nixcraft.in|subdomain.nixcraft.in)$ ){
         return 444;}
 
     ## redirect www to nowww
     # if ($host = 'www.nixcraft.in' ){
     #    rewrite  ^/(.*)$  http://nixcraft.in/$1  permanent;
     # }
 
     ## Only allow these request methods
     if ($request_method !~ ^(GET|HEAD|POST)$ ){
         return 444;}
 
     ## PROXY - Web
      location / {
        proxy_pass  http://nixcraft;
        proxy_cache            cache;
        proxy_cache_valid      200 24h;
        proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_ignore_headers   Expires Cache-Control;
 
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;}
 
     # redirect server error pages to the static page /50x.html
        error_page   500502503504  /50x.html;location = /50x.html {
            root   html;}}

Start nginx web server:
# /usr/local/nginx/sbin/nginx
# netstat -tulpn | grep :80
# echo ' /usr/local/nginx/sbin/nginx' >> /etc/rc.local
Fire a webbrowser and type domain name such as nixcraft.in:
http://nixcraft.in

References:

• nginx wiki