Centos keepalived 双机热备从零开始搭建

阅读 (2437)
配置不难,过程中会遇到一些问题和注意事项,需要vim、nginx、keepalived等,监测nginx,如果nginx停止工作,则尝试启动nginx,实现高可用

本文为双主轮切,两个真实IP,两个虚拟IP

##虚拟主机A 真实IP
http://10.10.10.175 
##虚拟主机B 真实IP
http://10.10.10.176
##虚拟主机A 虚拟IP
http://10.10.10.177
##虚拟主机B 虚拟IP
http://10.10.10.178

实现功能:

访问4个IP中任意IP,均可访问网站,

如果其中一台 keepalived 停止工作,仍可轮切

如果其中一台 nginx 停止工作,执行脚本启动nginx,即使无法启动,两个虚拟IP均访问另一台主机(除停止工作的nginx真实IP无法访问,其它三个IP仍可访问)

开始搭建前需要准备:先实现主机A

1.Linux环境

测试环境:虚拟机 -> CentOS minimal 6.5 64位

IP:用桥接网络,并手动指定IP地址: 10.10.10.175

minimal 版本的CentOS 默认装了精简版的vim 可以直接用vi命令编缉文件

[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

如下图所以,BOOTPROTO=dhcp改成status,并添加固定IP网关和子网掩码

配置DNS

vi /etc/resolv.conf

如下图,值可以改成你自己的DNS,当前用的虚拟机测试环境,配置和实体主机相同的DNS即可


 

其它相关命令:

# 启动网卡
ifup eth0

# 禁用网卡
ifdown eth0

# 重启网络
service network restart

# 开启网络
service network start

# 关闭网络
service network stop

ping 下内网外网是否都正常,这里不细表,具体问题不同解决方法,有问题自行再google

vim安装

网络通了后,安装vim,前面说了默认系统装了精简版的vim,也可以直接用vi命令,自行选择是否要安装vim

yum install vim-enhanced

2.关闭CentOS防火墙

#停止
service iptables stop 

#关闭自动启动
chkconfig iptables off 

3.彻底关闭SELinux

vim /etc/selinux/config

ESC :wq 保存,退出

reboot 重启系统

4.添加epel源 (Centos官方源很多软件都没有,编译安装又比较麻烦,可以安装EPEL源解决此问题

最新版本 (可在:http://mirror.centos.org/centos/ 找contos对应版本最新版),

当前测试环境为centos 6.5 64位,所以

rpm -ivh "http://mirror.centos.org/centos/6/extras/x86_64/Packages/epel-release-6-8.noarch.rpm"

5.安装wget

yum -y install wget

6.安装依赖库

yum install bzip2-devel curl-devel db4-devel libjpeg-devel libpng-devel libXpm-devel libc-client-devel libxml2-devel libmcrypt-devel pcre-devel openssl openssl-devel make gcc-c++ cmake bison-devel ncurses-devel gcc gcc-c++ kernel-devel readline-devel pcre-devel openssl-devel openssl zlib zlib-devel pcre-devel libmcrypt libmcrypt-devel

如下图安装完成

 

开始部署nginx

1.命令安装 nginx下载安装包解压进入目录配置安装

[root@localhost ~]# cd ~
[root@localhost ~]# wget http://nginx.org/download/nginx-1.6.3.tar.gz
[root@localhost ~]# tar zxvf nginx-1.6.3.tar.gz
[root@localhost ~]# cd nginx-1.6.3
[root@localhost ~]# ./configure --prefix=/usr/local/nginx
[root@localhost ~]# make && make install

2.创建nginx启动脚本

vim /etc/init.d/nginx

粘贴以下内容

#!/bin/bash
#
# Startup script for Nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /usr/local/nginx/conf/nginx.conf
# pidfile:     /usr/local/nginx/logs/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac

ESC :wq保存并退出

3.设置文件权限并设置开机自动启动

[root@localhost nginx-1.6.3]# chmod +x /etc/init.d/nginx
[root@localhost nginx-1.6.3]# chkconfig nginx on

检查一下

[root@localhost nginx-1.6.3]# chkconfig --list nginx

4.修改nginx配置文件

vim /usr/local/nginx/conf/nginx.conf

键入以下内容,内容中涉及另一台主机B,虽然还没有主机B,但可以先写上,后面克隆主机后,只需要修改这里的IP地址,即可实现反转

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  /var/www/logs/error.log  info;

#pid        logs/nginx.pid;


events {
    use epoll;
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;
    upstream web1 {
        server 127.0.0.1:8080 weight=5;
        server 10.10.10.176:8080 weight=5; #另外一台主机B真实IP
    }

    server {
        listen       80;
        server_name  localhost;
        root /var/www;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            index  index.html index.htm;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_pass http://web1;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

    }

    server {
        listen 8080;
        server_name 10.10.10.175; #当前主机A真实IP
        root /var/www;

        location / {
            index index.html index.htm;
        }

    }
}

以上配置修改了默认的网站目录

5.创建相关目录(默认网站目录因第4步已修改)

cd /var
mkdir www
cd www
mkdir logs
cp /usr/local/nginx/html/index.html /var/www

6.启动Nginx

service nginx start

相关命令

## 相关命令
service nginx start #开启nginx
service nginx stop  #停止nginx
service nginx restart #重启nginx
service nginx reload #重新加载nginx
service nginx status #nginx状态

## 相同效果
/etc/init.d/nginx start
/etc/init.d/nginx stop
/etc/init.d/nginx restart
/etc/init.d/nginx reload
/etc/init.d/nginx status 

7.尝试远程访问

做到这里,虚拟机的nginx环境已经搭建好,在真实主机访问:http://10.10.10.175 (改成你自己的测试机IP), 可以看到下图效果

8.修改nginx默认首页

vim /var/www/index.html

保存并退出,再次回到真实主机在浏览器访问10.10.10.175,可以看到页面也发生了变化

Nginx安装先到这里,后面还有双机热备需要配置另一台主机的Nginx配置,只是反转一下IP

 

安装Keepalived

1.下载keepalived,解压、配置并安装

[root@localhost ~]# cd ~
[root@localhost ~]# wget http://www.keepalived.org/software/keepalived-1.3.2.tar.gz
[root@localhost ~]# tar -zxvf keepalived-1.3.2.tar.gz
[root@localhost ~]# cd keepalived-1.3.2
[root@localhost keepalived-1.3.2]# ./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-279.el6.x86_64
[root@localhost keepalived-1.3.2]# make && make install

2.创建启动脚本

vim /etc/init.d/keepalived

键入以下内容

#!/bin/sh
#
# Startup script for the Keepalived daemon
#
# processname: keepalived
# pidfile: /var/run/keepalived.pid
# config: /etc/keepalived/keepalived.conf
# chkconfig: - 21 79
# description: Start and stop Keepalived

# Source function library
. /etc/rc.d/init.d/functions

# Source configuration file (we set KEEPALIVED_OPTIONS there)
#. /etc/sysconfig/keepalived
. /usr/local/keepalived/etc/sysconfig/keepalived
PATH="$PATH:/usr/local/keepalived/sbin"

export PAT

RETVAL=0

prog="keepalived"

start() {
    echo -n $"Starting $prog: "
    daemon keepalived ${KEEPALIVED_OPTIONS}
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
}

stop() {
    echo -n $"Stopping $prog: "
    killproc keepalived
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
}

reload() {
    echo -n $"Reloading $prog: "
    killproc keepalived -1
    RETVAL=$?
    echo
}

# See how we were called.
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    reload)
        reload
        ;;
    restart)
        stop
        start
        ;;
    condrestart)
        if [ -f /var/lock/subsys/$prog ]; then
            stop
            start
        fi
        ;;
    status)
        status keepalived
        RETVAL=$?
        ;;
    *)
        echo "Usage: $0 {start|stop|reload|restart|condrestart|status}"
        RETVAL=1
esac

exit $RETVAL

权限修改:

chmod +x /etc/init.d/keepalived

3.设置keepalived自启动,并检查

chkconfig keepalived on
chkconfig --list keepalived

如下图显示,自启动正常

4.这里重要了!!!keepalived默认会去找 /etc/keepalived目录下的keepalived.conf配置文件

创建目录,并创建文件

cd /etc
mkdir keepalived
vim /etc/keepalived/keepalived.conf

键入配置内容

! Configuration File for keepalived
global_defs{
   notification_email {
      email@test.com #填被通知的邮箱地址
   }
   notification_email_from test@qq.com # 发件人
   smtp_server smtp.qq.com
   smtp_connect_timeout 30
   router_id LVS_212
}

vrrp_script chk_nginx{
    script "/usr/local/keepalived/chk_nginx.sh"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER 
    interface eth0
    mcast_src_ip 10.10.10.175 #当前主机真实IP
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        10.10.10.177/24 #当前主机虚拟IP
    }
}

vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 52
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        10.10.10.178/24 #另一台主机的虚拟IP
    }
}

5.注意到上面的配置中有一段:vrrp_script chk_nginx,里面设置了chk_nginx.sh

我们用这个文件定时检查nginx服务的文件

vim /usr/local/keepalived/chk_nginx.sh

键入以下内容,定时查看nginx是否存在,如果不存在则启动nginx,如果启动失败,则停止keepalived

#!/bin/bash
# description:
# 定时查看nginx是否存在,如果不存在则启动nginx
# 如果启动失败,则停止keepalived
status=$(ps -C nginx --no-heading|wc -l)
if [ "${status}" = "0" ]; then
        /usr/local/nginx/sbin/nginx
        status2=$(ps -C nginx --no-heading|wc -l)
        if [ "${status2}" = "0"  ]; then
                /etc/init.d/keepalived stop
        fi
fi

修改权限:

chmod +x /usr/local/keepalived/chk_nginx.sh

6.相关命令

service keepalived start
service keepalived stop
service keepalived restart
service keepalived reload

/etc/init.d/keepalived start
/etc/init.d/keepalived stop
/etc/init.d/keepalived restart
/etc/init.d/keepalived reload

至此,第一台主机的环境已经搭好,VM虚拟机关机,完整克隆第一台主机,后面修改第二台主机的少许配置即可

另一台主机修改配置

1.修改虚拟机 mac地址和IP地址

vim /etc/sysconfig/network-scripts/ifcfg-eth0

reboot重启

2.修改keepalived.config

! Configuration File for keepalived
global_defs{
   notification_email {
      zxs@yangxun.com
   }
   notification_email_from 858785716@qq.com
   smtp_server smtp.qq.com
   smtp_connect_timeout 30
   router_id LVS_212
}

vrrp_script chk_nginx {
    script "/usr/local/keepalived/chk_nginx.sh"
    interval 2
    weight 2
}
## 注:以下内容与主机1不同,注意观察区分
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        10.10.10.177/24 #另一台主机的虚拟IP
    }
}

vrrp_instance VI_2 {
    state MASTER
    interface eth0
    mcast_src_ip 10.10.10.176 #当前主机真实IP
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        10.10.10.178/24 #当前主机真实IP
    }
}

3.修改nginx配置

vim /usr/local/nginx/conf/nginx.conf

 与主机A的配置反转一下

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  /var/www/logs/error.log  info;

#pid        logs/nginx.pid;


events {
    use epoll;
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;
    upstream web1 {
        server 127.0.0.1:8080 weight=5;
        server 10.10.10.176:8080 weight=5; #另外一台主机真实IP
    }

    server {
        listen       80;
        server_name  localhost;
        root /var/www;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            index  index.html index.htm;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_pass http://web1;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

    }

    server {
        listen 8080;
        server_name 10.10.10.175; #当前主机真实IP
        root /var/www;

        location / {
            index index.html index.htm;
        }

    }
}

4.修改默认index.html内容,用与区分不同主机

vim /var/www/index.html

把 175 改成 176

到此配置结束试试访问4个地址,查看效果

##虚拟主机A 真实IP
http://10.10.10.175 
##虚拟主机B 真实IP
http://10.10.10.176
##虚拟主机A 虚拟IP
http://10.10.10.177
##虚拟主机B 虚拟IP
http://10.10.10.178

用真实主机访问不同IP,再刷新页面,效果应该是 看到175 和176轮换

关闭其中一台主机的nginx,再访问4个地址,

service nginx stop

其中一台主机nginx挂掉

例:停掉主机B的nginx,首先会执行

/usr/local/keepalived/chk_nginx.sh

尝试启动nginx,如果启动失败,则这掉主机B的keepalived

主机A会接管主机B的虚拟IP ,所以仍正常访问

关闭其中一台主机的keepalived,再访问4个地址

service keepalived stop

两台主机中仍有一台主机的keepalived在工作,所以两台主机4个地址仍然工作正常

===========================================

其它问题排查思路

如果keepalived启动出现问题,查看系统日志

cat /var/log/message

如果nginx启动出现问题,检查错误日志

cat /var/www/logs/error.log

 

更新于:2016-12-15 01:45:11
返回顶部