实验：Keepalived + Nginx + Tomcat 搭建高可用主从模式Web服务器

实验： Keepalived + Nginx + Tomcat 搭建高可用主从模式Web服务器

1. 安装必要组件

实验环境
IP 应用 组件
10.192.15.125（ubuntu-18.04.6） nginx-master nginx，keepalived
10.192.15.126 （ubuntu-18.04.6） nginx-slaver nginx，keepalived
10.192.15.124 tomcat tomcat1占用8082端口 tomcat2占用8083端口

Tomcat安装省略...........................

两台nginx服务器相同操作

安装 gcc g++依赖库

apt-get install build-essential -y
apt-get install libtool -y

安装 pcre依赖库

sudo apt-get update
sudo apt-get install libpcre3 libpcre3-dev -y

安装zlib

apt-get install zlib1g-dev

安装openssl

apt-get install openssl

2. 安装nginx

安装nginx

apt-get install nginx -y
nginx

配置nginx

rm /etc/nginx/sites-enabled/default
touch /etc/nginx/sites-enabled/tomcat.conf
vim /etc/nginx/sites-enabled/tomcat.conf

nginx-master tomcat.conf配置

	include       /etc/nginx/mime.types;
        include /etc/nginx/conf.d/*.conf;

        #########################
        upstream 10.192.15.125 {
        server 10.192.15.124:8083 weight=1;
        server 10.192.15.124:8082 weight=1;
        }
        server {
                listen 80;
                server_name localhost;
                location / {
                proxy_pass http://10.192.15.125;
                proxy_set_header X-NGINX "NGINX-master";  #区分两台nginx请求
                }
        }

nginx-slaver tomcat.conf配置

	include       /etc/nginx/mime.types;
        include /etc/nginx/conf.d/*.conf;
        
        #########################
        upstream 10.192.15.126 {
        server 10.192.15.124:8083 weight=1;
        server 10.192.15.124:8082 weight=1;
        }
        server {
                listen 80;
                server_name localhost;
                location / {
                proxy_pass http://10.192.15.126;
                proxy_set_header X-NGINX "NGINX-slaver";
                }
        }

使新的配置生效

nginx -s reload

3. 安装keepalived

安装keepalived

apt-get insall keepalived

配置keepalived

cd /etc/keepalived/
touch  check_nginx.sh keepalived.conf

check_nginx.sh

#!/bin/bash
#代码一定注意空格，逻辑就是：如果nginx进程不存在则启动nginx,如果nginx无法启动则kill掉keepalived所有进程
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
 /etc/init.d/nginx start
 sleep 3
 if [ `ps -C nginx --no-header |wc -l`-eq 0 ];then
  killall keepalived
 fi
fi
两台nginx的keepalived.conf有所不同

配置nginx-master上 /etc/keepalived/keepalived.conf

global_defs {
 notification_email {
      //可以添加邮件提醒
 }
}
vrrp_instance VI_1 {
 state MASTER //主服务器
 interface enp0s3
 virtual_router_id 51
 mcast_src_ip 10.192.15.125  //真实ip
 priority 250
 advert_int 1

 authentication {
        auth_type PASS
        auth_pass 123456
 }
 track_script {
        chk_nginx
 }
 virtual_ipaddress {
        10.192.15.130 # 虚拟ip 主从配置一致
 }
}

配置nginx-slaver上 /etc/keepalived/keepalived.conf

global_defs {
 notification_email {
      //可以添加邮件提醒
 }
}
vrrp_instance VI_1 {
 state BACKUP //从服务器
 interface enp0s3  
 virtual_router_id 51 //虚拟router id 主从一致
 mcast_src_ip 10.192.15.126 //真实ip
 priority 240
 advert_int 1

 authentication {
        auth_type PASS
        auth_pass 123456
 }
 track_script {
        chk_nginx
 }
 virtual_ipaddress {
        10.192.15.130 #虚拟ip 主从配置一致
 }
}

tips

  1. state 主为 MASTER 从为BACKUP
  2. virtual_router_id 虚拟路由id主从需要保持一致
  3. mcast_src_ip nginx主或从服务器真实ip
  4. interface 使用ifconfig 查看2 里ip对应的网卡 我这里vbox 第一块网卡默认是enp0s3 第二块是enp0s8
  5. virtual_ipaddress{ } 这里是虚拟出来的ip，应该是同一网段未被占用的ip
  6. authentication{ } 主从通信验证的密钥 ，主从两边需要一致

启动keepalived

/etc/init.d/keepalived start

查看 keepalived log

tail -f /var/log/syslog

检查keepalived 及 nginx运行是否正常

ps -ef | grep keepalived
ps -ef | grep nginx

4. 验证高可用主备模式是否正常

正常状态

检查虚拟ip绑定

ip add
----------------
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:55:7b:bf brd ff:ff:ff:ff:ff:ff
    inet 10.192.15.125/24 brd 10.192.15.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.192.15.130/32 scope global enp0s3  ## 可以看到虚拟ip现在绑定再 主nginx的网卡上
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe55:7bbf/64 scope link
       valid_lft forever preferred_lft forever
master down机

在master上执行

killall keepalived
killall nginx
ip add
--------------------------
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:55:7b:bf brd ff:ff:ff:ff:ff:ff
    inet 10.192.15.125/24 brd 10.192.15.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe55:7bbf/64 scope link
       valid_lft forever preferred_lft forever

在slaver上执行

ip add
---------------------------------------
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:e1:1c:42 brd ff:ff:ff:ff:ff:ff
    inet 10.192.15.126/24 brd 10.192.15.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.192.15.130/32 scope global enp0s3  # 可以看到 130这个ip 帮到 126这个台机器的网卡上了
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fee1:1c42/64 scope link
       valid_lft forever preferred_lft forever

5. 非抢占模式

非抢占模式指master down机重启后不会和slaver争抢虚拟ip地址。 除非slaver也down机或者不能正常使用

master的 keepalived.conf配置

vrrp_script chk_nginx {  
 script "/etc/keepalived/check_nginx.sh" //检测nginx进程的脚本  
 interval 2  
 weight -20  
}  

global_defs {  
 notification_email {  
      //可以添加邮件提醒  
 }  
}  
vrrp_instance VI_1 {  
 state BACKUP //主服务器（非抢占模式需要配置成BACKUP）  
 interface ens33  
 virtual_router_id 51   
 mcast_src_ip 10.192.15.125  
 priority 250  
 advert_int 1  
 nopreempt //非抢占模式
 authentication {  
        auth_type PASS  
        auth_pass 123456  
 }  
 track_script {  
        chk_nginx  
 }  
 virtual_ipaddress {  
        10.192.15.130  
 }  
}

slaver的keepalived.conf配置

vrrp_script chk_nginx {  
 script "/etc/keepalived/check_nginx.sh" //检测nginx进程的脚本  
 interval 2  
 weight -20  
}  

global_defs {  
 notification_email {  
      //可以添加邮件提醒  
 }  
}  
vrrp_instance VI_1 {  
 state BACKUP //从服务器  
 interface ens33  
 virtual_router_id 51   
 mcast_src_ip 10.192.15.126  
 priority 240  
 advert_int 1  
 nopreempt //非抢占模式
 authentication {  
        auth_type PASS  
        auth_pass 123456  
 }  
 track_script {  
        chk_nginx  
 }  
 virtual_ipaddress {  
        10.192.15.130  
 }  
}

tips 非抢占模式的和抢占模式的区别

  1. state 主从服务器 都是BACKUP
  2. vrrp_instance 下增加了nopreempt项

测试结果：

? 1. 强占模式下 重起 master服务器上keepalived和nginx 虚拟ip 130 重新绑定到master服务器（125）网卡

? 2. 非抢占模式 重启 master后 虚拟ip 130仍旧绑定在slaver服务器（126）上未变

参考资料 ： https://blog.csdn.net/apei830/article/details/78685855

keepalivedNginx高可用

相关

linux环境开机自启动nginx

nginx下配置多个web服务

一步一步配置docker（tomcat+jenkins+phpmyadmin+nginx）

阿里云服务器搭建Nginx+rtmp推流服务器

nginx反向代理websocket wss或ws端口

001.MySQL高可用主从复制简介

docker安装与配置nginx详细过程

ceryx+redis实现nginx动态路由功能、动态管理upstream

nginx配置文件

k8s中的nginx-ingress如何配置路径重定向

Nginx实现负载均衡

部署Centos7下Haproxy实现Exchange反向代理负载并通过Keepalived主备负载

标签