实时同步服务
第5章 实时同步
5.1 实时同步的概念
5.1.1 实时同步什么时候使用
客户上传的一些重要的数据需要进行实时同步
5.1.2 实时同步和定时同步的相同和不同之处
- 实时同步实现的是只要存储服务器的数据发生变化,就会立马同步,保证了客户的数据不会出现丢失的情况,实时同步一般针对的是外面人传来的数据
- 定时同步最短的同步时间是一分钟,定时同步主要在一些内部人员的数据,配置文件,数据库等等一些不需要一直备份的数据,定时同步一般针对的是内部的人传来的数据
5.2 实时同步的原理
5.2.1 文字详述
- 客户将数据发到存储服务器
- 监视存储服务器发现数据发生变化(inotity)
- 将数据传送到备份服务器(rsync)
5.2.2 画图解释
.
5.3 实时同步的常用参数
5.3.1 参数介绍
- --exclude
数据监控的时候,排除那些数据信息不进行监控 - --excludei
数据进行监控的时候,以忽略大小写的方法进行数 据的排除的进行监控 - -m|--monitor 一直对指定的目录进行监控
- -r|--recursive 递归的监控目录里面数据的变化
- --format
定义监控数据输出的信息格式 - --timefmt
定义监控数据输出的时间信息 - -q|--quiet 将某些信息不进行显示输出
- -e|--event 指定监控的时间信息
5.3.2 监控事件介绍
- access file or directory contents were read(文件或者目录被读取)
- modify file or directory contents were written(文件或者目录被写入)
- attrib file or directory attributes changed(文件或者目录的属性信息发生 改变)
- close_write file or directory closed, after being opened in writeable mode(文 件或目录被关闭,在文件打开写入新的东西以后关闭的)
- close_nowrite file or directory closed, after being opened in read-only mode(文 件或目录被关闭,在文件打开没有写入新的东西以后关闭的)
- close file or directory closed, regardless of read/write mode(文件或者目 录关闭,不考虑现在是查看内容还是写入内容)
- move from file or directory opened moved_to file or directory moved to watched directory(文件或者目录被移动到监控目录中)
- moved_from file or directory moved from watched directory(文件或者目录从监 控目录中移出来)
- move file or directory moved to or from watched directory(文件或者目录, 有数据移动的操作)
- create file or directory created within watched directory(在监控目录中,有 文件创建的信息产生)
- delete file or directory deleted within watched directory(在监控目录中,有 文件删除的信息产生)
- open file or directory opened(文件或者目录打开)
5.4 实时同步的操作
5.4.1 实时同步软件的安装(在存储服务器上面安装)
[root@nfs01 ~] # yum -y reinstall inotify-tools
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
base
Dependencies Resolved
Installing : inotify-tools-3.14-9.el7.x86_64 1/1
Verifying : inotify-tools-3.14-9.el7.x86_64 1/1
Installed:
inotify-tools.x86_64 0:3.14-9.el7
Complete!
5.4.2 查看安装包对应的文件
[root@nfs01 ~] # rpm -ql inotify-tools
/usr/bin/inotifywait 监控数据信息的变化(重要)
/usr/bin/inotifywatch 统计监控的次数(了解即可)
/usr/lib64/libinotifytools.so.0
/usr/lib64/libinotifytools.so.0.4.1
/usr/share/doc/inotify-tools-3.14
/usr/share/doc/inotify-tools-3.14/AUTHORS
/usr/share/doc/inotify-tools-3.14/COPYING 其他的相关文件,目前不需要关注
/usr/share/doc/inotify-tools-3.14/ChangeLog
/usr/share/doc/inotify-tools-3.14/NEWS
/usr/share/doc/inotify-tools-3.14/README
/usr/share/man/man1/inotifywait.1.gz
/usr/share/man/man1/inotifywatch.1.gz
5.4.3 开始进行操作
5.4.3.1 不加任何参数的时候
[root@nfs01 ~] # inotifywait /data01
Setting up watches.
Watches established. 控连接成功
/data01/ OPEN,ISDIR 查看了监控目录下面的数据,产生了open打开事件的发生
You have new mail in /var/spool/mail/root
[root@nfs01 ~] #
[root@nfs01 ~] # cd /data01
[root@nfs01 data01] # ll 执行查看操作
total 0
-rw-r--r-- 1 root root 0 Oct 25 21:53 aa.txt
drwxr-xr-x 2 nfsnobody nfsnobody 6 Oct 26 12:21 r
drwxr-xr-x 2 nfsnobody nfsnobody 6 Oct 26 12:21 w
[root@nfs01 data01] #
5.4.3.2 加-m进行一直监控
[root@nfs01 ~] # inotifywait -m /data01 -m会一直进行监控
Setting up watches.
Watches established.
/data01/ OPEN,ISDIR
/data01/ CLOSE_NOWRITE,CLOSE,ISDIR
/data01/ CREATE aa.ttx
/data01/ OPEN aa.ttx
/data01/ ATTRIB aa.ttx
/data01/ CLOSE_WRITE,CLOSE aa.ttx
/data01/ DELETE aa.txt
/data01/ CREATE,ISDIR aa
[root@nfs01 data01] # ll
total 0
-rw-r--r-- 1 root root 0 Oct 25 21:53 aa.txt
drwxr-xr-x 2 nfsnobody nfsnobody 6 Oct 26 12:21 r
drwxr-xr-x 2 nfsnobody nfsnobody 6 Oct 26 12:21 w
You have new mail in /var/spool/mail/root
[root@nfs01 data01] # touch aa.ttx
[root@nfs01 data01] # rm -rf aa.txt
[root@nfs01 data01] # mkdir -p aa
[root@nfs01 data01] # mkdir -p aa/aa.txt 发现创建aa目录下面的子文件监控不成功
[root@nfs01 data01] #
5.4.3.3 加-r进行递归监控
[root@nfs01 ~] # inotifywait -mr /data01 递归进行监控
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/data01/aa/ ATTRIB,ISDIR aa.txt
/data01/aa/aa.txt/ ATTRIB,ISDIR
[root@nfs01 data01] # touch aa/aa.txt 监控成功
[root@nfs01 data01] #
5.4.3.4 加- -format和- -timefmt来进行输出信息格式的调整
[root@nfs01 ~] # inotifywait -mr /data01 --format "%T %w%f %e" --timefmt "%F %T" 配置输出监控信息的格式
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
2019-10-27 17:41:02 /data01/bb.ttx OPEN
2019-10-27 17:41:02 /data01/bb.ttx ATTRIB
2019-10-27 17:41:02 /data01/bb.ttx CLOSE_WRITE,CLOSE
[root@nfs01 data01] # touch bb.ttx 在监控目录创建bb.ttx
[root@nfs01 data01] #
5.4.3.5 指定哪些事件进行显示发生(-e)
[root@nfs01 ~] # inotifywait -mr /data01 --format "%T %w%f %e" --timefmt "%F %T" -e "close_write,move,delete,create" 指定哪些事件可以显示出来
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
2019-10-27 17:46:08 /data01/cc.txt CREATE
2019-10-27 17:46:08 /data01/cc.txt CLOSE_WRITE,CLOSE
[root@nfs01 data01] # touch cc.txt 创建一个cc.txt
[root@nfs01 data01] #
5.4.3.6 将监控信息调节到最优
[root@nfs01 ~] # inotifywait -mrq /data01 --format "%w%f" -e "create,delete,move,close_write" 最优
/data01/aa/bb
/data01/aa/cc.txt
/data01/aa/cc.txt
[root@nfs01 data01] # mkdir -p aa/bb
[root@nfs01 data01] # touch aa/cc.txt
[root@nfs01 data01] #
5.5 脚本实现实时同步
5.5.1 脚本实现修改,创建,移动,删除的操作
[root@nfs01 ~] # vim /server/scripts/inotity_backup.sh
#!/bin/bash
##author:liangyuxing
##date:20191027
##things:Real time synchronization
data="/data01"
inotifywait -mrq /data01 --format "%w%f" -e create,delete,move,close_write |\ 监控/data01目录下的数据
while read line 获取变化的数据
do
rsync -azL $data/ --delete rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 进行增量无差异传输
done
5.5.2 怎么进行服务的后台持续监控,在终端关闭的情况下
使用nohup sh /server/scripts/inotity_backup.sh &
5.5.3 利用脚本数据同步完毕后, 脚本会依旧持续运行?
现象
5.6 软件实现实时同步
5.6.1 下载serync软件包
https://github.com/wsgzao/sersync
5.6.2 二进制安装软件
5.6.2.1 开始解压软件
[root@nfs01 tools] # unzip sersync_installdir_64bit
Archive: sersync_installdir_64bit.zip
creating: sersync_installdir_64bit/sersync/
creating: sersync_installdir_64bit/sersync/bin/
inflating: sersync_installdir_64bit/sersync/bin/sersync
creating: sersync_installdir_64bit/sersync/conf/
inflating: sersync_installdir_64bit/sersync/conf/confxml.xml
creating: sersync_installdir_64bit/sersync/logs/
[root@nfs01 tools] # ll
5.6.2.2 将软件移动到/usr/local/目录下
[root@nfs01 tools] # cd /server/tools/sersync_installdir_64bit/
[root@nfs01 sersync_installdir_64bit] # ll
total 0
drwxr-xr-x 5 root root 41 Dec 23 2012 sersync
[root@nfs01 sersync_installdir_64bit] # mv sersync /usr/local/
[root@nfs01 tools] #
5.6.2.3 开始进行文件的配置
[root@nfs01 conf] # vim confxml.xml
1 <?xml version="1.0" encoding="ISO-8859-1"?>
2
3
4 <debug start="true"/> 开启了调试模式，在有问题的时候，可以使用
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
5.6.2.4 运行sersync服务
5.6.2.4.1 将启动程序加入系统环境变量中
exportPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/sersync/bin/:/usr/local/sersync/bin/
5.6.2.4.2 运行
[root@nfs01 conf] # sersync -h 查找sersync服务的帮助手册
set the system param
execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
_______________________________________________________
参数-d:启用守护进程模式
参数-r:在监控前，将监控目录与远程主机用rsync命令推送一遍
c参数-n: 指定开启守护线程的数量，默认为10个
参数-o:指定配置文件，默认使用confxml.xml文件
参数-m:单独启用其他模块，使用 -m refreshCDN 开启刷新CDN模块
参数-m:单独启用其他模块，使用 -m socket 开启socket模块
参数-m:单独启用其他模块，使用 -m http 开启http模块
不加-m参数，则默认执行同步程序
________________________________________________________________
[root@nfs01 conf] # sersync -dro /usr/local/sersync/conf/confxml.xml 使用dro参数来进行开启服务
set the system param
execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
option: -d run as a daemon
option: -r rsync all the local files to the remote servers before the sersync work
option: -o config xml name： /usr/local/sersync/conf/confxml.xml
daemon thread num: 10
parse xml config file
host ip : localhost host port: 8008
Open debug, you will see debug infomation
daemon start，sersync run behind the console
use rsync password-file :
user is rsync_backup
passwordfile is /etc/rsync.password
config xml parse success
please set /etc/rsyncd.conf max connections=0 Manually
sersync working thread 12 = 1(primary thread) + 1(fail retry thread) + 10(daemon sub threads)
Max threads numbers is: 22 = 12(Thread pool nums) + 10(Sub threads)
please according your cpu ，use -n param to adjust the cpu rate
------------------------------------------
rsync the directory recursivly to the remote servers once
working please wait...
execute command: cd /data01 && rsync -az -R --delete ./ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
crontab command:cd /data01 && rsync -az -R --delete ./ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
rsync: delete_file: unlink(true_finger/finger_2019-10-28.txt) failed: Permission denied (13)
cannot delete non-empty directory: true_finger
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
run the sersync:
watch path is: /data01
add watch: /data01 return wd is: 1
[root@nfs01 conf] #
5.6.2.5 测试是否备份成功
5.6.2.5.1 在备份服务器监视的目录上面创建文件
[root@nfs01 data01] # touch oldboy02.txt 创建oldboy02.txt文件
inotify wd:1 name:oldboy02.txt mask:256
inotify wd:1 name:oldboy02.txt mask:8
[root@nfs01 data01] # ll
total 0
-rw-r--r-- 1 root root 0 Oct 28 11:38 oldboy01.txt
-rw-r--r-- 1 root root 0 Oct 28 20:24 oldboy02.txt
[root@nfs01 data01] #
5.6.2.5.2 在备份服务器上面查看发现是否有这个备份文件
[root@backup ~] # ll /backup
total 0
-rw-r--r-- 1 rsync rsync 0 Oct 28 11:38 oldboy01.txt
-rw-r--r-- 1 rsync rsync 0 Oct 28 20:24 oldboy02.txt 备份目录是有这个文件的
drwxr-xr-x 2 root root 35 Oct 28 20:00 true_finger
[root@backup ~] #
5.6.2.6 关闭sersync服务
[root@nfs01 ~] # vim /usr/local/sersync/bin/shutdown_sersync.sh
#!/bin/bash
pkill -9 sersync
if [ $? -eq 0 ]
then
echo "sersync server kill success"
else
echo "sersync server kill faild,The server not start"
exit
fi