Redis集群（一）搭建

系统环境

# 系统
jie01@jie-ubuntu01:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal
# Docker
jie01@jie-ubuntu01:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:44:28 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:05 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
  
  # Redis
  redis:6.0.14

准备工作

jie01@jie-ubuntu01:~$ sudo groupadd docker          # 添加docker用户组
jie01@jie-ubuntu01:~$ sudo gpasswd -a $XXX docker   # 检测当前用户是否已经在docker用户组中，其中XXX为用户名，例如我的，jie01
jie01@jie-ubuntu01:~$ sudo gpasswd -a $USER docker  # 将当前用户添加至docker用户组
jie01@jie-ubuntu01:~$ newgrp docker                 # 更新docker用户组

# 配置文件存放位置
jie01@jie-ubuntu01:~$ sudo mkdir /etc/redis7000
jie01@jie-ubuntu01:~$ sudo mkdir /etc/redis7001
jie01@jie-ubuntu01:~$ sudo mkdir /etc/redis7002
jie01@jie-ubuntu01:~$ sudo mkdir /etc/redis7003
jie01@jie-ubuntu01:~$ sudo mkdir /etc/redis7004
jie01@jie-ubuntu01:~$ sudo mkdir /etc/redis7005

# 下载redis.conf并修改配置文件, 如果curl下载不了就手动下载
jie01@jie-ubuntu01:~$ sudo curl -o /etc/redis7000/redis.conf https://raw.githubusercontent.com/redis/redis/6.0/redis.conf
jie01@jie-ubuntu01:~$ sudo cp /etc/redis7000/redis.conf /etc/redis7001/redis.conf
jie01@jie-ubuntu01:~$ sudo cp /etc/redis7000/redis.conf /etc/redis7002/redis.conf
jie01@jie-ubuntu01:~$ sudo cp /etc/redis7000/redis.conf /etc/redis7003/
jie01@jie-ubuntu01:~$ sudo cp /etc/redis7000/redis.conf /etc/redis7004/
jie01@jie-ubuntu01:~$ sudo cp /etc/redis7000/redis.conf /etc/redis7005/

# 配置文件有几处需要修改的地方
cluster-enabled no : cluster-enabled yes # 启用集群支持，默认是单机模式运行Redis
bind 127.0.0.1     : # bind 127.0.0.1    # 允许远程连接
protected-mode no  : protected-mode yes  # 允许远程连接

# 启动六台redis组建集群，三台master，三台slave
jie01@jie-ubuntu01:~$ docker run -p 7000:6379 -v /etc/redis7000:/usr/local/etc/redis --name redis7000 -d redis:6.0.14 redis-server /usr/local/etc/redis/redis.conf
jie01@jie-ubuntu01:~$ docker run -p 7001:6379 -v /etc/redis7001:/usr/local/etc/redis --name redis7001 -d redis:6.0.14 redis-server /usr/local/etc/redis/redis.conf
jie01@jie-ubuntu01:~$ docker run -p 7002:6379 -v /etc/redis7002:/usr/local/etc/redis --name redis7002 -d redis:6.0.14 redis-server /usr/local/etc/redis/redis.conf
jie01@jie-ubuntu01:~$ docker run -p 7003:6379 -v /etc/redis7003:/usr/local/etc/redis --name redis7003 -d redis:6.0.14 redis-server /usr/local/etc/redis/redis.conf
jie01@jie-ubuntu01:~$ docker run -p 7004:6379 -v /etc/redis7004:/usr/local/etc/redis --name redis7004 -d redis:6.0.14 redis-server /usr/local/etc/redis/redis.conf
jie01@jie-ubuntu01:~$ docker run -p 7005:6379 -v /etc/redis7005:/usr/local/etc/redis --name redis7005 -d redis:6.0.14 redis-server /usr/local/etc/redis/redis.conf

# 查看容器列表
jie01@jie-ubuntu01:~$ docker container ls 
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                                       NAMES
74a45e5c26a5   redis:6.0.14   "docker-entrypoint.s…"   11 seconds ago   Up 10 seconds   0.0.0.0:7005->6379/tcp, :::7005->6379/tcp   redis7005
bca612ea9a9e   redis:6.0.14   "docker-entrypoint.s…"   17 seconds ago   Up 16 seconds   0.0.0.0:7004->6379/tcp, :::7004->6379/tcp   redis7004
33bee89736b4   redis:6.0.14   "docker-entrypoint.s…"   23 seconds ago   Up 23 seconds   0.0.0.0:7003->6379/tcp, :::7003->6379/tcp   redis7003
53789be02103   redis:6.0.14   "docker-entrypoint.s…"   29 seconds ago   Up 29 seconds   0.0.0.0:7002->6379/tcp, :::7002->6379/tcp   redis7002
b17dd5c32a58   redis:6.0.14   "docker-entrypoint.s…"   36 seconds ago   Up 36 seconds   0.0.0.0:7001->6379/tcp, :::7001->6379/tcp   redis7001
136b580f9859   redis:6.0.14   "docker-entrypoint.s…"   44 seconds ago   Up 43 seconds   0.0.0.0:7000->6379/tcp, :::7000->6379/tcp   redis7000

创建集群，容器的IP请使用docker inspect 容器名查看

# 进入node7000
jie01@jie-ubuntu01:~$ docker exec -it redis7000 /bin/bash
root@136b580f9859:/data# redis-cli -c -h 172.17.0.2 -p 6379
172.17.0.2:6379> CLUSTER NODES
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 :6379@16379 myself,master - 0 0 0 connected
172.17.0.2:6379> 

# 添加节点到集群中
172.17.0.2:6379> CLUSTER MEET 172.17.0.3 6379
OK
172.17.0.2:6379> CLUSTER MEET 172.17.0.4 6379
OK
172.17.0.2:6379> CLUSTER MEET 172.17.0.5 6379
OK
172.17.0.2:6379> CLUSTER MEET 172.17.0.6 6379
OK
172.17.0.2:6379> CLUSTER MEET 172.17.0.7 6379
OK
172.17.0.2:6379> CLUSTER NODES
d3aeb1b137a6034936ec839fa4077c383d43648d 172.17.0.6:6379@16379 master - 0 1647009083930 4 connected
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 172.17.0.2:6379@16379 myself,master - 0 1647009081000 1 connected
d6ff8738b645774b435418fa216b416ce9d88ac6 172.17.0.7:6379@16379 master - 0 1647009083000 0 connected
0b0e4f69456e91aa788bcf8bfa6a8fabeb286374 172.17.0.4:6379@16379 master - 0 1647009082922 2 connected
b7ef1389777f9ab43041f9f2b05525dd8b8d5699 172.17.0.3:6379@16379 master - 0 1647009083000 3 connected
d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 172.17.0.5:6379@16379 master - 0 1647009081914 5 connected
172.17.0.2:6379>

查看集群状态

Redis集群通过分片的方式来保存数据库中的键值对：集群的整个数据库被分为16384个槽（slot），数据库中的每个键都属于16384个槽的其中一个，集群中的每个节点可以处理0个或最多16384个槽。

当数据库中的16384个槽都有节点在处理时，集群处于上线状态（ok）；相反地，如果数据库中有任何一个槽没有得到处理，那么集群处于下线状态（fail）。

172.17.0.2:6379> CLUSTER INFO
cluster_state:fail # 集群还处于下线状态
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:100
cluster_stats_messages_pong_sent:108
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:213
cluster_stats_messages_ping_received:108
cluster_stats_messages_pong_received:105
cluster_stats_messages_received:213

设置从节点

# 172.17.0.2 设置为 172.17.0.5 的从节点
172.17.0.2:6379> CLUSTER REPLICATE d6a7a815e8e6abd2c7142535cdc55ad7dea2581f
OK
172.17.0.2:6379> CLUSTER NODES
d3aeb1b137a6034936ec839fa4077c383d43648d 172.17.0.6:6379@16379 master - 0 1647009546000 4 connected
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 172.17.0.2:6379@16379 myself,slave d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 0 1647009548000 5 connected
d6ff8738b645774b435418fa216b416ce9d88ac6 172.17.0.7:6379@16379 master - 0 1647009548000 0 connected
0b0e4f69456e91aa788bcf8bfa6a8fabeb286374 172.17.0.4:6379@16379 master - 0 1647009549954 2 connected
b7ef1389777f9ab43041f9f2b05525dd8b8d5699 172.17.0.3:6379@16379 master - 0 1647009548944 3 connected
d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 172.17.0.5:6379@16379 master - 0 1647009546925 5 connected
172.17.0.2:6379> exit
# 172.17.0.3 设置为 172.17.0.6 的从节点
root@136b580f9859:/data# redis-cli -c -h 172.17.0.3 -p 6379
172.17.0.3:6379> CLUSTER REPLICATE d3aeb1b137a6034936ec839fa4077c383d43648d
OK
172.17.0.3:6379> CLUSTER NODES
0b0e4f69456e91aa788bcf8bfa6a8fabeb286374 172.17.0.4:6379@16379 master - 0 1647009665270 2 connected
b7ef1389777f9ab43041f9f2b05525dd8b8d5699 172.17.0.3:6379@16379 myself,slave d3aeb1b137a6034936ec839fa4077c383d43648d 0 1647009664000 4 connected
d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 172.17.0.5:6379@16379 master - 0 1647009662245 5 connected
d3aeb1b137a6034936ec839fa4077c383d43648d 172.17.0.6:6379@16379 master - 0 1647009663000 4 connected
d6ff8738b645774b435418fa216b416ce9d88ac6 172.17.0.7:6379@16379 master - 0 1647009664262 0 connected
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 172.17.0.2:6379@16379 slave d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 0 1647009665000 5 connected
172.17.0.3:6379> exit
# 172.17.0.4 设置为 172.17.0.7 的从节点
root@136b580f9859:/data# redis-cli -c -h 172.17.0.4 -p 6379
172.17.0.4:6379> CLUSTER REPLICATE d6ff8738b645774b435418fa216b416ce9d88ac6
OK
172.17.0.4:6379> CLUSTER NODES
0b0e4f69456e91aa788bcf8bfa6a8fabeb286374 172.17.0.4:6379@16379 myself,slave d6ff8738b645774b435418fa216b416ce9d88ac6 0 1647009712000 0 connected
b7ef1389777f9ab43041f9f2b05525dd8b8d5699 172.17.0.3:6379@16379 slave d3aeb1b137a6034936ec839fa4077c383d43648d 0 1647009709000 4 connected
d3aeb1b137a6034936ec839fa4077c383d43648d 172.17.0.6:6379@16379 master - 0 1647009712369 4 connected
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 172.17.0.2:6379@16379 slave d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 0 1647009713377 5 connected
d6ff8738b645774b435418fa216b416ce9d88ac6 172.17.0.7:6379@16379 master - 0 1647009711360 0 connected
d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 172.17.0.5:6379@16379 master - 0 1647009710353 5 connected
172.17.0.4:6379>

槽指派

# 由Redis退回到容器
root@74ad6e4fafc4:/data# redis-cli -h 172.17.0.5 -p 6379 cluster addslots {0..5000}
OK
root@74ad6e4fafc4:/data# redis-cli -h 172.17.0.6 -p 6379 cluster addslots {5001..10000}
OK
root@74ad6e4fafc4:/data# redis-cli -h 172.17.0.7 -p 6379 cluster addslots {10001..16383}
OK

# 进入Redis查看
root@136b580f9859:/data# redis-cli -c -h 172.17.0.2 -p 6379
172.17.0.2:6379> CLUSTER NODES
d3aeb1b137a6034936ec839fa4077c383d43648d 172.17.0.6:6379@16379 master - 0 1647009874000 4 connected 5001-10000
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 172.17.0.2:6379@16379 myself,slave d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 0 1647009875000 5 connected
d6ff8738b645774b435418fa216b416ce9d88ac6 172.17.0.7:6379@16379 master - 0 1647009876763 0 connected 10001-16383
0b0e4f69456e91aa788bcf8bfa6a8fabeb286374 172.17.0.4:6379@16379 slave d6ff8738b645774b435418fa216b416ce9d88ac6 0 1647009875000 0 connected
b7ef1389777f9ab43041f9f2b05525dd8b8d5699 172.17.0.3:6379@16379 slave d3aeb1b137a6034936ec839fa4077c383d43648d 0 1647009875754 4 connected
d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 172.17.0.5:6379@16379 master - 0 1647009875000 5 connected 0-5000

172.17.0.2:6379> CLUSTER INFO
cluster_state:ok # 集群已处于上线状态
cluster_slots_assigned:16384 # 所有的槽都被指派完毕
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:5
cluster_stats_messages_ping_sent:852
cluster_stats_messages_pong_sent:851
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:1708
cluster_stats_messages_ping_received:851
cluster_stats_messages_pong_received:857
cluster_stats_messages_received:1708

MOVED

172.17.0.2:6379> SET date "2020-01-01"
-> Redirected to slot [2022] located at 172.17.0.5:6379 # date 所在的槽2022由节点172.17.0.5:6379负责处理，自动转向至172.17.0.5:6379
OK
172.17.0.5:6379> GET date
"2020-01-01"
172.17.0.5:6379> SET msg "Hello Redis!"
-> Redirected to slot [6257] located at 172.17.0.6:6379
OK
172.17.0.6:6379> GET msg
"Hello Redis!"
172.17.0.6:6379>

计算给定的键属于哪个槽的算法

def solt_num(key):
		return CRC16(key) & 16383

其中CRC16(key)语句用于计算键key的CRC-16校验和，而 & 16383语句则用于计算出一个介于0至16383之间的整数作为key的槽号

可以使用CLUSTER KEYSLOT 命令查看一个给定的键属于哪个槽：

172.17.0.6:6379> CLUSTER KEYSLOT msg
(integer) 6257
172.17.0.6:6379> CLUSTER KEYSLOT date
(integer) 2022
172.17.0.6:6379> CLUSTER KEYSLOT apple
(integer) 7092
172.17.0.6:6379> CLUSTER KEYSLOT username
(integer) 14315
172.17.0.6:6379> CLUSTER KEYSLOT phone
(integer) 8939
172.17.0.6:6379>

重新分片

 # 当前槽指派信息
172.17.0.6:6379> CLUSTER SLOTS
1) 1) (integer) 10001
   2) (integer) 16383
   3) 1) "172.17.0.7"
      2) (integer) 6379
      3) "d6ff8738b645774b435418fa216b416ce9d88ac6"
   4) 1) "172.17.0.4"
      2) (integer) 6379
      3) "0b0e4f69456e91aa788bcf8bfa6a8fabeb286374"
2) 1) (integer) 5001
   2) (integer) 10000
   3) 1) "172.17.0.6"
      2) (integer) 6379
      3) "d3aeb1b137a6034936ec839fa4077c383d43648d"
   4) 1) "172.17.0.3"
      2) (integer) 6379
      3) "b7ef1389777f9ab43041f9f2b05525dd8b8d5699"
3) 1) (integer) 0
   2) (integer) 5000
   3) 1) "172.17.0.5"
      2) (integer) 6379
      3) "d6a7a815e8e6abd2c7142535cdc55ad7dea2581f"
   4) 1) "172.17.0.2"
      2) (integer) 6379
      3) "4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91"
      
      
# 重分片 172.17.0.2:6379 的 slot 拿出 2000 分给 172.17.0.6:6379
redis-cli --cluster reshard 172.17.0.2:6379 --cluster-from d6a7a815e8e6abd2c7142535cdc55ad7dea2581f --cluster-to d3aeb1b137a6034936ec839fa4077c383d43648d --cluster-slots 2000

# 再次查看槽指派信息
172.17.0.2:6379> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 1999
   3) 1) "172.17.0.6"
      2) (integer) 6379
      3) "d3aeb1b137a6034936ec839fa4077c383d43648d"
   4) 1) "172.17.0.3"
      2) (integer) 6379
      3) "b7ef1389777f9ab43041f9f2b05525dd8b8d5699"
2) 1) (integer) 5001
   2) (integer) 10000
   3) 1) "172.17.0.6"
      2) (integer) 6379
      3) "d3aeb1b137a6034936ec839fa4077c383d43648d"
   4) 1) "172.17.0.3"
      2) (integer) 6379
      3) "b7ef1389777f9ab43041f9f2b05525dd8b8d5699"
3) 1) (integer) 10001
   2) (integer) 16383
   3) 1) "172.17.0.7"
      2) (integer) 6379
      3) "d6ff8738b645774b435418fa216b416ce9d88ac6"
   4) 1) "172.17.0.4"
      2) (integer) 6379
      3) "0b0e4f69456e91aa788bcf8bfa6a8fabeb286374"
4) 1) (integer) 2000
   2) (integer) 5000
   3) 1) "172.17.0.5"
      2) (integer) 6379
      3) "d6a7a815e8e6abd2c7142535cdc55ad7dea2581f"
   4) 1) "172.17.0.2"
      2) (integer) 6379
      3) "4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91"

调整主从信息

172.17.0.3:6379和172.17.0.2:6379都设置为172.17.0.5:6379的从节点，用于测试故障转移

# 172.17.0.3 设置为 172.17.0.6 的从节点
root@136b580f9859:/data# redis-cli -c -h 172.17.0.3 -p 6379
172.17.0.3:6379> CLUSTER REPLICATE d6a7a815e8e6abd2c7142535cdc55ad7dea2581f
OK
172.17.0.3:6379> CLUSTER NODES
0b0e4f69456e91aa788bcf8bfa6a8fabeb286374 172.17.0.4:6379@16379 slave d6ff8738b645774b435418fa216b416ce9d88ac6 0 1647053508132 0 connected
b7ef1389777f9ab43041f9f2b05525dd8b8d5699 172.17.0.3:6379@16379 myself,slave d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 0 1647053508000 5 connected
d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 172.17.0.5:6379@16379 master - 0 1647053507122 5 connected 2000-5000
d3aeb1b137a6034936ec839fa4077c383d43648d 172.17.0.6:6379@16379 master - 0 1647053509139 6 connected 0-1999 5001-10000
d6ff8738b645774b435418fa216b416ce9d88ac6 172.17.0.7:6379@16379 master - 0 1647053506113 0 connected 10001-16383
4f9b7eee7cb4658e8e430ac9d2d44ab8f6762b91 172.17.0.2:6379@16379 slave d6a7a815e8e6abd2c7142535cdc55ad7dea2581f 0 1647053507000 5 connected
172.17.0.3:6379> exit

备注

以上内容是根据黄健宏老师的《Redis设计与实现》一书的个人总结与实践。

Redis

Redis集群（一）搭建

系统环境

准备工作

创建集群，容器的IP请使用docker inspect 容器名查看

查看集群状态

设置从节点

槽指派

MOVED

计算给定的键属于哪个槽的算法

重新分片

调整主从信息

备注

相关

ceryx+redis实现nginx动态路由功能、动态管理upstream

Go入门笔记-20 访问redis

为毛我的redis 非保护启动修改文件有问题呢？

org.springframework.data.redis.RedisConnectionFailureException

redis集群模式

hyperf 框架 redis 安装和使用

Laravel中Redis的使用

Win10安装Redis

redis哨兵原理

Redis高可用方案：Sentinel哨兵

那一天，我被Redis主从架构支配的恐惧

Redis | 第一部分：数据结构与对象上篇《Redis设计与实现》

标签

Redis集群（一）搭建

系统环境

准备工作

创建集群，容器的IP请使用docker inspect 容器名 查看

查看集群状态

设置从节点

槽指派

MOVED

计算给定的键属于哪个槽的算法

重新分片

调整主从信息

备注

相关

创建集群，容器的IP请使用docker inspect 容器名查看