Preface
Today I'm gonna implement a consul in my environment to discover service of MySQL database and check whether the master-slave is working normally.
Introduce
Consul is a tool like zookeeper which can discover services we've registered on it.It's just a simple binary command which can run with simple configuration file(json format).
official website:
https://www.consul.io/
Framework
| Hostname |
IP |
Port |
OS Version |
MySQL Version |
MySQL Role |
Consul Version |
Consul Role |
| zlm1 |
192.168.56.100 |
3306 |
CentOS 7.0 |
5.7.21 |
master |
Consul v1.2.2 |
server |
| zlm2 |
192.168.56.101 |
3306 |
CentOS 7.0 |
5.7.21 |
slave |
Consul v1.2.2 |
client |
Procedure
Download and install consul.
1 [root@zlm1 10:12:06 /vagrant]
2 #wget https://releases.hashicorp.com/consul/1.2.2/consul_1.2.2_linux_amd64.zip
3 --2018-08-16 10:02:13-- https://releases.hashicorp.com/consul/1.2.2/consul_1.2.2_linux_amd64.zip
4 Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.229.183, 2a04:4e42:36::439
5 Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.229.183|:443... connected.
6 HTTP request sent, awaiting response... 200 OK
7 Length: 18371676 (18M) [application/zip]
8 Saving to: ‘consul_1.2.2_linux_amd64.zip’
9
10 100%[===========================================================================================================>] 18,371,676 38.8KB/s in 6m 12s
11
12 2018-08-16 10:08:29 (48.3 KB/s) - ‘consul_1.2.2_linux_amd64.zip’ saved [18371676/18371676]
13
14 [root@zlm1 10:14:12 /vagrant/consul_1.2.2_linux_amd64]
15 #ls -l
16 total 82431
17 -rwxrwxrwx 1 vagrant vagrant 84408501 Jul 30 18:04 consul //There's only a binary command in the unziped directory of zip file.
18
19 [root@zlm1 10:26:18 /vagrant/consul_1.2.2_linux_amd64]
20 #mkdir /etc/consul.d
21
22 [root@zlm1 10:26:36 /vagrant/consul_1.2.2_linux_amd64]
23 #mkdir /data/consul
24
25 [root@zlm1 10:27:44 /vagrant/consul_1.2.2_linux_amd64]
26 #cp consul ~
27
28 [root@zlm1 10:29:58 ~]
29 #cd /usr/local/bin
30
31 [root@zlm1 10:30:02 /usr/local/bin]
32 #ls -l|grep consul
33 lrwxrwxrwx 1 root root 12 Aug 16 10:29 consul -> /root/consul
34
35 [root@zlm1 10:32:08 /usr/local/bin]
36 #consul --help
37 Usage: consul [--version] [--help] []
38
39 Available commands are:
40 agent Runs a Consul agent
41 catalog Interact with the catalog
42 connect Interact with Consul Connect
43 event Fire a new event
44 exec Executes a command on Consul nodes
45 force-leave Forces a member of the cluster to enter the "left" state
46 info Provides debugging information for operators.
47 intention Interact with Connect service intentions
48 join Tell Consul agent to join cluster
49 keygen Generates a new encryption key
50 keyring Manages gossip layer encryption keys
51 kv Interact with the key-value store
52 leave Gracefully leaves the Consul cluster and shuts down
53 lock Execute a command holding a lock
54 maint Controls node or service maintenance mode
55 members Lists the members of a Consul cluster
56 monitor Stream logs from a Consul agent
57 operator Provides cluster-level tools for Consul operators
58 reload Triggers the agent to reload configuration files
59 rtt Estimates network round trip time between nodes
60 snapshot Saves, restores and inspects snapshots of Consul server state
61 validate Validate config files/directories
62 version Prints the Consul version
63 watch Watch for changes in Consul
Start agent of consul by "dev" mode.
1 [root@zlm1 10:40:43 /usr/local/bin]
2 #consul agent -dev
3 ==> Starting Consul agent...
4 ==> Consul agent running!
5 Version: 'v1.2.2'
6 Node ID: '7c839914-8a47-ab36-8920-1a9da54fc6c3'
7 Node name: 'zlm1'
8 Datacenter: 'dc1' (Segment: '')
9 Server: true (Bootstrap: false)
10 Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
11 Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
12 Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
13
14 ==> Log data will now stream in as it occurs:
15
16 2018/08/16 10:47:41 [DEBUG] agent: Using random ID "7c839914-8a47-ab36-8920-1a9da54fc6c3" as node ID
17 2018/08/16 10:47:41 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:7c839914-8a47-ab36-8920-1a9da54fc6c3 Address:127.0.0.1:8300}]
18 2018/08/16 10:47:41 [INFO] serf: EventMemberJoin: zlm1.dc1 127.0.0.1
19 2018/08/16 10:47:41 [INFO] serf: EventMemberJoin: zlm1 127.0.0.1
20 2018/08/16 10:47:41 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
21 2018/08/16 10:47:41 [INFO] consul: Adding LAN server zlm1 (Addr: tcp/127.0.0.1:8300) (DC: dc1)
22 2018/08/16 10:47:41 [INFO] consul: Handled member-join event for server "zlm1.dc1" in area "wan"
23 2018/08/16 10:47:41 [DEBUG] agent/proxy: managed Connect proxy manager started
24 2018/08/16 10:47:41 [WARN] agent/proxy: running as root, will not start managed proxies
25 2018/08/16 10:47:41 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
26 2018/08/16 10:47:41 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
27 2018/08/16 10:47:41 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
28 2018/08/16 10:47:41 [INFO] agent: started state syncer
29 2018/08/16 10:47:41 [WARN] raft: Heartbeat timeout from "" reached, starting election
30 2018/08/16 10:47:41 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
31 2018/08/16 10:47:41 [DEBUG] raft: Votes needed: 1
32 2018/08/16 10:47:41 [DEBUG] raft: Vote granted from 7c839914-8a47-ab36-8920-1a9da54fc6c3 in term 2. Tally: 1
33 2018/08/16 10:47:41 [INFO] raft: Election won. Tally: 1
34 2018/08/16 10:47:41 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
35 2018/08/16 10:47:41 [INFO] consul: cluster leadership acquired
36 2018/08/16 10:47:41 [INFO] consul: New leader elected: zlm1
37 2018/08/16 10:47:41 [INFO] connect: initialized CA with provider "consul"
38 2018/08/16 10:47:41 [DEBUG] consul: Skipping self join check for "zlm1" since the cluster is too small
39 2018/08/16 10:47:41 [INFO] consul: member 'zlm1' joined, marking health alive
40 2018/08/16 10:47:41 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
41 2018/08/16 10:47:41 [INFO] agent: Synced node info
42 2018/08/16 10:47:44 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
43 2018/08/16 10:47:44 [DEBUG] agent: Node info in sync
44 2018/08/16 10:47:44 [DEBUG] agent: Node info in sync
45 2018/08/16 10:48:41 [DEBUG] consul: Skipping self join check for "zlm1" since the cluster is too small
46 2018/08/16 10:49:13 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
47 2018/08/16 10:49:13 [DEBUG] agent: Node info in sync
48 2018/08/16 10:49:41 [DEBUG] manager: Rebalanced 1 servers, next active server is zlm1.dc1 (Addr: tcp/127.0.0.1:8300) (DC: dc1)
49 2018/08/16 10:49:41 [DEBUG] consul: Skipping self join check for "zlm1" since the cluster is too small
50 2018/08/16 10:50:41 [DEBUG] consul: Skipping self join check for "zlm1" since the cluster is too small
51 2018/08/16 10:51:06 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
52 2018/08/16 10:51:06 [DEBUG] agent: Node info in sync
53
54 //Now the consul cluser has only one node "zlm1".
55
56 [root@zlm1 11:11:17 ~]
57 #consul members
58 Node Address Status Type Build Protocol DC Segment
59 zlm1 127.0.0.1:8301 alive server 1.2.2 2 dc1
60
61 //Type "Ctrl+C" can exit consul gracefully.
62 ^C 2018/08/16 11:24:50 [INFO] agent: Caught signal: interrupt
63 2018/08/16 11:24:50 [INFO] agent: Graceful shutdown disabled. Exiting
64 2018/08/16 11:24:50 [INFO] agent: Requesting shutdown
65 2018/08/16 11:24:50 [WARN] agent: dev mode disabled persistence, killing all proxies since we can't recover them
66 2018/08/16 11:24:50 [DEBUG] agent/proxy: Stopping managed Connect proxy manager
67 2018/08/16 11:24:50 [INFO] consul: shutting down server
68 2018/08/16 11:24:50 [WARN] serf: Shutdown without a Leave
69 2018/08/16 11:24:50 [WARN] serf: Shutdown without a Leave
70 2018/08/16 11:24:50 [INFO] manager: shutting down
71 2018/08/16 11:24:50 [INFO] agent: consul server down
72 2018/08/16 11:24:50 [INFO] agent: shutdown complete
73 2018/08/16 11:24:50 [INFO] agent: Stopping DNS server 127.0.0.1:8600 (tcp)
74 2018/08/16 11:24:50 [INFO] agent: Stopping DNS server 127.0.0.1:8600 (udp)
75 2018/08/16 11:24:50 [INFO] agent: Stopping HTTP server 127.0.0.1:8500 (tcp)
76 2018/08/16 11:24:50 [INFO] agent: Waiting for endpoints to shut down
77 2018/08/16 11:24:50 [INFO] agent: Endpoints down
78 2018/08/16 11:24:50 [INFO] agent: Exit code: 1
79
80 [root@zlm1 11:24:50 /usr/local/bin]
81 #ps aux|grep consul
82 root 4245 0.0 0.0 112640 960 pts/0 R+ 11:24 0:00 grep --color=auto consul
83
84 [root@zlm1 11:24:59 /usr/local/bin]
85 #consul members
86 Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: dial tcp 127.0.0.1:8500: connect: connection refused
Prepare the server configuration file and start it again.
1 [root@zlm1 11:38:35 /etc/consul.d]
2 #cat server.json
3 {
4 "data_dir":"/data/consul",
5 "datacenter":"dc1",
6 "log_level":"INFO",
7 "server":true,
8 "bootstrap_expect":1,
9 "bind_addr":"192.168.56.100",
10 "client_addr":"192.168.56.100",
11 "ports":{
12 },
13 "ui":true,
14 "retry_join":[],
15 "retry_interval":"3s",
16 "raft_protocol":3,
17 "rejoin_after_leave":true
18 }
19
20 [root@zlm1 11:38:39 /etc/consul.d]
21 #consul agent --config-dir=/etc/consul.d/ > /data/consul/consul.log 2>&1 &
22 [1] 4320
23
24 [root@zlm1 11:39:02 /etc/consul.d]
25 #ps aux|grep consul
26 root 4320 1.3 2.0 54388 21252 pts/0 Sl 11:39 0:00 consul agent --config-dir=/etc/consul.d/
27 root 4333 0.0 0.0 112640 960 pts/0 R+ 11:39 0:00 grep --color=auto consul
28
29 [root@zlm1 11:39:08 /etc/consul.d]
30 #tail -f /data/consul/consul.log
31 2018/08/16 11:39:02 [INFO] agent: Started HTTP server on 192.168.56.100:8500 (tcp)
32 2018/08/16 11:39:02 [INFO] agent: started state syncer
33 2018/08/16 11:39:07 [WARN] raft: Heartbeat timeout from "" reached, starting election
34 2018/08/16 11:39:07 [INFO] raft: Node at 192.168.56.100:8300 [Candidate] entering Candidate state in term 2
35 2018/08/16 11:39:07 [INFO] raft: Election won. Tally: 1
36 2018/08/16 11:39:07 [INFO] raft: Node at 192.168.56.100:8300 [Leader] entering Leader state
37 2018/08/16 11:39:07 [INFO] consul: cluster leadership acquired
38 2018/08/16 11:39:07 [INFO] consul: New leader elected: zlm1
39 2018/08/16 11:39:07 [INFO] consul: member 'zlm1' joined, marking health alive
40 2018/08/16 11:39:07 [INFO] agent: Synced node info
41
42 [root@zlm1 11:51:38 ~]
43 #consul members
44 Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: dial tcp 127.0.0.1:8500: connect: connection refused
45
46 [root@zlm1 11:51:44 ~]
47 #consul members --http-addr=192.168.56.100:8500
48 Node Address Status Type Build Protocol DC Segment
49 zlm1 192.168.56.100:8301 alive server 1.2.2 2 dc1
Add a client on zlm2.
1 [root@zlm1 13:33:57 /etc/consul.d]
2 #scp /root/consul zlm2:~
3 consul 100% 80MB 40.3MB/s 00:02
4
5 [root@zlm2 13:35:33 ~]
6 #mkdir /etc/consul.d
7
8 [root@zlm2 13:36:37 ~]
9 #mkdir /data/consul
10
11 [root@zlm2 13:37:11 ~]
12 #cd /etc/consul.d/
13
14 [root@zlm2 13:37:46 /etc/consul.d]
15 #ls -l
16 total 4
17 -rw-r--r-- 1 root root 241 Aug 16 13:35 client.json
18
19 [root@zlm2 13:37:47 /etc/consul.d]
20 #cat client.json
21 {
22 "data_dir": "/data/consul",
23 "enable_script_checks": true,
24 "bind_addr": "192.168.56.101",
25 "retry_join": ["192.168.56.100"],
26 "retry_interval": "30s",
27 "rejoin_after_leave": true,
28 "start_join": ["192.168.56.100"]
29 }
30
31 [root@zlm2 13:43:42 /etc/consul.d]
32 #consul agent -client 192.168.56.101 -bind 192.168.56.101 --config-dir=/etc/consul.d
33 ==> Starting Consul agent...
34 ==> Joining cluster...
35 Join completed. Synced with 1 initial agents
36 ==> Consul agent running!
37 Version: 'v1.2.2'
38 Node ID: 'a69eae21-4e31-7edf-1f1a-3ec285a8fb3b'
39 Node name: 'zlm2'
40 Datacenter: 'dc1' (Segment: '')
41 Server: false (Bootstrap: false)
42 Client Addr: [192.168.56.101] (HTTP: 8500, HTTPS: -1, DNS: 8600)
43 Cluster Addr: 192.168.56.101 (LAN: 8301, WAN: 8302)
44 Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
45
46 ==> Log data will now stream in as it occurs:
47
48 2018/08/16 13:43:48 [INFO] serf: EventMemberJoin: zlm2 192.168.56.101
49 2018/08/16 13:43:48 [INFO] agent: Started DNS server 192.168.56.101:8600 (udp)
50 2018/08/16 13:43:48 [WARN] agent/proxy: running as root, will not start managed proxies
51 2018/08/16 13:43:48 [INFO] agent: Started DNS server 192.168.56.101:8600 (tcp)
52 2018/08/16 13:43:48 [INFO] agent: Started HTTP server on 192.168.56.101:8500 (tcp)
53 2018/08/16 13:43:48 [INFO] agent: (LAN) joining: [192.168.56.100]
54 2018/08/16 13:43:48 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os packet scaleway softlayer triton vsphere
55 2018/08/16 13:43:48 [INFO] agent: Joining LAN cluster...
56 2018/08/16 13:43:48 [INFO] agent: (LAN) joining: [192.168.56.100]
57 2018/08/16 13:43:48 [INFO] serf: EventMemberJoin: zlm1 192.168.56.100
58 2018/08/16 13:43:48 [INFO] agent: (LAN) joined: 1 Err:
59 2018/08/16 13:43:48 [INFO] agent: started state syncer
60 2018/08/16 13:43:48 [INFO] consul: adding server zlm1 (Addr: tcp/192.168.56.100:8300) (DC: dc1)
61 2018/08/16 13:43:48 [INFO] agent: (LAN) joined: 1 Err:
62 2018/08/16 13:43:48 [INFO] agent: Join LAN completed. Synced with 1 initial agents
63 2018/08/16 13:43:48 [INFO] agent: Synced node info
64
65 [root@zlm2 13:45:36 /etc/consul.d]
66 #consul members --http-addr=192.168.56.100:8500
67 Node Address Status Type Build Protocol DC Segment
68 zlm1 192.168.56.100:8301 alive server 1.2.2 2 dc1
69 zlm2 192.168.56.101:8301 alive client 1.2.2 2 dc1
Add two services on node zlm2.
1 [root@zlm2 13:58:37 /etc/consul.d]
2 #cat service_master_check.json
3 {
4 "service":
5 {
6 "name": "w_db3306",
7 "tags": [
8 "Master Satus Check"
9 ],
10 "address": "192.168.56.100",
11 "port": 3306,
12 "check":
13 {
14 "args": [
15 "/data/consul/script/CheckMaster.py",
16 "3306"
17 ],
18 "interval": "15s"
19 }
20 }
21 }
22
23 [root@zlm2 13:59:42 /etc/consul.d]
24 #cat service_slave_check.json
25 {
26 "service":
27 {
28 "name": "r_db3306",
29 "tags": [
30 "Slave Satus Check"
31 ],
32 "address": "192.168.56.101",
33 "port": 3306,
34 "check":
35 {
36 "args": [
37 "/data/consul/script/CheckSlave.py",
38 "3306"
39 ],
40 "interval": "15s"
41 }
42 }
43 }
44
45 [root@zlm2 14:02:53 /etc/consul.d]
46 #ls -l
47 total 12
48 -rw-r--r-- 1 root root 241 Aug 16 13:35 client.json
49 -rw-r--r-- 1 root root 593 Aug 16 13:56 service_master_check.json
50 -rw-r--r-- 1 root root 590 Aug 16 13:57 servi_slave_check.json
Check CheckMaster.py script.
1 [root@zlm2 14:05:13 /data/consul/script]
2 #cat CheckMaster.py
3 #!/usr/bin/python
4 import sys
5 import os
6 import pymysql
7
8 port = int(sys.argv[1])
9 var={}
10 conn = pymysql.connect(host='192.168.56.100',port=port,user='zlm',passwd='zlmzlm')
11 #cur = conn.cursor(pymysql.cursor.DictCursor)
12 cur = conn.cursor()
13
14 cur.execute("show global variables like \"%read_only%\"")
15 rows = cur.fetchall()
16
17 for r in rows:
18 var[r[0]]=r[1]
19 if var['read_only']=='OFF' and var['super_read_only']=='OFF':
20 print "MySQL %d master instance." % port
21 else:
22 print "This is read only instance."
23 sys.exit(2)
24
25 sys.exit(0)
26 cur.close()
27 conn.close()
Check CheckSlave.py script.
1 [root@zlm2 14:07:11 /data/consul/script]
2 #cat CheckSlave.py
3 #!/usr/bin/python
4 import sys
5 import os
6 import pymysql
7
8 port = int(sys.argv[1])
9 var={}
10 conn = pymysql.connect(host='192.168.56.101',port=port,user='zlm',passwd='zlmzlm')
11 cur = conn.cursor
12 cur.execute("show global variables like \"%read_only%\"")
13 rows = cur.fetchall()
14
15 for r in rows:
16 var[r[0]]=r[1]
17 if var['read_only']=='OFF' and var['super_read_only']=='OFF':
18 print "MySQL %d master instance." % port
19 sys.exit(2)
20 else:
21 print "MySQL %d is read only instance." % port
22
23 cur = conn.cursor(pymysql.cursors.DictCursor)
24 cur.execute("show slave status")
25 slave_status = cur.fetchone()
26
27 if len(slave_status)<30:
28 print "Slave replication setup error.";
29 sys.exit(2)
30
31 if slave_status['Slave_IO_Running'] !='Yes' or slave_status['Slave_SQL_Running'] !='Yes':
32 print "Replication error: replication from host=%s, port=%s, io_thread=%s, sql_thread=%s, error info %s %s" % (slave_status['Master_Host'],slave_status['Master_Port'],slave_status['Slave_IO_Running'],slave_status['Slave_SQL_Running'],slave_status['Last_IO_Error'],slave_status['Last_SQL_Error'])
33 sys.exit(1)
34 print slave_status
35
36 sys.exit(0)
37 cur.close()
38 conn.close()
Restartt consul(or reload).
1 [root@zlm2 14:12:37 /etc/consul.d]
2 #consul members --http-addr=192.168.56.100:8500
3 Node Address Status Type Build Protocol DC Segment
4 zlm1 192.168.56.100:8301 alive server 1.2.2 2 dc1
5 zlm2 192.168.56.101:8301 alive client 1.2.2 2 dc1
6
7 [root@zlm2 14:13:27 /etc/consul.d]
8 #consul leave --http-addr=192.168.56.101:8500
9 Graceful leave complete
10
11 [root@zlm2 14:13:41 /etc/consul.d]
12 #consul members --http-addr=192.168.56.100:8500
13 Node Address Status Type Build Protocol DC Segment
14 zlm1 192.168.56.100:8301 alive server 1.2.2 2 dc1
15 zlm2 192.168.56.101:8301 left client 1.2.2 2 dc1
16
17 [root@zlm2 14:18:44 /etc/consul.d]
18 #consul agent --config-dir=/etc/consul.d/ -client 192.168.56.101 > /data/consul/consul.log 2>&1 &
19 [1] 5013
20
21 [root@zlm2 14:19:59 /etc/consul.d]
22 #!ps
23 ps aux|grep consul
24 root 5013 2.3 2.0 56440 21132 pts/2 Sl 14:19 0:00 consul agent --config-dir=/etc/consul.d/ -client 192.168.56.101
25 root 5026 0.0 0.0 112640 960 pts/2 R+ 14:19 0:00 grep --color=auto consul
26
27 [root@zlm2 14:20:49 /etc/consul.d]
28 #consul reload -http-addr=192.168.56.101:8500
29 Configuration reload triggered
Well,now we can login the dashbord(GUI) of consul to check MySQL working status.