Hadoop高可用搭建-完整可用包含安装各种版本
序
这几年，容器技术不断迭代更新，为大数据提供了更广阔的发展空间。大数据已不只是互联网刚需，也是各传统行业的刚需。为了追赶技术的发展，降低内卷损耗，技术人员需要更快的学习大数据相关的知识提升自己的技术栈，为自己创造价值。
折腾了好久，终于把hadoop+hbase+yarn+flink的环境搭建起来了，有什么不明白的可以加Q184377367 注明：大数据
版本
flink-1.13.5
hadoop 3.2.2
hbase.2.3.5
zookeeper 3.4.14
修改主机名
- 修改hosts法
依次在每台服务器上设置hosts配置
vim /etc/hosts
192.168.40.134 node1
192.168.40.138 node2
192.168.40.139 node3
SSH免密码登录设置
为了方便在各个服务器上相互传输文件，还需要设置SSH免登。
- 生成公钥
cd ~
ssh-keygen -t rsa
一路回车
会在 ~.ssh 目录生成两个文件
cd .ssh
ll 查看一下
id_rsa #私钥
id_rsa.pub # 公钥
cat id_rsa.pub # 查看生成的公钥
#将当前服务器权限写入authorized_keys 其他服务器有次文件后 就可以免登到当前服务器
cat id_rsa.pub >> authorized_keys # 加入授权
chmod 600 authorized_keys # 修改文件权限，如果不修改文件权限，那么其它用户就能查看该授权
- 将sshkey文件拷贝到node2 node3
scp authorized_keys root@node2:~/.ssh/
scp authorized_keys root@node3:~/.ssh/
// 出现传输大小的字样表示成功了
// node2需要创建好 ~.ssh目录(如果没有的话)
-
测试在node1登录node2
ssh node2
所有服务器全部重启
nice 没有什么问题 -
SSH免登梳理
在node1 生成了私钥和公钥后 执行了授权指令
cat id_rsa.pub >> authorized_keys++id_rsa.pub++ 就是公钥文件，写入
到了++authorized_keys++ ,然后自己登陆自己 ssh node1 就OK了，接着把++authorized_keys++ 发给Node2 node3 然后都执行
权限指令chmod 600 authorized_keys
在node1 上就可以任意登陆NODE2 和 NODE3了。
由此可见，
1 我们需要在node1 node2 node3 分别生成公钥，
2 发给NODE1，NODE1执行授权，把node2和node3的公钥追加到++authorized_keys++
3 再发给node2 node3 ,这样三个节点 互相 任意 都可以免ssh密码登录了。
其中 2 说的 发给node1授权，这个其实无所谓在哪儿授权，只要有三个服务器的公钥授权内容就行了。
这样三台服务器都有相互的公钥授权，所以就可以任意交互登录了.
zookeeper
同步到其他服务器
cd /usr/local/apps/
# scp -r ./hadoop-3.2.2/ root@node2:/usr/local/apps/
为每个服务器配置zookeeper的工作目录
配置完成后让配置文件生效
source /etc/profile
启动zk
zkServer.sh
正常情况下的输出
ZooKeeper JMX enabled by default
Using config: /usr/local/apps/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: leader
查看ZK的选举服务
jps
QuorumPeerMain 这个就是ZK的选举服务，每个启动ZK的服务器上都会有
查看zookeeper状态
zkServer.sh status
- 问题
Error contacting service. It is probably not running.
1 ZK搭建的是高可用的话，其他服务器上的ZK不启动的话就会出现这个问题
2 关闭防火墙
hadoop配置
hdfs-site.xml配置
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
dfs.datanode.data.dir
/opt/hadoop/data/dfs/datanode
datanode本地文件存放地址
dfs.replication
3
文件副本数
dfs.namenode.name.dir
/opt/hadoop/data/dfs/namenode
namenode本地文件存放地址
dfs.permissions.enabled
false
是否开启目录权限
dfs.nameservices
ns1
提供服务的NS逻辑名称，与core-site.xml里的对应
dfs.ha.namenodes.ns1
hd1,hd2
列出该逻辑名称下的NameNode逻辑名称
dfs.namenode.rpc-address.ns1.hd1
node1:9000
指定NameNode的RPC位置
dfs.namenode.http-address.ns1.hd1
node1:50070
指定NameNode的Web Server位置
dfs.namenode.rpc-address.ns1.hd2
node2:9000
指定NameNode的RPC位置
dfs.namenode.http-address.ns1.hd2
node2:50070
指定NameNode的Web Server位置
dfs.namenode.shared.edits.dir
qjournal://node1:8485;node2:8485/ns1
指定用于HA存放edits的共享存储，通常是namenode的所在机器
dfs.journalnode.edits.dir
/opt/hadoop/data/dfs/journaldata/
journaldata服务存放文件的地址
dfs.ha.fencing.methods
sshfence
指定HA做隔离的方法，缺省是ssh，可设为shell，稍后详述
dfs.client.failover.proxy.provider.ns1
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
指定客户端用于HA切换的代理类，不同的NS可以用不同的代理类
dfs.client.failover.proxy.provider.auto-ha
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.automatic-failover.enabled
true
fs.trash.interval
2880
回收周期
dfs.datanode.balance.bandwidthPerSec
104857600
dfs.namenode.handler.count
77
dfs.blocksize
134217728
文件块的大小
dfs.datanode.max.transfer.threads
8192
相当于linux下的打开文件最大数量，文档中无此参数，当出现DataXceiver报错的时候，需要调大。默认256
dfs.datanode.du.reserved
2147483648
每个存储卷保留用作其他用途的磁盘大小
dfs.datanode.fsdataset.volume.choosing.policy
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy
存储卷选择策略
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold
2147483648
允许的卷剩余空间差值，2G
dfs.client.read.shortcircuit
true
dfs.domain.socket.path
/opt/hadoop/data/dn_socket_PORT
core-site.xml 配置
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
io.native.lib.available
true
开启本地库支持
fs.defaultFS
hdfs://ns1
默认文件服务的协议和NS逻辑名称，和hdfs-site里的对应此配置替代了1.0里的fs.default.name
hadoop.tmp.dir
/opt/hadoop/data/tmp
io.compression.codecs
org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec
相应编码的操作类
io.file.buffer.size
131072
SequenceFiles在读写中可以使用的缓存大小
ha.zookeeper.quorum
node1:2181,node2:2181,node2:2181
HA使用的zookeeper地址
ipc.client.connection.maxidletime
60000
mapreduce.output.fileoutputformat.compress.type
BLOCK
io.seqfile.compressioin.type
BLOCK
hadoop.proxyuser.root.groups
hadoop
hdfs dfsadmin –refreshSuperUserGroupsConfiguration,yarn rmadmin –refreshSuperUserGroupsConfiguration使用这两个命令不用重启就能刷新
hadoop.proxyuser.root.hosts
localhost
编辑workers文件 /usr/local/apps/hadoop-3.2.2/etc/hadoop/workers
这儿的名字和etc/hosts中一样
node1
node2
node3
start-dfs.sh stop-dfs.sh start-all.sh stop-all.sh
# 分别在这些文件的开头加上
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
etc/hadoop/hadoop-env.sh
顶部加上
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
JAVA_HOME=/usr/local/apps/jdk1.8.0_181
同步配置文件到其他服务器
在/usr/local/apps/hadoop-3.2.2/etc/hadoop目录下执行
scp -r ./hadoop-env.sh root@node2:/usr/local/apps/hadoop-3.2.2/etc/hadoop目录下执行
scp -r ./hadoop-env.sh root@node3:/usr/local/apps/hadoop-3.2.2/etc/hadoop
将整个hadoop目录分发的其他服务器（因为sbin下的目录也修改过了..）
cd /usr/local/apps/
# scp -r ./hadoop-3.2.2/ root@node2:/usr/local/apps/
注意/etc/profile 中需要协商hadoop的配置，关于这个文件的配置在本文底部
启动hdfs
初始化zk集群
hdfs zkfc -formatZK
验证是否初始化成功
输出zookeeper指令 其实是ZK的客户端
zkCli.sh
在zk客户端的命令模式下 输入
ls /
这个时候会看到 [zookeeper,hadoop-ha] 两个应用
zookeeper 为搭建zookeeper集群时创建的
hadoop-ha 为hdfs zkfc -formatZK创建的
启动zookeeper
zkServer.sh start
启动 journalnode (同步两台namenode信息)
- 在hd1上启动journalnode
hadoop-daemon.sh start journalnode - 在hd2上启动journalnode
hadoop-daemon.sh start journalnode
对hd1上的namenode进行格式化(第一次启动前执行,以后不用每次启动都执行)
hadoop namenode -format
启动hd1上的namenode
hadoop-daemon.sh start namenode
第二个namenode同步第一个namenode状态(第一次启动前执行,以后不用每次都执行)
hadoop namenode -bootstrapStandby
在hd2上启动namenode
hadoop-daemon.sh start namenode
启动ZKFC
# 在 node1和 node2分别启动ZKFC，这时两个namenode，一个变成active，一个变成standby
hadoop-daemon.sh start zkfc
# 查看HA状态
hdfs haadmin -getServiceState hd1
standby
hdfs haadmin -getServiceState hd2
active
启动namenode
- 先格式化node1上的namenode
hadoop namenode -format - 在启动Node1上的hdfs
hadoop-daemon.sh start namenode
- 第二个namenode同步第一个namenode状态(第一次启动前执行,以后不用每次都执行)
hadoop namenode -bootstrapStandby
- 在node2上执行 如果上一个步骤不执行，会出现错误！
hadoop-daemon.sh start namenode
启动datanode
# 在 标记为active的namenode节点上执行
# 注意这里是hadoop-daemons.sh而不是hadoop-daemon.sh
# 每台服务器都有datanode.
hadoop-daemons.sh start datanode
验证hdfs
- 验证web访问
http://node1:50070
在WEB界面上可以查看到每台服务器的高可用状态
- 验证集群名称
后续hbase访问时，必须要能正常访问
执行下面的命令时，需要保证集群中有namenode是active的..
hadoop fs -ls hdfs://ns1
没有输出的就表示能正常使用集群的地址
小结
第二次启动时直接用start-dfs 可以省略上面的步骤了
yarn高可用
规划
| node1 | node2 | node3 |
|---|---|---|
| resmanager | resmanager | |
| nodemanager | nodemanager | nodemanager |
nodemanager 应该是每台服务器自动就有的管理服务
yarn-site.xml
yarn.scheduler.minimum-allocation-vcores
1
单个任务可申请的最小虚拟CPU个数
yarn.scheduler.maximum-allocation-vcores
3
单个任务可申请的最大虚拟CPU个数，此参数对应yarn.nodemanager.resource.cpu-vcores，建议最大为一个物理CPU的数量
yarn.nodemanager.resource.memory-mb
1024
该节点上可分配的物理内存总量
yarn.nodemanager.resource.cpu-vcores
3
该节点上YARN可使用的虚拟CPU个数，一个物理CPU对应3个虚拟CPU
yarn.scheduler.maximum-allocation-mb
43008
单个任务可申请的最多物理内存量
yarn.resourcemanager.ha.enabled
true
是否开启yarn ha
yarn.resourcemanager.ha.automatic-failover.embedded
true
ha状态切换为自动切换
yarn.resourcemanager.ha.rm-ids
rm1,rm2
RMs的逻辑id列表
yarn.resourcemanager.hostname.rm1
node1
yarn.resourcemanager.hostname.rm2
node2
yarn.resourcemanager.zk-address
node1:2181,node2:2181,node3:2181
ha状态的存储地址
yarn.resourcemanager.address.rm1
node1:8032
ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等
yarn.resourcemanager.scheduler.address.rm1
node1:8030
ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资>源等。
yarn.resourcemanager.webapp.https.address.rm1
node1:8089
yarn.resourcemanager.webapp.address.rm1
node1:8088
ResourceManager对外web ui地址。用户可通过该地址在浏览器中查看集群各类信息。
yarn.resourcemanager.resource-tracker.address.rm1
node1:8031
ResourceManager 对NodeManager暴露的地址.。NodeManager通过该地址向RM汇报心跳，领取任务等。
yarn.resourcemanager.admin.address.rm1
node1:8033
ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等
yarn.resourcemanager.address.rm2
node2:8032
ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等
yarn.resourcemanager.scheduler.address.rm2
node2:8030
ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资>源等。
yarn.resourcemanager.webapp.https.address.rm2
node2:8089
yarn.resourcemanager.webapp.address.rm2
node2:8088
ResourceManager对外web ui地址。用户可通过该地址在浏览器中查看集群各类信息。
yarn.resourcemanager.resource-tracker.address.rm2
node2:8031
ResourceManager 对NodeManager暴露的地址.。NodeManager通过该地址向RM汇报心跳，领取任务等。
yarn.resourcemanager.admin.address.rm2
node2:8033
ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
调度器实现类
yarn.scheduler.fair.allocation.file
fair-scheduler.xml
自定义XML配置文件所在位置，该文件主要用于描述各个队列的属性，比如资源量、权重等
yarn.scheduler.fair.user-as-default-queue
true
当应用程序未指定队列名时，是否指定用户名作为应用程序所在的队列名。如果设置为false或者未设置，所有未知队列的应用程序将被提交到default队列中，默认值为true
yarn.scheduler.fair.preemption
true
是否支持抢占
yarn.scheduler.fair.sizebasedweight
false
在一个队列内部分配资源时，默认情况下，采用公平轮询的方法将资源分配各各个应用程序，而该参数则提供了外一种资源分配方式：按照应用程序资源需求数目分配资源，即需求资源数量越多，分配的资源越多。默认情况下，该参数值为false
yarn.scheduler.increment-allocation-mb
256
内存规整化单位，默认是1024，这意味着，如果一个Container请求资源是700mB，则将被调度器规整化为 (700mB / 256mb) * 256mb=768mb
yarn.scheduler.assignmultiple
true
是否启动批量分配功能。当一个节点出现大量资源时，可以一次分配完成，也可以多次分配完成。默认情况下，参数值为false
yarn.scheduler.fair.max.assign
10
如果开启批量分配功能，可指定一次分配的container数目。默认情况下，该参数值为-1，表示不限制
yarn.scheduler.fair.allow-undeclared-pools
false
如果提交的队列名不存在，Scheduler会自动创建一个该队列，默认开启
yarn.nodemanager.local-dirs
/opt/hadoop/yarn/local
中间结果存放位置，存放执行Container所需的数据如可执行程序或jar包，配置文件等和运行过程中产生的临时数据
yarn.nodemanager.log-dirs
/opt/hadoop/yarn/logs
Container运行日志存放地址（可配置多个目录）
yarn.log-aggregation-enable
true
是否启用日志聚集功能
yarn.log.server.url
node1:19888/jobhistory/logs/
yarn.nodemanager.remote-app-log-dir
/tmp/app-logs
当应用程序运行结束后，日志被转移到的HDFS目录（启用日志聚集功能时有效）
yarn.log-aggregation.retain-seconds
1209600
nodemanager上所有Container的运行日志在HDFS中的保存时间，保留半个月
yarn.nodemanager.address
0.0.0.0:9103
yarn.nodemanager.aux-services
mapreduce_shuffle
NodeManager上运行的附属服务。需配置成mapreduce_shuffle，才可运行MapReduce程序
yarn.resourcemanager.cluster-id
yarn-cluster
集群的Id
yarn.resourcemanager.recovery.enabled
true
默认值为false，也就是说resourcemanager挂了相应的正在运行的任务在rm恢复后不能重新启动
yarn.resourcemanager.store.class
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
配置RM状态信息存储方式3有两种，一种是FileSystemRMStateStore,另一种是MemoryRMStateStore，还有一种目前较为主流的是zkstore
yarn.resourcemanager.zk.state-store.address
node1:2181,node2:2181,node3:2181
当使用ZK存储时，指定在ZK上的存储地址。
yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms
5000
yarn.nodemanager.webapp.address
0.0.0.0:8042
yarn.nodemanager.localizer.address
0.0.0.0:8040
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
mapreduce.shuffle.port
23080
yarn.app.mapreduce.am.staging-dir
/user
yarn.web-proxy.address
node1:8041
yarn.nodemanager.vmem-check-enabled
false
虚拟内存检测，默认是True
yarn.nodemanager.pmem-check-enabled
false
物理内存检测，默认是True
mapred-site.xml
mapreduce.framework.name
yarn
mapreduce.application.classpath
/usr/local/apps/hadoop-3.2.2/share/hadoop/mapreduce/*:/usr/local/apps/hadoop-3.2.2/share/hadoop/mapreduce/lib/*
mapreduce.jobhistory.address
node1:10020
mapreduce.jobhistory.webapp.address
node1:19888
mapreduce.jobhistory.done-dir
/user/history/done
mapreduce.jobhistory.intermediate-done-dir
/user/history/done_intermediate
yarn.app.mapreduce.am.resource.mb
1000
表示MRAppMaster需要的总内存大小，默认是1536
yarn.app.mapreduce.am.command-opts
-Xmx800m
表示MRAppMaster需要的堆内存大小，默认是：-Xmx1024m
yarn.app.mapreduce.am.resource.cpu-vcores
1
表示MRAppMaster需要的的虚拟cpu数量，默认是：1
mapreduce.map.memory.mb
512
表示MapTask需要的总内存大小，默认是1024
mapreduce.map.java.opts
-Xmx300m
表示MapTask需要的堆内存大小，默认是-Xmx200m
mapreduce.map.cpu.vcores
1
表示MapTask需要的虚拟cpu大小，默认是1
mapreduce.reduce.memory.mb
512
表示ReduceTask需要的总内存大小，默认是1024
mapreduce.reduce.java.opts
-Xmx300m
表示ReduceTask需要的堆内存大小，默认是-Xmx200m
mapreduce.reduce.cpu.vcores
1
表示ReduceTask需要的虚拟cpu大小，默认是1
start-yarn.sh stop-yarn.sh 分别在这两个文件的开头加上
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
YARN_PROXYSERVER_USER=root
同步配置文件
# scp -r ./yarn-site.xml root@node2:/usr/local/apps/hadoop-3.2.2/etc/hadoop
# scp -r ./mapred-site.xml root@node2:/usr/local/apps/hadoop-3.2.2/etc/hadoop
验证yarn
http://node1:8088/
再次查看ZK中的存储的信息,先进入zkCli.sh 客户端模式（目的是了解ZK的原理）
ls /
[zookeeper, yarn-leader-election, hadoop-ha, rmstore]
Hbase
hbase 默认端口
引用hdfs的配置文件
hbase需要使用hadoop的一些配置,创建文件软件链接链接即可
# cd /usr/local/apps/hbase-2.3.5/conf
# ln -s /usr/local/apps/hadoop-3.2.2/etc/hadoop/core-site.xml core-site.xml
# ln -s /usr/local/apps/hadoop-3.2.2/etc/hadoop/hdfs-site.xml hdfs-site.xml
/usr/local/apps/hbase-2.3.5/conf/hbase-site.xml
hbase.rootdir
hdfs://ns1/hbase
region server的共享目录，用来持久化HBase
hbase.cluster.distributed
true
HBase的运行模式。false是单机模式，true是分布式模式
hbase.tmp.dir
/opt/hadoop/hbase/tmpdata
本地文件系统的临时文件夹。
hfile.block.cache.size
0.39
storefile的读缓存占用Heap的大小百分比，当然是越大越好，如果读比写多，开到0.4-0.5也没问题。如果读写较均衡，0.3左右。如果写比读多，果断默认吧。
hbase.rpc.timeout
900000
hbase client中的rpc请求超时时间
hbase.master
node1:60000
hbase.master.info.port
60010
HBase Master web 界面端口. 设置为-1 意味着你不想让他运行。
hbase.regionserver.port
60020
HBase RegionServer绑定的端口
hbase.regionserver.info.port
60030
HBase RegionServer web 界面绑定的端口 设置为 -1 意味这你不想与运行 RegionServer 界面
hbase.regionserver.lease.period
180000
客户端租用HRegion server 期限，即超时阀值。单位是毫秒。默认情况下，客户端必须在这个时间内发一条信息，否则视为死掉。
hbase.regionserver.restart.on.zk.expire
true
遇到ZooKeeper session expired（过期）， regionserver将选择 restart 而不是 abort（终止）
hbase.regionserver.handler.count
100
RegionServers处理远程请求的线程数，如果注重TPS（每秒事务数），可以调大，默认10。
1）值设得越大，意味着内存开销变大；
2）对于提高write的速度，如果瓶颈在做flush、compact、split的速度，磁盘io跟不上，提高线程数，意义不大。
hbase.hregion.memstore.block.multiplier
2
regionserver在写入时会检查每个region对应的memstore的总大小是否超过了memstore默认大小的2倍(hbase.hregion.memstore.block.multiplier决定)，如果超过了则锁住memstore不让新写请求进来并触发flush，避免产生OOM。
hbase.hregion.max.filesize
256000000
在当前ReigonServer上单个Reigon的最大存储空间，单个Region超过该值时，这个Region会被自动split成更小的region。
hbase.client.scanner.caching
10000
客户端参数，HBase scanner一次从服务端抓取的数据条数
hbase.client.scanner.timeout.period
900000
scanner过期时间
hbase.zookeeper.quorum
node1:2181,node2:2181,node3:2181
zookeeper.session.timeout
1200000
RegionServer与Zookeeper间的连接超时时间。当超时时间到后，ReigonServer会被Zookeeper从RS集群清单中移除，HMaster收到移除通知后，会对这台server负责的regions重新balance，让其他存活的RegionServer接管
hbase.zookeeper.property.dataDir
/opt/hadoop/zookeeper/data
ZooKeeper的zoo.conf中的配置。 快照的存储位置
修改hbase-env.sh
修改下面两项
export JAVA_HOME=/usr/local/jdk/
export HBASE_MANAGES_ZK=false
regionservers配置
在此文件里配置哪些机器运行regionserver
node1
node2
node3
同步到其他服务器
cd /usr/local/apps/
scp -r hbase-2.3.5 root@node2:/usr/local/apps/
scp -r hbase-2.3.5 root@node3:/usr/local/apps/
启动
时间同步
启动之前需要先同步下每台服务器的时间，否则HRegionServer无法启动起来
时间不同步的问题
在每个集群下面输入这个命令就可以更新时间
ntpdate pool.ntp.org
然后start-hbase.sh就可以了
- 地址 http://node1:60010
phoenix的搭建和使用
其实flink 是有fsql的，为什么还要用phoenix?phoenix支持JDBC的方式连接hbase,就跟以前的操作MYSQL 一样。
phoenix和hive的区别
hive是在hdfs上建立的SQL层 phoenix是在hbase上建立的SQL层，明白了吧
相关的介绍
本文对应的phoenix版本下载
http://archive.apache.org/dist/phoenix版本下载/phoenix-5.1.1/
问题
- Hbase 使用高可用 （HA）的hadoop集群，hbase.rootdir如何配置
问题描述
在前面文章中搭建了高可用的hadoop集群，然后hbase使用这个集群的hdfs。但是我们d hbase.rootdir配置的仍然是写死的机器：
hbase.rootdir
hdfs://node1:9000/hbase
如果node1此时宕机，处于active状态的nameserver变成了node2，那么hbase集群将不可用，启动hbase集群时将会报错org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby：
2019-07-22 11:11:24,431 ERROR [master/node1:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master node1,16000,1563765070980: Unhandled exception. Starting shutdown. *****
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1802)
//省略部分日志
2019-07-22 11:11:24,431 INFO [master/node1:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'node1,16000,1563765070980' *****
2019-07-22 11:11:24,432 INFO [master/node1:16000:becomeActiveMaster] regionserver.HRegionServer: STOPPED: Stopped by master/node1:16000:becomeActiveMaster
2019-07-22 11:11:27,078 INFO [master/node1:16000] ipc.NettyRpcServer: Stopping server on /192.168.229.128:16000
2019-07-22 11:11:27,097 WARN [master/node1:16000] regionserver.HRegionServer: Initialize abort timeout task failed
java.lang.IllegalAccessException: Class org.apache.hadoop.hbase.regionserver.HRegionServer can not access a member of class org.apache.hadoop.hbase.regionserver.HRegionServer$SystemExitWhenAbortTimeout with modifiers "private"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)
at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)
at java.lang.reflect.Constructor.newInstance(Constructor.java:413)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1044)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:598)
at java.lang.Thread.run(Thread.java:745)
2019-07-22 11:11:27,097 INFO [master/node1:16000] regionserver.HRegionServer: Stopping infoServer
2019-07-22 11:11:27,128 INFO [master/node1:16000] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@168cd36b{/,null,UNAVAILABLE}{file:/data/program/hbase-2.1.5/hbase-webapps/master}
2019-07-22 11:11:27,143 INFO [master/node1:16000] server.AbstractConnector: Stopped ServerConnector@319c3a25{HTTP/1.1,[http/1.1]}{0.0.0.0:16010}
2019-07-22 11:11:27,147 INFO [master/node1:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@3b2f4a93{/static,file:///data/program/hbase-2.1.5/hbase-webapps/static/,UNAVAILABLE}
2019-07-22 11:11:27,148 INFO [master/node1:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@3ffb3598{/logs,file:///data/program/hbase-2.1.5/logs/,UNAVAILABLE}
2019-07-22 11:11:27,151 INFO [master/node1:16000] regionserver.HRegionServer: aborting server node1,16000,1563765070980
2019-07-22 11:11:27,169 INFO [master/node1:16000] regionserver.HRegionServer: stopping server node1,16000,1563765070980; all regions closed.
2019-07-22 11:11:27,170 INFO [master/node1:16000] hbase.ChoreService: Chore service for: master/node1:16000 had [] on shutdown
2019-07-22 11:11:27,175 WARN [master/node1:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
2019-07-22 11:11:27,193 INFO [master/node1:16000] zookeeper.ZooKeeper: Session: 0x16c178f38b80009 closed
2019-07-22 11:11:27,194 INFO [master/node1:16000] regionserver.HRegionServer: Exiting; stopping=node1,16000,1563765070980; zookeeper connection closed.
2019-07-22 11:11:27,195 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:244)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3117)
解决办法
这里需要配置HA高可用的hdfs集群地址，而不是写死的某台机器。
修改hbase-site.xml
hbase.rootdir
hdfs://ns1/hbase
1
2
3
4
备注：这里ns1来自于hdfs-site.xml的配置dfs.nameservices:
dfs.nameservices
ns1
//省略其他配置
同时将hadoop的配置文件hdfs-site.xml和core-site.xml复制到hbase的conf目录下。不然会报找不到myha的错误。
重启hbase集群即可。
网上很多都说遇到java.lang.RuntimeException: HMaster Aborted错误需要清理zk：因为在CDH时重新添加删除HBASE导致的，需要清理zk中的hbase缓存，将zk的/hbase删除即可。
但是并不是所有问题都说由于zk造成的，看日志要多看一点，看到其根本原因。
flink
java 操作
flink官方的问答
- 本地模式?
- on yarn模式?
https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/try-flink/local_installation/
配置
flink-conf.yaml
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
#==============================================================================
# Common
#==============================================================================
# The external address of the host on which the JobManager runs and can be
# reached by the TaskManagers and any clients which want to connect. This setting
# is only used in Standalone mode and may be overwritten on the JobManager side
# by specifying the --host parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.
jobmanager.rpc.address: node1
# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 6123
# The total process memory size for the JobManager.
#
# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
jobmanager.memory.process.size: 1600m
# The total process memory size for the TaskManager.
#
# Note this accounts for all memory usage within the TaskManager process, including JVM metaspace and other overhead.
taskmanager.memory.process.size: 1728m
# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
#
# taskmanager.memory.flink.size: 1280m
# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.
taskmanager.numberOfTaskSlots: 1
# The parallelism used for programs that did not specify and other parallelism.
parallelism.default: 1
# The default file system scheme and authority.
#
# By default file paths without scheme are interpreted relative to the local
# root file system 'file:///'. Use this to override the default and interpret
# relative paths relative to a different file system,
# for example 'hdfs://mynamenode:12345'
#
# fs.default-scheme
#==============================================================================
# High Availability
#==============================================================================
# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
high-availability: zookeeper
# The path where metadata for master recovery is persisted. While ZooKeeper stores
# the small ground truth for checkpoint and leader election, this location stores
# the larger objects, like persisted dataflow graphs.
#
# Must be a durable file system that is accessible from all nodes
# (like HDFS, S3, Ceph, nfs, ...)
#
high-availability.storageDir: hdfs://ns1/flink/ha/
# The list of ZooKeeper quorum peers that coordinate the high-availability
# setup. This must be a list of the form:
# "host1:clientPort,host2:clientPort,..." (default clientPort: 2181)
#
high-availability.zookeeper.quorum: node1:2181,node2:2181,node3:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /FlinkCluster
# ACL options are based on https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_BuiltinACLSchemes
# It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" (ZOO_OPEN_ACL_UNSAFE)
# The default value is "open" and it can be changed to "creator" if ZK security is enabled
#
# high-availability.zookeeper.client.acl: open
#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================
# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# .
#
state.backend: filesystem
# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
state.checkpoints.dir: hdfs://ns1/flink/flink-checkpoints
# Default target directory for savepoints, optional.
#
state.savepoints.dir: hdfs://ns1/flink/flink-savepoints
# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend).
#
# state.backend.incremental: false
# The failover strategy, i.e., how the job computation recovers from task failures.
# Only restart tasks that may have been affected by the task failure, which typically includes
# downstream tasks and potentially upstream tasks if their produced data is no longer available for consumption.
jobmanager.execution.failover-strategy: region
#==============================================================================
# Rest & web frontend
#==============================================================================
# The port to which the REST client connects to. If rest.bind-port has
# not been specified, then the server will bind to this port as well.
#
#rest.port: 8081
# The address to which the REST client will connect to
#
#rest.address: 0.0.0.0
# Port range for the REST and web server to bind to.
#
#rest.bind-port: 8080-8090
# The address that the REST & web server binds to
#
#rest.bind-address: 0.0.0.0
# Flag to specify whether job submission is enabled from the web-based
# runtime monitor. Uncomment to disable.
#web.submit.enable: false
#==============================================================================
# Advanced
#==============================================================================
# Override the directories for temporary files. If not specified, the
# system-specific Java temporary directory (java.io.tmpdir property) is taken.
#
# For framework setups on Yarn or Mesos, Flink will automatically pick up the
# containers' temp directories without any need for configuration.
#
# Add a delimited list for multiple directories, using the system directory
# delimiter (colon ':' on unix) or a comma, e.g.:
# /data1/tmp:/data2/tmp:/data3/tmp
#
# Note: Each directory entry is read from and written to by a different I/O
# thread. You can include the same directory multiple times in order to create
# multiple I/O threads against that directory. This is for example relevant for
# high-throughput RAIDs.
#
# io.tmp.dirs: /tmp
# The classloading resolve order. Possible values are 'child-first' (Flink's default)
# and 'parent-first' (Java's default).
#
# Child first classloading allows users to use different dependency/library
# versions in their application than those in the classpath. Switching back
# to 'parent-first' may help with debugging dependency issues.
#
# classloader.resolve-order: child-first
# The amount of memory going to the network stack. These numbers usually need
# no tuning. Adjusting them may be necessary in case of an "Insufficient number
# of network buffers" error. The default min is 64MB, the default max is 1GB.
#
# taskmanager.memory.network.fraction: 0.1
# taskmanager.memory.network.min: 64mb
# taskmanager.memory.network.max: 1gb
#==============================================================================
# Flink Cluster Security Configuration
#==============================================================================
# Kerberos authentication for various components - Hadoop, ZooKeeper, and connectors -
# may be enabled in four steps:
# 1. configure the local krb5.conf file
# 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit)
# 3. make the credentials available to various JAAS login contexts
# 4. configure the connector to use JAAS/SASL
# The below configure how Kerberos credentials are provided. A keytab will be used instead of
# a ticket cache if the keytab path and principal are set.
# security.kerberos.login.use-ticket-cache: true
# security.kerberos.login.keytab: /path/to/kerberos/keytab
# security.kerberos.login.principal: flink-user
# The configuration below defines which JAAS login contexts
# security.kerberos.login.contexts: Client,KafkaClient
#==============================================================================
# ZK Security Configuration
#==============================================================================
# Below configurations are applicable if ZK ensemble is configured for security
# Override below configuration to provide custom ZK service name if configured
# zookeeper.sasl.service-name: zookeeper
# The configuration below must match one of the values set in "security.kerberos.login.contexts"
# zookeeper.sasl.login-context-name: Client
#==============================================================================
# HistoryServer
#==============================================================================
# The HistoryServer is started and stopped via bin/historyserver.sh (start|stop)
# Directory to upload completed jobs to. Add this directory to the list of
# monitored directories of the HistoryServer as well (see below).
#jobmanager.archive.fs.dir: hdfs:///completed-jobs/
# The address under which the web-based HistoryServer listens.
#historyserver.web.address: 0.0.0.0
# The port under which the web-based HistoryServer listens.
#historyserver.web.port: 8082
# Comma separated list of directories to monitor for completed jobs.
#historyserver.archive.fs.dir: hdfs:///completed-jobs/
# Interval in milliseconds for refreshing the monitored directories.
#historyserver.archive.fs.refresh-interval: 10000
上面的主要参数是
jobmanager.rpc.address: node1
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 1
parallelism.default: 1
high-availability: zookeeper
high-availability.storageDir: hdfs://ns1/flink/ha/
high-availability.zookeeper.quorum: hd1:2181,hd2:2181,hd3:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /FlinkCluster
state.backend: filesystem
state.checkpoints.dir: hdfs://ns1/flink/checkpoints
jobmanager.execution.failover-strategy: region
masters文件的配置
双master
node1
node2
workers任务执行服务器的配置
node3
配置/etc/profile
目前为止的配置
export HBASE_HOME=/usr/local/apps/hbase-2.3.5
export HADOOP_HOME=/usr/local/apps/hadoop-3.2.2
export JAVA_HOME=/usr/local/apps/jdk1.8.0_181
export PATH=$PATH:$JAVA_HOME/bin
export ZOOKEEPER_HOME=/usr/local/apps/zookeeper-3.4.14
export HADOOP_CLASSPATH=`${HADOOP_HOME}/bin/hadoop classpath`
export FLINK_HOME=/usr/local/apps/flink-1.13.5
PATH=$PATH:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$FLINK_HOME/bin
export PATH
传输到其他服务器
进入apps目录
scp -r flink-1.13.5 root@node2:/usr/local/apps/
scp -r flink-1.13.5 root@node3:/usr/local/apps/
进入etc目录
scp -r ./profile root@node2:/etc/
scp -r ./profile root@node3:/etc/
启动flink
[root@node1 ~]# start-cluster.sh
Starting HA cluster with 2 masters.
Starting standalonesession daemon on host node1.
Starting standalonesession daemon on host node2.
Starting taskexecutor daemon on host node3.
查看 http://node1:8081
flink的各种集成模式
https://blog.csdn.net/zxr0130/article/details/109116749
on yarn模式的搭建
总结
启动顺序
启动顺序：zk------------hadoop(hdfs-yarn)------hbase
关闭顺序:hbase-------hadoop(yarn-hdfs)--------zk
安装中必须要掌握的几个查询工具
-查询ZK数据中，到底那些服务使用了ZK做高可用
[zk: localhost:2181(CONNECTED) 0] ls /
[flink, zookeeper, yarn-leader-election, hadoop-ha, hbase, rmstore]
[zk: localhost:2181(CONNECTED) 1]
-查询那些服务的数据存储到了hdfs
[root@node1 ~]# hadoop fs -ls hdfs://ns1/
Found 2 items
drwxr-xr-x - root supergroup 0 2022-01-04 23:55 hdfs://ns1/flink
drwxr-xr-x - root supergroup 0 2022-01-05 00:06 hdfs://ns1/hbase
启动HBase集群：
bin/start-hbase.sh
单独启动一个HMaster进程：
bin/hbase-daemon.sh start master
单独停止一个HMaster进程：
bin/hbase-daemon.sh stop master
单独启动一个HRegionServer进程：
bin/hbase-daemon.sh start regionserver
单独停止一个HRegionServer进程：
bin/hbase-daemon.sh stop regionserver
彻底删除HBase数据
有时遇到诡异的问题 实在没办法解决就直接删库吧..
1、hadoop的bin目录下，执行命令以下命令清除Hbase数据
hadoop fs -rm -r /hbase
2、连接ZK，执行以下命令清除Hbase数据
rmr /hbase
3、重启ZK、重启hadoop（hdfs、yarn）、重启hbase
Hbase运维手册
https://blog.csdn.net/mnasd/article/details/81304762
参考
本文使用的参考
https://www.kingc.top/categories/bigdata
https://blog.csdn.net/o_guolin/article/details/116991396