Datax往hdfs写数据配置HA高可用


问题描述:

datax往hdfs写数据如何配置高可用

问题解析:

datax官网有相关配置,但是太过于笼统不够详细

解决方案:(nameservice1)

"defaultFS":"hdfs://nameservice1",
"hadoopConfig":{
"dfs.nameservices": "nameservice1",
"dfs.ha.namenodes.nameservice1": "namenode142,namenode35",
"dfs.namenode.rpc-address.nameservice1.namenode142": "hadoop-prod-datanode7:8020",
"dfs.namenode.rpc-address.nameservice1.namenode35": "hadoop-prod-namenode:8020",
"dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
},

每个参数分别对用hdfs-site.xml文件配置信息