【hadoop集群】一、安装hadoop3.1.3 核心配置文件内容以及可能出现的问题

1. INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:41576. Already tried 0 time(s); 

或者卡在

2022-01-04 01:26:35,209 INFO mapreduce.Job: Running job: job_1641277332916_0002，后面跑不下去了

解决办法：vim /etc/hosts 

把127.0.0.1这行注释掉

2、hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

Got exception: java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:43696 failed on connection exception: java.net.ConnectException: Connection refused;

解决办法：vim /etc/hosts 

把127.0.0.1这行注释掉

3、bin/beeline -u jdbc:hive2://hadoop103:10000/metastore -n zhyx

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
解决办法换jar包，hive包与hadoop冲突，需要编译

4、WARN jdbc.HiveConnection: Failed to connect to hadoop103:10000

Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://hadoop103:10000/metastore: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: zhyx is not allowed to impersonate zhyx (state=08S01,code=0)

netstat -nlpt |grep 9083 查看这个进程是否开启

netstat -nlpt |grep 10000  查看这个进程是否开启

查看namenode datanode占用内存：

jmap -heap  2611（namenode的进程号）

权限不够：增加配置  

zhyx是用户名，用户组

    hadoop.proxyuser.zhyx.hosts
    *


    hadoop.proxyuser.zhyx.groups
    *

hadoop四个配置文件

 
 fs.defaultFS
 hdfs://hadoop101:8020
 
 
 
 hadoop.tmp.dir
 /opt/module/hadoop-3.1.3/data
 
 
 
 hadoop.http.staticuser.user
 zhyx
 




    hadoop.proxyuser.zhyx.hosts
    *


    hadoop.proxyuser.zhyx.groups
    *






 dfs.namenode.http-address
 hadoop101:9870
 

 
 dfs.namenode.secondary.http-address
 hadoop103:9868
 






 
 yarn.nodemanager.aux-services
 mapreduce_shuffle
 
 
 
 yarn.resourcemanager.hostname
 hadoop102
 
 
 
 yarn.nodemanager.env-whitelist
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
 



 yarn.log-aggregation-enable
 true


 
 yarn.log.server.url 
 http://hadoop103:19888/jobhistory/logs



 yarn.log-aggregation.retain-seconds
 604800

 yarn.nodemanager.vmem-check-enabled
 false

 
 mapreduce.framework.name
 yarn
 



 mapreduce.jobhistory.address
 hadoop103:10020



 mapreduce.jobhistory.webapp.address
 hadoop103:19888



start-dfs.sh
start-yarn.sh
mapred --daemon start historyserver

mapred --daemon stop historyserver

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input/ /output


hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

 Got exception: java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:43696 failed on connection exception: java.net.ConnectException: Connection refused;


-X)0yn6k2ZR6



<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 
 
 javax.jdo.option.ConnectionURL
 jdbc:mysql://hadoop103:3306/metastore?useSSL=false

 
 
 javax.jdo.option.ConnectionDriverName
 com.mysql.jdbc.Driver


 
 javax.jdo.option.ConnectionUserName
 root
 
 
 
 javax.jdo.option.ConnectionPassword
 zhyx

 
 
 hive.metastore.schema.verification
 false

 
 
 hive.metastore.event.db.notification.api.auth
 false
 
 
 
hive.metastore.warehouse.dir
 /user/hive/warehouse
 


 
 hive.metastore.uris
 thrift://hadoop103:9083
 


 
 hive.server2.thrift.bind.host
hadoop103
 
 
 
 hive.server2.thrift.port
 10000