【hadoop集群】三、hive添加tez、spark引擎

一、添加tez

上传jar包到hdfs

hadoop配置添加

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


tez.lib.uris
 ${fs.defaultFS}/tez/tez-0.10.1-SNAPSHOT.tar.gz


 tez.use.cluster.hadoop-libs
 true


 tez.am.resource.memory.mb
 1024


 tez.am.resource.cpu.vcores
 1


 tez.container.max.java.heap.fraction
 0.4


 tez.task.resource.memory.mb
 1024


 tez.task.resource.cpu.vcores
 1

再添加

hadoop_add_profile tez
function _tez_hadoop_classpath
{
 hadoop_add_classpath "$HADOOP_HOME/etc/hadoop" after
 hadoop_add_classpath "/opt/module/tez/*" after
 hadoop_add_classpath "/opt/module/tez/lib/*" after
}

　　hive添加

 hive.execution.engine
 tez


 hive.tez.container.size
 1024

　问题：

Current usage: 303.3 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.

关闭虚拟机检查  yarn-site.xml

	yarn.nodemanager.vmem-check-enabled
    false

 二、添加spark引擎

把spark的jar上传到hdfs

hive配置信息spark-defaults.conf

spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir                        hdfs://hadoop102:8020/spark-history
spark.executor.memory                    1g
spark.driver.memory					   1g

hive-site.xml 配置信息

    spark.yarn.jars
    hdfs://hadoop102:8020/spark-jars/*

  


    hive.execution.engine
    spark

 执行成功