【Spark集群】一、配置搭建

一、standalone的搭建

export JAVA_HOME=/opt/module/jdk1.8.0_301
SPARK_MASTER_HOST=hadoop101
SPARK_MASTER_PORT=7077


bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://hadoop101:7077 \
./examples/jars/spark-examples_2.12-3.0.0.jar \
10

spark.eventLog.enabled true
spark.eventLog.dir hdfs://hadoo101:8020/directory




hadoop fs -mkdir /spark_eventLog_directory
export SPARK_HISTORY_OPTS="
-Dspark.history.ui.port=18080 
-Dspark.history.fs.logDirectory=hdfs://hadoop101:8020/spark_eventLog_directory
-Dspark.history.retainedApplications=30"

二、yarn模式启动

spark-yarn相当作为客户端，只需要一个

yarn-site.xml



 yarn.nodemanager.pmem-check-enabled
 false



 yarn.nodemanager.vmem-check-enabled
 false







提交脚本命令

bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
./examples/jars/spark-examples_2.12-3.0.0.jar \
10