11、安装Kafka

安装Kafka

安装scala

cd /usr/local
wget https://downloads.lightbend.com/scala/2.13.7/scala-2.13.7.tgz
mv scala-2.13.7 scala

配置环境变量
echo "export SCALA_HOME=/usr/local/scala" >> /etc/profile
echo -e 'export PATH=$PATH:$SCALA_HOME/bin'>> /etc/profile
source /etc/profile

测试：scala -version

将scala发送给其它节点，同时更改环境变量

scp -r /usr/local/scala root@hadoop02:/usr/local/
scp -r /usr/local/scala root@hadoop03:/usr/local/

echo "export SCALA_HOME=/usr/local/scala" >> /etc/profile
echo -e 'export PATH=$PATH:$SCALA_HOME/bin'>> /etc/profile
source /etc/profile

完成。

安装kafka

cd /usr/local
wget https://archive.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
tar -zxvf kafka_2.13-2.7.0
mv kafka_2.13-2.7.0 kafka
cd /usr/local/kafka/
mkdir data

cd config/
vim server.properties

修改为broker.id、zookeeper.connect、log.dirs

vim zookeeper.properties，将dataDir=/temp/zookeeper修改为dataDir=../zookeeper

配置环境变量：
echo "export KAFKA_HOME=/usr/local/kafka" >> /etc/profile
echo -e 'export PATH=$PATH:$KAFKA_HOME/bin'>> /etc/profile
source /etc/profile

将kafka发送给其它节点，同时更改环境变量

scp -r /usr/local/kafka root@hadoop02:/usr/local/
scp -r /usr/local/kafka root@hadoop03:/usr/local/

echo "export KAFKA_HOME=/usr/local/kafka" >> /etc/profile
echo -e 'export PATH=$PATH:$KAFKA_HOME/bin'>> /etc/profile
source /etc/profile

分别在hadoop102和hadoop103上修改配置文件server.properties中的broker.id=1、broker.id=2

cd /usr/local/kafka/config
vim server.properties

测试

启动集群，先启动zookeeper:
kafka-server-start.sh -daemon $KAFKA_HOME/config/zookeeper.properties
kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

创建topic：
kafka-topics.sh --create --bootstrap-server hadoop01:9092 --topic first --partitions 2 --replication-factor 2
查看当前服务器中的所有topic：
kafka-topics.sh --list --bootstrap-server hadoop01:9092
删除topic
kafka-topics.sh --bootstrap-server hadoop01:9092 --delete --topic first

发送消息
kafka-console-producer.sh  --broker-list hadoop01:9092 --topic first
消费消息
kafka-console-consumer.sh --bootstrap-server hadoop01:9092 --from-beginning --topic first

kafka-topics.sh --bootstrap-server hadoop01:9092 --describe --topic first
kafka-topics.sh --bootstrap-server hadoop01:9092 --delete --topic first

修改zookeeper-server-stop.sh

vim /usr/local/kafka/bin/kafka-server-stop.sh

kafka-server-stop.sh	//关闭kafka