Harbor镜像仓库（含clair镜像扫描） - 完整部署记录

 

 Harbor环境部署的要求：系统版本在Centos7.5以上、内核版本在4.4X以上、ip_forward路由转发功能要打开。

一、环境准备

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

[root@k8s-harbor01 ~]# cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core)    [root@k8s-harbor01 ~]# uname -r 4.4.232-1.el7.elrepo.x86_64    [root@k8s-harbor01 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@k8s-harbor01 ~]# vim /etc/sysctl.conf net.ipv4.ip_forward = 1 [root@k8s-harbor01 ~]# sysctl -p    [root@k8s-harbor01 ~]# systemctl stop firewalld && systemctl disable firewalld && firewall-cmd --state    [root@k8s-harbor01 ~]# vim /etc/sysconfig/selinux SELINUX=disabled [root@k8s-harbor01 ~]# getenforce Disabled    [root@k8s-harbor01 ~]# python --version Python 2.7.5

二、安装Docker

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

提前下载二进制安装包docker-18.09.6.tgz到/usr/local/src路径下，解压安装 [root@k8s-harbor01 ~]# cd /usr/local/src/ [root@k8s-harbor01 src]# ll docker-18.09.6.tgz -rw-r--r-- 1 root root 48047231 Oct 19  2019 docker-18.09.6.tgz [root@k8s-harbor01 src]# tar -zvxf docker-18.09.6.tgz    [root@k8s-harbor01 src]# cp docker/* /usr/local/bin/ [root@k8s-harbor01 src]# chmod 755 /usr/local/bin/*    /usr/local/bin默认已经加到系统环境变量中 [root@k8s-harbor01 src]# echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin    编辑docker启动文件 注意"WorkingDirectory"路径要和/etc/docker/daemon.json文件中的data-root、exec-root路径一致 [root@k8s-harbor01 src]# cat > /etc/systemd/system/docker.service << EOF [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.io    [Service] WorkingDirectory=/data/docker Environment="PATH=/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin" EnvironmentFile=-/run/flannel/docker ExecStart=/usr/local/bin/dockerd ExecReload=/bin/kill -s HUP Restart=on-failure RestartSec=5 LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity Delegate=yes KillMode=process    [Install] WantedBy=multi-user.target    EOF    授执行权限 [root@k8s-harbor01 src]# chmod 755 /etc/systemd/system/docker.service    编辑docker 配置文件 编辑docker 配置文件 [root@k8s-harbor01 src]# mkdir -p /etc/docker && mkdir -p /data/docker/data && mkdir -p /data/docker/exec [root@k8s-harbor01 src]# cat > /etc/docker/daemon.json << EOF {     "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],     "insecure-registries": ["docker02:35000"],     "max-concurrent-downloads": 20,     "live-restore": true,     "max-concurrent-uploads": 10,     "debug": true,     "data-root": "/data/docker/data",     "exec-root": "/data/docker/exec",     "log-opts": {       "max-size": "100m",       "max-file": "5"     } }    EOF    启动 docker 服务 [root@k8s-harbor01 src]# systemctl daemon-reload && systemctl enable docker && systemctl restart docker [root@k8s-harbor01 src]# systemctl status docker|grep Active    Active: active (running) since Wed 2020-08-12 13:41:07 CST; 28s ago    查看 Docker 版本号 [root@k8s-harbor01 src]# docker --version Docker version 18.09.6, build 481bc77

三、安装Docker-Compose

1 2 3 4 5 6 7 8 9 10 11

下载docker-compose二进制执行文件 百度网盘下载地址：https://pan.baidu.com/s/1er0rM0vxEubYOLHx7LI62A 提取密码：eer9 [root@k8s-harbor01 ~]# cd /usr/local/src/ [root@k8s-harbor01 src]# curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose [root@k8s-harbor01 src]# cp docker-compose /usr/local/bin/ [root@k8s-harbor01 src]# chmod 755 /usr/local/bin/*    查看 docker-compose 版本号 [root@k8s-harbor01 ~]# docker-compose --version docker-compose version 1.26.0, build d4451659

四、部署Harbor镜像仓库

1）HTTPS证书自签
如果线上环境有已购买好的HTTPS证书可以直接拿过来用，如果没有，就在Harbor本机进行HTTPS证书自签。这里Harbor本机ip地址是172.16.60.238

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

生成CA证书私钥 [root@k8s-harbor01 ~]# openssl genrsa -out ca.key 4096    生成CA证书 [root@k8s-harbor01 ~]# openssl req -x509 -new -nodes -sha512 -days 3650 \  -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.60.238" \  -key ca.key \  -out ca.crt    生成服务器证书 1）生成私钥 [root@k8s-harbor01 ~]# openssl genrsa -out 172.16.60.238.key 4096    2）生成证书签名请求（CSR） [root@k8s-harbor01 ~]# openssl req -sha512 -new \     -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.60.238" \     -key 172.16.60.238.key \     -out 172.16.60.238.csr    3）生成一个x509 v3扩展文件（两种方式根据情况二选一） #################################################################################### 第一种方式：域名 cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names    [alt_names] DNS.1=172.16.60.238 DNS.2=yourdomain DNS.3=hostname EOF #################################################################################### 第二种方式：IP cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = IP:172.16.60.238 EOF ####################################################################################    这里选择第二种的IP方式 [root@k8s-harbor01 ~]# cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = IP:172.16.60.238 EOF    4）使用该v3.ext文件为您的Harbor主机生成证书 [root@k8s-harbor01 ~]# openssl x509 -req -sha512 -days 3650 \     -extfile v3.ext \     -CA ca.crt -CAkey ca.key -CAcreateserial \     -in 172.16.60.238.csr \     -out 172.16.60.238.crt

2）提供证书给Harbor和Docker

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

1）将服务器证书和密钥复制到Harbor主机上的certficates文件夹中 根据自己实际环境需求创建Harbor的certficates文件夹 [root@k8s-harbor01 ~]# mkdir -p /data/cert/ [root@k8s-harbor01 ~]# cp 172.16.60.238.crt /data/cert/ [root@k8s-harbor01 ~]# cp 172.16.60.238.key /data/cert/    2）转换 172.16.60.238.crt 为172.16.60.238.cert，供Docker使用。 Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书。 [root@k8s-harbor01 ~]# openssl x509 -inform PEM -in 172.16.60.238.crt -out 172.16.60.238.cert    3）将服务器证书，密钥和CA文件复制到Harbor主机上的Docker certificate文件夹中。 记住必须首先创建适当的文件夹 [root@k8s-harbor01 ~]# mkdir -p /etc/docker/certs.d/172.16.60.238/ [root@k8s-harbor01 ~]# cp 172.16.60.238.cert /etc/docker/certs.d/172.16.60.238/ [root@k8s-harbor01 ~]# cp 172.16.60.238.key /etc/docker/certs.d/172.16.60.238/ [root@k8s-harbor01 ~]# cp ca.crt /etc/docker/certs.d/172.16.60.238/    4）重新启动Docker [root@k8s-harbor01 ~]# systemctl restart docker [root@k8s-harbor01 ~]# systemctl status docker    5）将名为"ca.crt"的CA证书下载到本地电脑，然后安装证书。 这样就可以在本地电脑的浏览器里正常访问https地址的Harbor了（证书可被信任）

3）安装Harbor
到 Harbor的GitHub仓库的Release页面 , 下载最新的在线安装包

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84

这里下载Harbor V2.0.2版本的安装包 [root@k8s-harbor01 ~]# cd /usr/local/src/ [root@k8s-harbor01 src]# wget https://github.com/goharbor/harbor/releases/download/v2.0.2/harbor-online-installer-v2.0.2.tgz [root@k8s-harbor01 src]# tar -zvxf harbor-online-installer-v2.0.2.tgz [root@k8s-harbor01 src]# mv harbor /opt/    修改harbor配置信息 [root@k8s-harbor01 src]# cd /opt/harbor/ [root@k8s-harbor01 harbor]# cp harbor.yml.tmpl harbor.yml ......... ........ hostname: 172.16.60.238    # http related config http:   # port for http, default is 80. If https enabled, this port will redirect to https port   port: 80    # https related config https:   # https port for harbor, default is 443   port: 443   # The path of cert and key files for nginx   certificate: /data/cert/172.16.60.238.crt   private_key: /data/cert/172.16.60.238.key ........ ........ harbor_admin_password: Harbor@123456 ........ ........ data_volume: /data       运行install.sh, 注意运行时加上--with-clair 选项，启动clair镜像扫描功能 [root@k8s-harbor01 harbor]# ./install.sh --with-clair ........ ........ ? ----Harbor has been installed and started successfully.----    出现上面的信息，说明Harbor已经安装完成了。    查看harbor启动情况。 docker-compose 命令必须要在harbor安装目录 (这里就是/opt/harbor)路径下才能执行。 [root@k8s-harbor01 harbor]# docker-compose ps       Name                     Command                  State                          Ports --------------------------------------------------------------------------------------------------------------- clair               ./docker-entrypoint.sh           Up (healthy)   6060/tcp, 6061/tcp clair-adapter       /home/clair-adapter/entryp ...   Up (healthy)   8080/tcp harbor-core         /harbor/entrypoint.sh            Up (healthy) harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp harbor-jobservice   /harbor/entrypoint.sh            Up (healthy) harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp registry            /home/harbor/entrypoint.sh       Up (healthy)   5000/tcp registryctl         /home/harbor/start.sh            Up (healthy)    查看harbor镜像 [root@k8s-harbor01 ~]# docker images REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE goharbor/redis-photon           v2.0.2              e547529bb6a1        3 weeks ago         72.3MB goharbor/clair-adapter-photon   v2.0.2              9ec8853dc3cb        3 weeks ago         62MB goharbor/clair-photon           v2.0.2              73885002dda7        3 weeks ago         171MB goharbor/harbor-registryctl     v2.0.2              9f8b7bb0f1ff        3 weeks ago         101MB goharbor/registry-photon        v2.0.2              eac8c5fc9ca8        3 weeks ago         83.6MB goharbor/nginx-photon           v2.0.2              eee4771b916c        3 weeks ago         43.6MB goharbor/harbor-log             v2.0.2              b2db762a6c3a        3 weeks ago         82.1MB goharbor/harbor-jobservice      v2.0.2              3960e027ccb9        3 weeks ago         164MB goharbor/harbor-core            v2.0.2              de2495b944cf        3 weeks ago         145MB goharbor/harbor-portal          v2.0.2              90088a0e64a9        3 weeks ago         52.5MB goharbor/harbor-db              v2.0.2              81e98a7af097        3 weeks ago         161MB goharbor/prepare                v2.0.2              7e804db05454        3 weeks ago         160MB    确保harbpr启动后的80和443端口都起来了 [root@k8s-harbor01 harbor]# lsof -i:80 COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME docker-pr 3095 root    4u  IPv6  26027      0t0  TCP *:http (LISTEN) [root@k8s-harbor01 harbor]# lsof -i:443 COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME docker-pr 3082 root    4u  IPv6  26015      0t0  TCP *:https (LISTEN)    到这里就可以访问harbor了,访问地址为：https://172.16.60.238 用户名为admin，密码为配置文件中定义的"Harbor@123456"

查看clair镜像扫描器

Habor 服务启停
注意：如果harbor.yml配置修改了，要先执行"./prepare"命令进行配置载入，然后再重启harbor服务。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

查看Habror docker-compose ps    启动Harbor docker-compose start    停止Harbor docker-compose stop    重启Harbor docker-compose restart    另外： Harbor还可以通过down和up命令去停止和启动， 只不过这种方式是删除、创建的关停和启动。 docker-compose down -v docker-compose up -d

五、客户端登录Harbor

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78

在Habror客户端机器（如k8s的node节点、harbor节点）配置登录：    默认情况下，在客户端登录Habor是会报错的： [root@k8s-node01 ~]# docker login 172.16.60.238 Authenticating with existing credentials... Login did not succeed, error: Error response from daemon: Get https://172.16.60.238/v2/: x509: certificate signed by unknown authority    原因： 客户端登录Harbor，https证书不被信任。    解决办法：下面两种方法选其一 1）方法一 将Harbor服务器证书，密钥和CA文件复制到Harbor客户主机上的Docker certificate文件夹中 [root@k8s-node01 ~]# mkdir -p /etc/docker/certs.d/172.16.60.238/ [root@k8s-node01 ~]# cd /etc/docker/certs.d/172.16.60.238/ [root@k8s-node01 172.16.60.238]# rsync -e "ssh -p22" -avpgolr 172.16.60.238:/etc/docker/certs.d/172.16.60.238/* ./ [root@k8s-node01 172.16.60.238]# ll total 12 -rw-r--r-- 1 root root 2053 Aug 19 14:34 172.16.60.238.cert -rw-r--r-- 1 root root 3243 Aug 19 14:34 172.16.60.238.key -rw-r--r-- 1 root root 2033 Aug 19 14:34 ca.crt    重启docker服务 [root@k8s-node01 172.16.60.238]# systemctl restart docker [root@k8s-node01 172.16.60.238]# systemctl status docker    再次验证登录harbor [root@k8s-node01 172.16.60.238]# docker login 172.16.60.238 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store    Login Succeeded    2）方法二 配置docker服务的daemon.json文件，添加"insecure-registries"参数，表示忽略ssl证书认证。 [root@k8s-node01 ~]# vim /etc/docker/daemon.json ........     "insecure-registries": ["https://172.16.60.238"],    重启docker服务 [root@k8s-node01 ~]# systemctl restart docker [root@k8s-node01 ~]# systemctl status docker    再次验证登录harbor [root@k8s-node01 ~]# docker login 172.16.60.238 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store    Login Succeeded    ======================================================================== 另外，注意客户端机器登录Harbor时，只要首次登录需要输入用户名和密码。 登录成功后的信息默认保存到/root/.docker/config.json文件里。 下次登录时就不用再输入harbor用户名和密码了，直接读取config.json文件内容 [root@k8s-node01 ~]# cat /root/.docker/config.json {         "auths": {                 "172.16.60.238": {                         "auth": "YWRtaW46SGFyYm9yQDEyMzQ1Ng=="                 }         },         "HttpHeaders": {                 "User-Agent": "Docker-Client/18.09.6 (linux)"         }       [root@k8s-node01 ~]# docker login 172.16.60.238 Authenticating with existing credentials... WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store    Login Succeeded

六、Harbor镜像扫描

选中镜像，进行漏洞扫描

如果扫描出漏洞，在漏洞报告了会告知漏洞当前版本和修复版本，按照修复版本修复即可。

修复方法：
可以依据当前基础镜像做Dockerfile，使用"yum update -y 漏洞所属软件名" 进行升级操作，然后再重新做一个基础镜像。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

1) 编译Dockerfile 升级原来centos7.7基础镜像里报出来漏洞的软件 [root@k8s-harbor01 ~]# cat Dockerfile FROM 172.16.60.238/kevin/centos7.7:latest RUN yum update -y sqlite \ && yum update -y nss-util \ && yum update -y nss-sysinit \ && yum update -y dbus-libs \ && yum update -y bind-license \ && yum update -y nss \ && yum update -y nss-softokn \ && yum update -y dbus \ && yum update -y nss-softokn-freebl \ && yum update -y nss-tools \ && yum update -y bash \ && yum update -y python-libs \ && yum update -y python \ && yum update -y bind-license \ && yum update -y expat \ && yum update -y libxml2-python \ && yum update -y libxml2 \ && yum update -y shared-mime-info \ && yum update -y libcurl \ && yum update -y file-libs \ && yum update -y curl    2）制作新的基础镜像 [root@k8s-harbor01 ~]# docker build -t 172.16.60.238/kevin/centos7.7:updatev1 .    3）上传到Harbor仓库 [root@k8s-harbor01 ~]# docker push 172.16.60.238/kevin/centos7.7:updatev1

将修复好漏洞的新基础镜像上传到Harbor仓库，再扫描新镜像，发现漏洞已修复。

*************** 当你发现自己的才华撑不起野心时，就请安静下来学习吧！***************   分类: