Harbor私有仓库搭建并配置https对接docker与kubernetes

李朝阳李朝阳 - 2023-01-11 17:52:28

一、环境介绍

默认情况下，Harbor 不附带证书。可以在没有安全保护的情况下部署 Harbor，以便您可以通过 HTTP 连接到它。在生产环境中，推荐始终使用 HTTPS。要配置 HTTPS，必须创建 SSL 证书。可以使用由受信任的第三方 CA 签名的证书，也可以使用自签名证书。本文以自签名证书为例。

使用到的各个软件版本

操作系统版本：ubuntu 20.04
harbor版本：v2.5.3-797c3536
docker版本：20.10.8
kubernetets版本：1.22.2

harbor地址：192.168.10.112   域名：harbor.snow.com
master01地址：192.168.10.100

二、仓库部署

配置主机名与hosts文件

root@harbor:~# cat /etc/hosts
127.0.0.1 localhost
192.168.10.112 harbor.snow.com
修改主机名
root@barbor:~# hostnamectl set-hostname harbor
root@harbor:~# bash
root@harbor:~# hostname
harbor

下载harbor安装包

root@harbor:~#wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz
解压harbor安装包
root@harbor:~# tar xf harbor-offline-installer-v2.5.3.tgz -C /usr/local/src/
安装docker-compose
root@harbor:~# curl -SL https://github.com/docker/compose/releases/download/v2.7.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

生成证书颁发机构证书及私钥

root@harbor:/usr/local/src/harbor/certs# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
............................++++
............................................++++
e is 65537 (0x010001)
root@harbor:/usr/local/src/harbor/certs# openssl req -x509 -new -nodes -sha512 -days 3650 \
>  -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.snow.com" \
>  -key ca.key \
>  -out ca.crt
root@harbor:/usr/local/src/harbor/certs# ls
ca.crt  ca.key

生成服务器私钥及证书签名请求（CSR）

root@harbor:/usr/local/src/harbor/certs# openssl genrsa -out harbor.snow.com.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
...........++++
.....................................................................++++
e is 65537 (0x010001)

生成证书签名请求
root@harbor:/usr/local/src/harbor/certs# openssl req -sha512 -new \
> -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.snow.com" \
> -key harbor.snow.com.key \
> -out harbor.snow.com.csr
root@harbor:/usr/local/src/harbor/certs# ls
ca.crt  ca.key  harbor.snow.com.csr  harbor.snow.com.key

生成 x509 v3 扩展文件。
无论使用 FQDN 还是 IP 地址连接到Harbor服务器，都必须创建此文件，以便keyi 为的 Harbor 主机生成符合主题备用名称 (SAN) 和 x509 v3 的证书扩展要求。

root@harbor:/usr/local/src/harbor/certs# cat v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.snow.top
DNS.2=snow.top
DNS.3=harbor

使用该v3.ext文件为 Harbor 服务器生成证书。

root@harbor:/usr/local/src/harbor/certs# openssl x509 -req -sha512 -days 3650 \
> -extfile v3.ext \
> -CA ca.crt -CAkey ca.key -CAcreateserial \
> -in harbor.snow.com.csr \
> -out harbor.snow.com.crt
Signature ok
subject=C = CN, ST = Shanghai, L = Shanghai, O = SmartX, OU = Lab, CN = harbor.snow.com
Getting CA Private Key

将 harbor.snow.com.crt 转换为 harbor.snow.com.cert , 供 Docker 使用。Docker 守护进程将.crt文件解释为 CA 证书，.cert将文件解释为客户端证书。

root@harbor:/usr/local/src/harbor/certs# openssl x509 -inform PEM -in harbor.snow.com.crt -out harbor.snow.com.cert

配置 harbor yml文件，注意需要修改hostname与指定证书位置。

root@harbor:/usr/local/src/harbor# cat harbor.yml |more
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.snow.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /usr/local/src/harbor/certs/harbor.snow.com.cert
  private_key: /usr/local/src/harbor/certs/harbor.snow.com.key

安装harbor

root@harbor:/usr/local/src/harbor# ./install.sh 

[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.12

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.25.0

[Step 2]: loading Harbor images ...
Loaded image: goharbor/harbor-portal:v2.5.3
Loaded image: goharbor/harbor-core:v2.5.3
Loaded image: goharbor/redis-photon:v2.5.3
Loaded image: goharbor/prepare:v2.5.3
Loaded image: goharbor/harbor-db:v2.5.3
Loaded image: goharbor/chartmuseum-photon:v2.5.3
Loaded image: goharbor/harbor-jobservice:v2.5.3
Loaded image: goharbor/harbor-registryctl:v2.5.3
Loaded image: goharbor/nginx-photon:v2.5.3
Loaded image: goharbor/notary-signer-photon:v2.5.3
Loaded image: goharbor/harbor-log:v2.5.3
Loaded image: goharbor/harbor-exporter:v2.5.3
Loaded image: goharbor/registry-photon:v2.5.3
Loaded image: goharbor/notary-server-photon:v2.5.3
Loaded image: goharbor/trivy-adapter-photon:v2.5.3


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /usr/local/src/harbor
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/db/env
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating registryctl   ... done
Creating registry      ... done
Creating redis         ... done
Creating harbor-db     ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

安装完harbor后验证

root@harbor:/usr/local/src/harbor# docker ps 
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS                    PORTS                                                                            NAMES
0304ebf0cddc   goharbor/nginx-photon:v2.5.3         "nginx -g 'daemon of…"   38 seconds ago   Up 35 seconds (healthy)   0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp   nginx
10d1ff19120b   goharbor/harbor-jobservice:v2.5.3    "/harbor/entrypoint.…"   38 seconds ago   Up 36 seconds (healthy)                                                                                    harbor-jobservice
2d5e3d8877f8   goharbor/harbor-core:v2.5.3          "/harbor/entrypoint.…"   39 seconds ago   Up 38 seconds (healthy)                                                                                    harbor-core
ee6c12da89d9   goharbor/harbor-db:v2.5.3            "/docker-entrypoint.…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    harbor-db
5b7455e56733   goharbor/redis-photon:v2.5.3         "redis-server /etc/r…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    redis
a0c279ca0206   goharbor/harbor-registryctl:v2.5.3   "/home/harbor/start.…"   43 seconds ago   Up 41 seconds (healthy)                                                                                    registryctl
478f35433f69   goharbor/registry-photon:v2.5.3      "/home/harbor/entryp…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    registry
eb33121f4126   goharbor/harbor-portal:v2.5.3        "nginx -g 'daemon of…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    harbor-portal
b0c7ab3a7133   goharbor/harbor-log:v2.5.3           "/bin/sh -c /usr/loc…"   44 seconds ago   Up 43 seconds (healthy)   127.0.0.1:1514->10514/tcp                                                        harbor-log

修改宿主机hosts文件，做好harbor主机的域名解析，在浏览器使用域名访问harbor服务器发现提示（忽略域名，域名是第一次测试的时候用的）

在当前页面用键盘输入thisisunsafe，不是在地址栏输入，就直接敲键盘就行了，页面即会自动刷新进入网页

使用默认用户名admin，密码Harbor12345即可登录。

三、配置docker客户端

将证书发送到master01服务器,注意在操作之前要先在master01创建好存放证书的文件夹/etc/docker/certs.d/harbor.snow.com

root@master01:~# mkdir -p /etc/docker/certs.d/harbor.snow.com

root@harbor:/usr/local/src/harbor/certs# scp harbor.snow.com.cert 192.168.10.100:/etc/docker/certs.d/harbor.snow.com/
root@192.168.10.100's password: 
harbor.snow.com.cert                                                                                        100% 2102     2.3MB/s   00:00      

root@harbor:/usr/local/src/harbor/certs# scp harbor.snow.com.key 192.168.10.100:/etc/docker/certs.d/harbor.snow.com/
root@192.168.10.100's password: 
harbor.snow.com.key                                                                                         100% 2045     2.0MB/s   00:00
  
root@harbor:/usr/local/src/harbor/certs# scp ca.crt  192.168.10.100:/etc/docker/certs.d/harbor.snow.com
root@192.168.10.100's password: 
ca.crt                                                                                                      100% 2045     2.0MB/s   00:00

在master01节点看到证书已经成功发送过来

root@master01:/etc/docker/certs.d/harbor.snow.com# ls
ca.crt  harbor.snow.com.cert  harbor.snow.com.key

修改docker的daemon.json文件，使其信任harbor.snow.com仓库

root@master01:/etc/docker# cat daemon.json 
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries":["harbor.snow.com"],
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}

重启docker服务并修改其hosts文件使master01节点能够通过域名找到对应的harbor仓库

root@master01:~# systemctl restart docker
root@master01:~# cat /etc/hosts
127.0.0.1 localhost
192.168.10.112 harbor.snow.com

登录仓库

root@master01:~# docker login harbor.snow.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

上传镜像进行测试，注意要提前在web页面创建好新的项目。

root@master01:~# docker images
REPOSITORY                                TAG             IMAGE ID       CREATED         SIZE
calico/cni                                v3.19.2         05bf027c9836   17 months ago   146MB

为镜像改名字
root@master01:~# docker tag calico/cni:v3.19.2 harbor.snow.com/calico/cni:v3.19.2

推送到镜像仓库
root@master01:~# docker push harbor.snow.com/calico/cni:v3.19.2
The push refers to repository [harbor.snow.com/calico/cni]
758850cabe72: Pushed 
e88fd392bb0c: Pushed 
fb44224b4823: Pushed 
v3.19.2: digest: sha256:00a619424a20d1b31c0f65d1194484aebe9f90dfc1d4d6396ce706941004860b size: 946

推送到镜像仓库后即可在web页面看到刚刚推送上去镜像

镜像拉取测试

root@master01:~# docker images
REPOSITORY                                  TAG             IMAGE ID       CREATED         SIZE
kubernetesui/dashboard                      v2.3.1          e1482a24335a   18 months ago   220MB

root@master01:~# docker pull harbor.snow.com/calico/cni:v3.19.2
v3.19.2: Pulling from calico/cni
Digest: sha256:00a619424a20d1b31c0f65d1194484aebe9f90dfc1d4d6396ce706941004860b
Status: Downloaded newer image for harbor.snow.com/calico/cni:v3.19.2
harbor.snow.com/calico/cni:v3.19.2

root@master01:~# docker images
REPOSITORY                                  TAG             IMAGE ID       CREATED         SIZE
kubernetesui/dashboard                      v2.3.1          e1482a24335a   18 months ago   220MB
harbor.snow.com/calico/cni                  v3.19.2         05bf027c9836   17 months ago   146MB

四、K8S集群配置使用私有镜像仓库Harbor

在配置k8s使用harbor之前需要先完成本文的第三步，先让docker可以正常使用harbor。
在docker成功登录harbor后会在用户家目录生成一个秘钥文件

root@master01:~# cat /root/.docker/config.json
{
    "auths": {
        "harbor.snow.com": {
            "auth": "YWRtaW46SGFyYm9yMTIzNDU="
        }
    }
}

将秘钥进行base64加密

root@master01:~# cat /root/.docker/config.json | base64 -w 0
ewoJImF1dGhzIjogewoJCSJoYXJib3Iuc25vdy5jb20iOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=

创建用户harbor认证的secret的yaml文件

root@master01:~/secret# cat registry-pull-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret
  namespace: kube-system
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoYXJib3Iuc25vdy5jb20iOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
type: kubernetes.io/dockerconfigjson

创建secret，注意secret为名称空间级别的资源，调用这个secret的yaml文件要与此secret在同一个名称空间下。

root@master01:~/secret# kubectl delete -f registry-pull-secret.yaml 
secret "registry-pull-secret" deleted
root@master01:~/secret# kubectl apply -f registry-pull-secret.yaml 
secret/registry-pull-secret created
root@master01:~/secret# kubectl get secret
NAME                   TYPE                                  DATA   AGE
default-token-rxd89    kubernetes.io/service-account-token   3      5h2m
registry-pull-secret   kubernetes.io/dockerconfigjson        1      3h

最后只需要在需要去harbor拉取镜像的pod的yaml文件中添加如下配置即可（只展示出了完成yaml文件的部分内容）注意其是与containers语句是平级的。

template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      imagePullSecrets:
        - name: registry-pull-secret
      containers:
        image: harbor.snow.com/base/metrics-server:v0.5.0
        imagePullPolicy: Always

配置完成后k8s集群即可成功拉取harbor中的镜像。如果在harbor中将要访问的镜像级别设置为公开，则不需要secret认证即可正常拉取。

ps：以上内容在本人实现环境中已试验成功，如发现有问题或表述不清的地方欢迎指正。

特别申明：本文内容来源网络，版权归原作者所有，如有侵权请立即与我们联系（cy198701067573@163.com），我们将及时处理。

上一篇： k8s安装dashboard

下一篇： Kali Linux渗透测试之主动信息收集