Helm安装Harbor

“ 💋 戏 子. ✟ 发表于: 2023-02-01   最后更新时间: 2023-02-01 11:27:31  
{{totalSubscript}} 订阅， 959 游览

前言

文章中用到的harbor存储是hostpath，service的类型是ingress。根据harbor官网提示，需要提前创建pv和pvc，既然选择了hostpath，那么就需要将pod固定到某一个node上面，本文所有的资源副本都是1，harbor的所有pod放在同一个namespace，并且需要将namespace固定到某个node上，具体步骤见正文。

一.环境准备

1.二进制安装helm

[root@k8s-master helm]# wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz
[root@k8s-master helm]# tar zxf helm-v3.10.3-linux-amd64.tar.gz
[root@k8s-master helm]# cd linux-amd64/
[root@k8s-master helm]# cp helm /usr/local/bin/
[root@k8s-master helm]# helm version
version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.18.9"}

2.创建namespace

[root@k8s-master helm]# cat namespace-harbor.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: harbor
[root@k8s-master helm]# kubectl apply -f namespace-harbor.yaml

3.开启准入控制器

--enable-admission-plugins 添加 PodNodeSelector

[root@k8s-master helm]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.20:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.0.20
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,PodNodeSelector

apiserver pod会自动重启

4.命名空间加注解

[root@k8s-master helm]# kubectl edit ns harbor -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: harbor=env
  name: harbor

5.node打标签

[root@k8s-master helm]# kubectl label node k8s-node3 harbor=env

6.创建pv和pvc

选择hostpath，需要提前准备pv和pvc，官网解释如下：
在这里插入图片描述

官网链接：https://goharbor.io/docs/2.7.0/install-config/harbor-ha-helm/

在k8s-node3节点创建目录，建议给/data下单独挂一块存储盘

[root@k8s-node3 data]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        500G   98M  500G   1% /data
[root@k8s-node3 data]# mkdir -pv harbor/{chartmuseum,database,jobservice,redis,registry,scandata,trivy}
mkdir: created directory ‘harbor’
mkdir: created directory ‘harbor/chartmuseum’
mkdir: created directory ‘harbor/database’
mkdir: created directory ‘harbor/jobservice’
mkdir: created directory ‘harbor/redis’
mkdir: created directory ‘harbor/registry’
mkdir: created directory ‘harbor/scandata’
mkdir: created directory ‘harbor/trivy’
[root@k8s-node3 data]# chmod 777 -R harbor/   ##必须要赋权，不然pod启动会有问题

创建pv和pvc

[root@k8s-master helm]# cat harbor-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-registry-pv"
  labels:
    name: harbor-registry-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/registry
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-chartmuseum-pv"
  labels:
    name: harbor-chartmuseum-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/chartmuseum
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-jobservice-pv"
  labels:
    name: harbor-jobservice-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/jobservice
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-database-pv"
  labels:
    name: harbor-database-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/database
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-redis-pv"
  labels:
    name: harbor-redis-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/redis
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-trivy-pv"
  labels:
    name: harbor-trivy-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/trivy
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-scandata-pv"
  labels:
    name: harbor-scandata-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/scandata
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-registry-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-registry-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-chartmuseum-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-chartmuseum-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-jobservice-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-jobservice-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-database-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-database-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-redis-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-redis-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-trivy-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-trivy-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-scandata-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-scandata-pv
      release: stable
[root@k8s-master helm]# kubectl apply -f harbor-pv.yaml
[root@k8s-master helm]# kubectl get pv,pvc -n harbor
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                        STORAGECLASS         REASON   AGE
persistentvolume/harbor-chartmuseum-pv                      5Gi        RWO            Retain           Bound    harbor/harbor-chartmuseum-pvc                                              31h
persistentvolume/harbor-database-pv                         5Gi        RWO            Retain           Bound    harbor/harbor-database-pvc                                                 31h
persistentvolume/harbor-jobservice-pv                       5Gi        RWO            Retain           Bound    harbor/harbor-jobservice-pvc                                               31h
persistentvolume/harbor-redis-pv                            5Gi        RWO            Retain           Bound    harbor/harbor-redis-pvc                                                    31h
persistentvolume/harbor-registry-pv                         5Gi        RWO            Retain           Bound    harbor/harbor-registry-pvc                                                 31h
persistentvolume/harbor-scandata-pv                         5Gi        RWO            Retain           Bound    harbor/harbor-scandata-pvc                                                 30h
persistentvolume/harbor-trivy-pv                            5Gi        RWO            Retain           Bound    harbor/harbor-trivy-pvc                                                    31h

NAME                                           STATUS   VOLUME                  CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/harbor-chartmuseum-pvc   Bound    harbor-chartmuseum-pv   5Gi        RWO                           31h
persistentvolumeclaim/harbor-database-pvc      Bound    harbor-database-pv      5Gi        RWO                           31h
persistentvolumeclaim/harbor-jobservice-pvc    Bound    harbor-jobservice-pv    5Gi        RWO                           31h
persistentvolumeclaim/harbor-redis-pvc         Bound    harbor-redis-pv         5Gi        RWO                           31h
persistentvolumeclaim/harbor-registry-pvc      Bound    harbor-registry-pv      5Gi        RWO                           31h
persistentvolumeclaim/harbor-scandata-pvc      Bound    harbor-scandata-pv      5Gi        RWO                           30h
persistentvolumeclaim/harbor-trivy-pvc         Bound    harbor-trivy-pv         5Gi        RWO                           31h

二.安装harbor

1.添加harbor的helm源

[root@k8s-master helm]# helm repo add harbor https://helm.goharbor.io
[root@k8s-master helm]# helm search repo
NAME            CHART VERSION   APP VERSION     DESCRIPTION
harbor/harbor   1.11.0          2.7.0           An open source trusted cloud native registry th...
[root@k8s-master helm]# helm pull harbor/harbor
[root@k8s-master helm]# tar zxf harbor-1.11.0.tgz
[root@k8s-master opt]# cd harbor/
[root@k8s-master harbor]# ls
cert  Chart.yaml  conf  LICENSE  README.md  templates  values.yaml

2.前方高能，修改values.yaml

这里用默认类型 ingress，https协议，如果想用http协议，那么需要删除掉“expose.ingress.annotations”里面的“ssl-redirect”相关注解

[root@k8s-master harbor]# vim values.yaml
expose:
  # Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
  # and fill the information in the corresponding section
  type: ingress
  tls:
    # Enable TLS or not.
    # Delete the "ssl-redirect" annotations in "expose.ingress.annotations" when TLS is disabled and "expose.type" is "ingress"
    # Note: if the "expose.type" is "ingress" and TLS is disabled,
    # the port must be included in the command when pulling/pushing images.
    # Refer to https://github.com/goharbor/harbor/issues/5291 for details.
    enabled: true  
    # The source of the tls certificate. Set as "auto", "secret"
    # or "none" and fill the information in the corresponding section
    # 1) auto: generate the tls certificate automatically
    # 2) secret: read the tls certificate from the specified secret.
    # The tls certificate can be generated manually or by cert manager
    # 3) none: configure no tls certificate for the ingress. If the default
    # tls certificate is configured in the ingress controller, choose this option
    certSource: auto

修改持久化配置，将pvc name添加在existingClaim后面

persistence:
  enabled: true
  # Setting it to "keep" to avoid removing PVCs during a helm delete
  # operation. Leaving it empty will delete PVCs after the chart deleted
  # (this does not apply for PVCs that are created for internal database
  # and redis components, i.e. they are never deleted automatically)
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound,
      # and specify the "subPath" if the PVC is shared with other components
      existingClaim: "harbor-registry-pvc"
      # Specify the "storageClass" used to provision the volume. Or the default

      # StorageClass will be used (the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    chartmuseum:
      existingClaim: "harbor-chartmuseum-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: "harbor-jobservice-pvc"
        storageClass: ""
        subPath: ""
        accessMode: ReadWriteOnce
        size: 5Gi
        annotations: {}
      scanDataExports:
        existingClaim: "harbor-scandata-pvc"
        storageClass: ""
        subPath: ""
        accessMode: ReadWriteOnce
        size: 5Gi
        annotations: {}
    # If external database is used, the following settings for database will
    # be ignored
    database:
      existingClaim: "harbor-database-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    # If external Redis is used, the following settings for Redis will
    # be ignored
    redis:
      existingClaim: "harbor-redis-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    trivy:
      existingClaim: "harbor-trivy-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}

3.helm安装harbor

[root@k8s-master harbor]# helm install harbor . -f values.yaml -n harbor

Have a cup of coffee

[root@k8s-master harbor]# kubectl get pod -n harbor -o wide
NAME                                    READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
harbor-chartmuseum-7b74f8b585-qcp89     1/1     Running   2          4h39m   10.244.107.231   k8s-node3   <none>           <none>
harbor-core-7fc48678c8-qcdqk            1/1     Running   2          4h39m   10.244.107.233   k8s-node3   <none>           <none>
harbor-database-0                       1/1     Running   2          4h39m   10.244.107.236   k8s-node3   <none>           <none>
harbor-jobservice-8486bb4bcb-2gjtp      1/1     Running   9          4h39m   10.244.107.237   k8s-node3   <none>           <none>
harbor-notary-server-7f7bf8f6d-zphpg    1/1     Running   2          4h39m   10.244.107.244   k8s-node3   <none>           <none>
harbor-notary-signer-5f9df848b7-skpxj   1/1     Running   2          4h39m   10.244.107.248   k8s-node3   <none>           <none>
harbor-portal-748c6db9c6-qw97j          1/1     Running   2          4h39m   10.244.107.243   k8s-node3   <none>           <none>
harbor-redis-0                          1/1     Running   2          4h39m   10.244.107.228   k8s-node3   <none>           <none>
harbor-registry-6777c99d8d-cb9rj        2/2     Running   4          4h39m   10.244.107.235   k8s-node3   <none>           <none>
harbor-trivy-0                          1/1     Running   2          4h39m   10.244.107.245   k8s-node3   <none>           <none>

4.ingress绑定ingress class

[root@k8s-master harbor]# kubectl get ingress -n harbor
NAME                    CLASS    HOSTS                  ADDRESS                     PORTS     AGE
harbor-ingress          <none>   core.harbor.domain     192.168.0.21,192.168.0.22   80, 443   4h55m
[root@k8s-master harbor]# kubectl edit ingress harbor-ingress -n harbor
......
spec:
  ingressClassName: nginx            ## k8s-1.20.X添加ingressClassName
  rules:
  - host: core.harbor.domain
    http:
      paths:
......
[root@k8s-master harbor]# kubectl get ingress -n harbor   ##查看添加成功
NAME                    CLASS    HOSTS                  ADDRESS                     PORTS     AGE
harbor-ingress          nginx    core.harbor.domain     192.168.0.21,192.168.0.22   80, 443   4h57m

5.服务器配置证书，不然docker login和docker pull 都会报错

在harbor-ingress 的secret里面找到data下的ca.crt

[root@k8s-master harbor]# kubectl get secret harbor-ingress -n harbor -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRY3d4c0NtVmNPV2RpZWJTSVREcU9kakFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05Nakl4TWpJNE1EWXdOVE13V2hjTk1qTXhNakk0TURZdwpOVE13V2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRQ3g0YVBFbkJ5K2xicElsSytyZmlsdndIWkJRMzM1a3lxMThKWHhpcDl1SFQ0VkExUzEKbG45anVtQUR4Smk4QnhWZStmVlBDWW5UeGJoNWJua3pqdm9XekptOTlMRUY5UDJPdW5VZnZCWGt6VFZEK0o1TgpBZjhyMmtqMFRTck5yZVVKL3JwR01mT1V1MFRSSjg2N2g1WHg1aEpBeDNlWlk4LzRFWmZQOVFGRlFsVGRxRDFBCmlkUkRydjFCSGJIajNxVjU3ckI4L3VxMVJ1a2hjQUZodlZXKzMvdmNkelBaS0hjc0c4d3VhNml2ekExK096a1gKTFA2K2wvcnlPektWWG9kU3ZHV1RmVzJpSlFEUjJqSWliR1loSUF3SDIrZlU4MVJrdGNsS050b2hZYW5aYjdOdQpTcTlzOFB1b3Y2V1JDMWpwRUd6cjhtR2hKTEhtcHJMNlRXeDdBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVVZc2xqdmpDZ3F5R2NRZ1ZWS0s5TnIreU8yWll3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFKd1NkMkFuMmpEdHp6NUIvOEl2UzliamxVN3hLVlR0M1pWWkc0N0R5aFJXTmRpVEs1NFF3dFdhCmtCYUZrV2xTQWJaUk5iOWwyak85NEFMelRCb0ttVU04QTBlZFBQbXdvTFFaZlV2NVp6SCtGM05PenNVT1I5dFIKbnhva3lkSU14WUlpQWp1YThudXRLK0g2TklsRDRiVVA1MVZSNFJsNjBhdHhSWi81OW1HMWw1ZUNVSWtObVJyaQpwY2FQTjBBa3NzR2dCNmpxK3VXMXhPU0l1M21idVl3amdzTEpqWTFkNkNuS1IvQ1RRZnRBKzB2NFNVRTJBTFRGClg0TEFRVXFsM0F2NU8wWkxFQXAzNE5aR0xGNCtnRmJuYi84MmdYTlBuY2IzQkxOR0tKVE5zL0RGTTBkYlgwajcKY3VIMHQvT01oV2YyZ084eTNBMDB6dkIzcTVZSkIwdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVekNDQWp1Z0F3SUJBZ0lSQU9QeW44V1FlOVZOaUlYL21LSGhrSHN3RFFZSktvWklodmNOQVFFTEJRQXcKRkRFU01CQUdBMVVFQXhNSmFHRnlZbTl5TFdOaE1CNFhEVEl5TVRJeU9EQTJNRFV6TUZvWERUSXpNVEl5T0RBMgpNRFV6TUZvd0hURWJNQmtHQTFVRUF4TVNZMjl5WlM1b1lYSmliM0l1Wkc5dFlXbHVNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2dno0N0l4d2JUZDBQOFcvWngxUTVYaS9IRWZLajJmRzFlK08KQ0RMb2psSWEvbmErUlpaRzZvSVNIbGt6TVo2cTh3WmhZeTRVRTFCQi9VM1hKK2xuNHNhTkgrVllTQjdDbmFuYQpiWW5DVkp2Y091d1J3LzVaNUJzU1UreE55U0tLd3dGN0hLcDZBdklIc1puaU85aHhMNUVKRjRmbmtHY3BQZlZKCjNQTzdMY3B4ZDBVcEJ0enJyYnBWWTV1eDlmY3ZKSWNFeEJTY3ZmaGxmN3VjR0pYSDBZa0xra3RJUVAveWdRekMKQWMvUG82NDJ6VUJqNE1RWGVaRVQ2WjRxaWtLWnc4VXRGTyttZmZBU3BPT3FmMkpncDJiaU4yR1VqR044MTkxTAo1NU1BUUY3aFBzUWo0Y0k2aEVWczk3VmtXci9ZKzdxN3EyNkZnQ1IxY0p6ZVBQY1puUUlEQVFBQm80R1dNSUdUCk1BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXcKREFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVdnQlJpeVdPK01LQ3JJWnhDQlZVb3IwMnY3STdabGpBegpCZ05WSFJFRUxEQXFnaEpqYjNKbExtaGhjbUp2Y2k1a2IyMWhhVzZDRkc1dmRHRnllUzVvWVhKaWIzSXVaRzl0CllXbHVNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUJpK3pFdFllS3A3VnRnamo1Vk9PMXUzRERLSmJFaXZYRFYKbG9ZeVRjQzdWRGg3YlF5UmhPVEtja2pJeGd6cFpubjUyczYvSHYxbkQ3MTk0UXFieE0yOGI1OEpxUEtvS0E4aQpWNHEzSE9xUGhlMlRoS1M4ZUt4anl4R0ZRYi8rQ2prdXNlTDV0bjVFbFM1SjBIRU93K2crZjk0WUdMMTFhdkNuCklnazdDbjQwTWovUUg1YTdhaC9lenNad3Y1aFJISHQrc2pyeE5TYWFTcXNjbHQ4bEV2M3VQWFVsdHBrRm1VY2IKc3hOM1lwRDg4d05LWElQSnU5bWpGMnFJUUdUYXBtME04a3lvTGZpMHlrdDlWM1I1cXoybVcwQWx5d3dob0NvMQpTTm9KbUJIOE42bnhqU1JPUzVJS01XU0RhYWtLc1ZVWGhiUjJRQVlyYVkvZ3YrWnhmZEdlCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdnZ6NDdJeHdiVGQwUDhXL1p4MVE1WGkvSEVmS2oyZkcxZStPQ0RMb2psSWEvbmErClJaWkc2b0lTSGxrek1aNnE4d1poWXk0VUUxQkIvVTNYSitsbjRzYU5IK1ZZU0I3Q25hbmFiWW5DVkp2Y091d1IKdy81WjVCc1NVK3hOeVNLS3d3RjdIS3A2QXZJSHNabmlPOWh4TDVFSkY0Zm5rR2NwUGZWSjNQTzdMY3B4ZDBVcApCdHpycmJwVlk1dXg5ZmN2SkljRXhCU2N2ZmhsZjd1Y0dKWEgwWWtMa2t0SVFQL3lnUXpDQWMvUG82NDJ6VUJqCjRNUVhlWkVUNlo0cWlrS1p3OFV0Rk8rbWZmQVNwT09xZjJKZ3AyYmlOMkdVakdOODE5MUw1NU1BUUY3aFBzUWoKNGNJNmhFVnM5N1ZrV3IvWSs3cTdxMjZGZ0NSMWNKemVQUGNablFJREFRQUJBb0lCQUEyOEVZYXBYdGhyMlg5UQprRUF4R29VMkZzQkk1V2RFdGtiaVVGdUVsYmJPYnNTSDg0ejdqcVFFc0pyZ0hVOWZNcm9CWm5XRWRjc2h4VzBhCjhQTWMxT3k1REtNVGtqSzFpNWRkQktsRjgrU3p4TFcwRGw4QzVxSUIxR0tXcHMxbEg0ZUFqc0x6KzR4RXJZcjMKSHc3Z3ROc1AyaENJSkgzOC9UbEliNFJsUHJ1L0tIV2p1SzJQWWQweFdzdDBtWVZBMytpYW9jWWJXTk1BV3p5cAo0WmR0c1Via2MrU3JPVUIvM2hrNE9iNUtkbUM3RE5aMTZJTnlSNGdpdXliSFI0Wk9JaHh1NGsyenFhVDBoU2J5CjREaUEyNytsczNWcGxETHhmanlKUVQyMEZHVW5jNTllb1pzdzBUeHZSUjM5cEFJQW9IdUhlL0ZCQW1RRnhZSEEKMS9qNUp5RUNnWUVBMnFnVDd4Sms4WFNLcStwenhDSTV4TlpTRVp0OU5WeVFZUUFzM25ubkxhemp0UUUwSGJGeQpWbk9ja1h6ZFd4Wnd5WUtUTktSb0ViSGE3V0NUVmd0T3RrR3VJWCtnenprQTRQaEI1YmhFNmp2bUhzS0laQkkvCmN5Z1lvMENZY0dZTGl5YnpRV0VLTzRPRHg1TlVxZmY2OTVRVVRGek1ORE10eSsvRnozc3lYNVVDZ1lFQTM1c3oKUmZHa0Qzd1lDTSszbWR3VldOZnRjbkNBYmw1YXppM3BWRG4yR2JRNC9GYVZkYVNmT3Z0Vk1oRUZpZkZGS2RzRApSNXJ3SmdXNmJSRUg0V1JOSEQxUzZ2WHc5ZnJJRkZFMnBRa2JZb01iV0tsUlFMNTlSYlRMQnEzQVJnV3Q4MXBYCmJtOXNNeFNBS2NBaVF6dGFxYXFTVDhOQjBHY1U0MjZSK3ZEYTcra0NnWUVBbFVRMlQzZE1KN1Q2VVgvOHhZRGoKMU9iR0liVDIxYTI1OWk0TGMxamVvMFNxTWM5L2gyR3lmeWZ2VXpaZFdpaEltSmVsN0VMcnRHQ281bkdPUXlmMgo5TjZEZytTL241YjNiWnlzUjZqeWlzQ1hTSnBlUjRwWmZFQjhDVDQ0a2twblNQZ3ZDWXU0VTRabE9LSHdJeFBoClpJL1hCNFkxOU5DWFMrZ0VMcTZZWmIwQ2dZRUExVVpsNEVlN2tOMXUyekNzVVQ5K3lPK2pWaTBDQXNOU1h5ZWgKVHFtK0Z3UXorbExuV1g4OE5QaTJhUHVkU2RYcmZ5R2JmamZFNks4OEFuMWxBOUUwVDBRYWkyc3JlcUxKSmIvVQpuQk1Vb0tDbWU2bDdpNEpsWUJBeU9kdU44ZnZHejc4U2Q0NGxLSTljTXZaRWQ5WHNBcnBqdFZwcXNza3ZQa0lmClVMZGNTOGtDZ1lCejZ2QlJqc2VlZnpySXVSRjR5UWorcUhZM2cxYXR2VDBMYlh5MWVvTk5RdW92ZXhoZmduTHoKangyVVB0Z29VMjlrMUpOWWhzMjByWVBhS1NDa2c5ZjNqUk1NNmZOc2M4QWQ5RFNJa3FaeS9rQkI0YjJPMDdKdwo2cVBRZEhyemJDK0Q2VUZhakFJMEFoVVV1WUZXbFRPaW5uLzlEc0p4Ky9CbEZSQzJFZTJtNEE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

转码

[root@k8s-master harbor]# echo 'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRY3d4c0NtVmNPV2RpZWJTSVREcU9kakFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05Nakl4TWpJNE1EWXdOVE13V2hjTk1qTXhNakk0TURZdwpOVE13V2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRQ3g0YVBFbkJ5K2xicElsSytyZmlsdndIWkJRMzM1a3lxMThKWHhpcDl1SFQ0VkExUzEKbG45anVtQUR4Smk4QnhWZStmVlBDWW5UeGJoNWJua3pqdm9XekptOTlMRUY5UDJPdW5VZnZCWGt6VFZEK0o1TgpBZjhyMmtqMFRTck5yZVVKL3JwR01mT1V1MFRSSjg2N2g1WHg1aEpBeDNlWlk4LzRFWmZQOVFGRlFsVGRxRDFBCmlkUkRydjFCSGJIajNxVjU3ckI4L3VxMVJ1a2hjQUZodlZXKzMvdmNkelBaS0hjc0c4d3VhNml2ekExK096a1gKTFA2K2wvcnlPektWWG9kU3ZHV1RmVzJpSlFEUjJqSWliR1loSUF3SDIrZlU4MVJrdGNsS050b2hZYW5aYjdOdQpTcTlzOFB1b3Y2V1JDMWpwRUd6cjhtR2hKTEhtcHJMNlRXeDdBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVVZc2xqdmpDZ3F5R2NRZ1ZWS0s5TnIreU8yWll3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFKd1NkMkFuMmpEdHp6NUIvOEl2UzliamxVN3hLVlR0M1pWWkc0N0R5aFJXTmRpVEs1NFF3dFdhCmtCYUZrV2xTQWJaUk5iOWwyak85NEFMelRCb0ttVU04QTBlZFBQbXdvTFFaZlV2NVp6SCtGM05PenNVT1I5dFIKbnhva3lkSU14WUlpQWp1YThudXRLK0g2TklsRDRiVVA1MVZSNFJsNjBhdHhSWi81OW1HMWw1ZUNVSWtObVJyaQpwY2FQTjBBa3NzR2dCNmpxK3VXMXhPU0l1M21idVl3amdzTEpqWTFkNkNuS1IvQ1RRZnRBKzB2NFNVRTJBTFRGClg0TEFRVXFsM0F2NU8wWkxFQXAzNE5aR0xGNCtnRmJuYi84MmdYTlBuY2IzQkxOR0tKVE5zL0RGTTBkYlgwajcKY3VIMHQvT01oV2YyZ084eTNBMDB6dkIzcTVZSkIwdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=' | base64 -d > ca.crt
-----BEGIN CERTIFICATE-----
MIIDEzCCAfugAwIBAgIQcwxsCmVcOWdiebSITDqOdjANBgkqhkiG9w0BAQsFADAU
MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMjIxMjI4MDYwNTMwWhcNMjMxMjI4MDYw
NTMwWjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCx4aPEnBy+lbpIlK+rfilvwHZBQ335kyq18JXxip9uHT4VA1S1
ln9jumADxJi8BxVe+fVPCYnTxbh5bnkzjvoWzJm99LEF9P2OunUfvBXkzTVD+J5N
Af8r2kj0TSrNreUJ/rpGMfOUu0TRJ867h5Xx5hJAx3eZY8/4EZfP9QFFQlTdqD1A
idRDrv1BHbHj3qV57rB8/uq1RukhcAFhvVW+3/vcdzPZKHcsG8wua6ivzA1+OzkX
LP6+l/ryOzKVXodSvGWTfW2iJQDR2jIibGYhIAwH2+fU81RktclKNtohYanZb7Nu
Sq9s8Puov6WRC1jpEGzr8mGhJLHmprL6TWx7AgMBAAGjYTBfMA4GA1UdDwEB/wQE
AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw
AwEB/zAdBgNVHQ4EFgQUYsljvjCgqyGcQgVVKK9Nr+yO2ZYwDQYJKoZIhvcNAQEL
BQADggEBAJwSd2An2jDtzz5B/8IvS9bjlU7xKVTt3ZVZG47DyhRWNdiTK54QwtWa
kBaFkWlSAbZRNb9l2jO94ALzTBoKmUM8A0edPPmwoLQZfUv5ZzH+F3NOzsUOR9tR
nxokydIMxYIiAjua8nutK+H6NIlD4bUP51VR4Rl60atxRZ/59mG1l5eCUIkNmRri
pcaPN0AkssGgB6jq+uW1xOSIu3mbuYwjgsLJjY1d6CnKR/CTQftA+0v4SUE2ALTF
X4LAQUql3Av5O0ZLEAp34NZGLF4+gFbnb/82gXNPncb3BLNGKJTNs/DFM0dbX0j7
cuH0t/OMhWf2gO8y3A00zvB3q5YJB0w=
-----END CERTIFICATE-----

将证书保存在docker目录下，创建根域名相同的子目录，这一步建议所有机器都执行

[root@k8s-master harbor]#  mkdir -p /etc/docker/certs.d/core.harbor.domain
[root@k8s-master harbor]# cp ca.crt /etc/docker/certs.d/core.harbor.domain/

6.配置host

建议所有机器执行
将harbor域名解析到ingress-nginx的svc的clusterIP，实现内网使用harbor的功能

[root@k8s-master harbor]# kubectl get svc -n ingress-nginx
NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller-admission-nginx   ClusterIP   10.106.125.71    <none>        443/TCP                      48d
ingress-nginx-controller-nginx             NodePort    10.108.233.252   <none>        80:31203/TCP,443:32292/TCP   48d
[root@k8s-master harbor]# echo '10.108.233.252 core.harbor.domain' >> /etc/hosts

三.测试

1.浏览器访问harbor

修改本机host

47.92.*.*  core.harbor.domain

查看ingress-nginx暴露的https协议端口，这里我们能看到https协议的端口是32292

[root@k8s-master harbor]# kubectl get svc -n ingress-nginx
NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller-admission-nginx   ClusterIP   10.106.125.71    <none>        443/TCP                      48d
ingress-nginx-controller-nginx             NodePort    10.108.233.252   <none>        80:31203/TCP,443:32292/TCP   48d

浏览器访问
在这里插入图片描述

2.服务器内部访问harbor

[root@k8s-master harbor]# docker login https://core.harbor.domain
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

3.上传镜像测试

这里使用我另外一篇文章制作的docker images，文章详细记录了dockerfile制作nginx镜像的过程
https://blog.csdn.net/weixin_43866248/article/details/128411117
另，推送命令可以在harbor的项目管理界面找到
在这里插入图片描述

[root@k8s-master harbor]# docker images
REPOSITORY                                                           TAG              IMAGE ID       CREATED         SIZE
nginx-1.16.1                                                         v1               507f86395522   5 days ago      471MB
centos7-base                                                         v1               c31fb99ef249   6 days ago      432MB
[root@k8s-master harbor]# docker tag nginx-1.16.1:v1 core.harbor.domain/test/nginx-1.16.1:v1
[root@k8s-master harbor]# docker tag centos7-base:v1 core.harbor.domain/test/centos7-base:v1
[root@k8s-master harbor]# docker images
REPOSITORY                                                           TAG              IMAGE ID       CREATED         SIZE
core.harbor.domain/test/nginx-1.16.1                                 v1               507f86395522   5 days ago      471MB
nginx-1.16.1                                                         v1               507f86395522   5 days ago      471MB
core.harbor.domain/test/centos7-base                                 v1               c31fb99ef249   6 days ago      432MB
centos7-base                                                         v1               c31fb99ef249   6 days ago      432MB
[root@k8s-master harbor]# docker push core.harbor.domain/test/nginx-1.16.1:v1
The push refers to repository [core.harbor.domain/test/nginx-1.16.1]
d2f532751821: Pushed
0049738c43da: Pushed
457ed1526068: Pushed
174f56854903: Pushed
v1: digest: sha256:94449ea711d91742ffe6ea07546cce43b4c26aabc809b1a6824f903bb60aa8c7 size: 1164
[root@k8s-master harbor]# docker push core.harbor.domain/test/centos7-base:v1
The push refers to repository [core.harbor.domain/test/centos7-base]
457ed1526068: Mounted from test/nginx-1.16.1
174f56854903: Mounted from test/nginx-1.16.1
v1: digest: sha256:d97f75eb40907e9a3b1b8624be318fb8722a3412ac536c510e838c3408643f23 size: 741

4.下载镜像测试

在k8s-node1节点下载镜像

[root@k8s-node1 ~]# docker login https://core.harbor.domain
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@k8s-node1 ~]# docker pull core.harbor.domain/test/nginx-1.16.1:v1
v1: Pulling from test/nginx-1.16.1
Digest: sha256:94449ea711d91742ffe6ea07546cce43b4c26aabc809b1a6824f903bb60aa8c7
Status: Downloaded newer image for core.harbor.domain/test/nginx-1.16.1:v1
core.harbor.domain/test/nginx-1.16.1:v1
[root@k8s-node1 ~]# docker images
REPOSITORY                                                                       TAG        IMAGE ID       CREATED         SIZE
core.harbor.domain/test/nginx-1.16.1                                             v1         507f86395522   5 days ago      471MB

5.创建一个工作负载测试

注意yaml文件里有个imagePullSecrets的参数，这是因为我在harbor里面创建了个访问级别是私有的test项目
在这里插入图片描述

[root@k8s-master test-namespace]# vim nginx-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s.kuboard.cn/name: nginx-test
  name: nginx-test
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s.kuboard.cn/name: nginx-test
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s.kuboard.cn/name: nginx-test
    spec:
      containers:
        - image: 'core.harbor.domain/test/nginx-1.16.1:v1'
          imagePullPolicy: Always
          name: nginx
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: harbor-secret
[root@k8s-master test-namespace]# vim harbor-secret.yaml   ## 这个里面是harbor的认证文件
apiVersion: v1 
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2NvcmUuaGFyYm9yLmRvbWFpbiI6eyJ1c2VybmFtZSI6ImFkbWluIiwicGFzc3dvcmQiOiJIYXJib3IxMjM0NSIsImF1dGgiOiJZV1J0YVc0NlNHRnlZbTl5TVRJek5EVT0ifX19
immutable: false
kind: Secret
metadata:
  name: harbor-secret
  namespace: test
type: kubernetes.io/dockerconfigjson
[root@k8s-master test-namespace]# kubectl apply -f harbor-secret.yaml
[root@k8s-master test-namespace]# kubectl apply -f nginx-test.yaml
[root@k8s-master test-namespace]# kubectl get pod -n test ## 创建成功
NAME                         READY   STATUS    RESTARTS   AGE
nginx-test-67f899486-tc95g   1/1     Running   0          21m

如果文章有遗漏，请评论或者私信
感谢

kubernetes docker Helm
更新于 2023-02-01
“ 💋 戏 子. ✟

查看shares更多相关的文章或提一个关于shares的问题，也可以与我们一起分享文章。