> [TOC] # 1、安装前的准备工作 ## 1.1、准备三台服务器 虚拟机的最低标准 **master** 至少4核心 内存至少8G 存储空间80G以上 **node** 至少2核心 内存至少4G 存储空间60G以上 需要准备三台服务器并设置静态IP,这里不再赘述。本文档的配置如下 | 节点名称 | ip | | -------- | -------------- | | master | 192.168.238.20 | | node1 | 192.168.238.21 | | node2 | 192.168.238.22 | ## 1.2、安装前的准备工作 ```sh # 关闭防火墙 systemctl stop firewalld && systemctl disable firewalld # 关闭selinux sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 setenforce 0 # 临时 cat /etc/selinux/config # 关闭swap swapoff -a # 临时 sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久 free -m # 根据规划设置主机名 hostnamectl set-hostname # 在master添加hosts cat >> /etc/hosts << EOF 192.168.238.20 master 192.168.238.21 node1 192.168.238.22 node2 EOF # 将桥接的IPv4流量传递到iptables的链 cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system # 生效 # 修改当前时间为北京时间 # 查看当前系统时间 date # 修改当前系统时间 date -s "2018-2-22 19:10:30 # 查看硬件时间 hwclock --show # 修改硬件时间 hwclock --set --date "2018-2-22 19:10:30" # 同步系统时间和硬件时间 hwclock --hctosys # 保存时钟 clock -w ``` 上述操作执行完毕后重启 ```sh reboot ``` 三台服务器,开始下一步的安装 # 2、离线安装Docker及其相应组件(三台服务机都需要安装) ## 2.1、离线安装Docker ```sh # 1.创建目录存放相应的安装包 mkdir -p /opt/package/docker # 2.上传安装包下的docker文件夹到上述的目录中 # 3.进入目录,进行解压缩操作 cd /opt/package/docker unzip docker19-rpm.zip # 4.安装docker rpm -ivh *.rpm --force --nodeps systemctl enable docker && systemctl start docker # 5.查看是否安装成功 docker --version ``` 输出docker版本号表示安装成功 ![image-20220313152413610](assets/image-20220313152413610.png) ## 2.2、离线安装Docker-compose 将安装包目录下的docker-compose-linux-x86_64文件上传到服务机的```/opt/package```目录下使用命令 ```sh // 1.将下载好的文件传入linux系统中,并重命名未docker-compose mv docker-compose-linux-x86_64 docker-compose // 2.给予docker-compose文件可执行权限 chmod u+x docker-compose // 3.将docker-compose文件移至/usr/local/bin mv docker-compose /usr/local/bin // 4.查看版本 docker-compose --version ``` 输出docker-compose版本号表示安装成功 ![image-20211108154405864](assets/image-20211108154405864.png) # 3、安装Harbor及相应的配置(只需要一台服务机安装) ## 3.0、安装注意事项 本文档Harbor安装在单独一台虚拟机上面,如果只有三台机器,请将下面的server.harbor.com换成固定的IP(这里的固定IP意思是安装harbor的虚拟机的IP),例如将harbor仓库安装在master节点就将server.harbor.com换为192.168.238.20。 ## 3.1、安装Harbor 1. 将压缩包harbor-offline-installer-v2.1.5.tgz上传到 ```sh /opt/package/ ``` 目录下 2. 解压该压缩包 ```sh tar xf harbor-offline-installer-v2.1.5.tgz ``` 3. 修改harbor安装的配置文件 首先备份一份压缩包 ```sh # 复制配置文件内容到harbor.yml 中(安装时只识别harbor.yml) cp harbor.yml.tmpl harbor.yml # 用于存放harbor的持久化数据 mkdir -p /opt/package/harbor/data # 用于存放harbor的日志 mkdir -p /opt/package/harbor/log ``` 其次对harbor.yml文件进行修改配置 ``` # 需要修改为域名,如果没有单独的机器,请修改为IP hostname: server.harbor.com # http related config http: port: 80 # 需要全部注释 # https related config # https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /your/certificate/path # private_key: /your/private/key/path data_volume: /opt/package/harbor/data # 需要添加一个自己的目录 log: location: /opt/package/harbor/log # 需要添加一个自己的目录 ``` 4. 安装并启动Harbor 保证此时在harbor安装文件中,执行install.sh文件进行安装,命令为: ``` ./install.sh ``` 5. 访问harbor Web界面 通过自己的ip+端口访问 ![image-20211207125209937](assets/image-20211207125209937.png) ## 3.2、配置Docker访问Harbor(三台服务机都要配置) 首先修改服务机的hosts,如果没有单独的harbor服务器,请忽略这一步 ``` # 将下面的ip缓存harbor的ip echo "10.168.59.60 server.harbor.com">> /etc/hosts ``` docker添加harbor配置-----注意这里要加harbor的端口号,这里配置的端口号为上述harbor配置文件的端口号 ```sh mkdir -p /etc/docker tee /etc/docker/daemon.json <<-'EOF' { "insecure-registries": ["server.harbor.com:80"] } EOF systemctl daemon-reload && systemctl restart docker # 如果是ip方式 mkdir -p /etc/docker tee /etc/docker/daemon.json <<-'EOF' { "insecure-registries": ["192.168.238.20:80", "192.168.238.20", "lab3", "la"] } EOF systemctl daemon-reload && systemctl restart docker ``` 输入命令 ```sh # 重新启动harbor,因为docker重启但是harbor不能自动重启 cd /opt/package/ docker-compose start # 登录harbor docker login server.harbor.com:80 ``` 输入用户名:admin 密码:Harbor12345 ![image-20211110155332792](assets/image-20211110155332792.png) 至此,harbor配置完成 # 4、K8S 离线安装及NFS配置(三台服务机都需要安装) ## 4.1 、上传kube1.9.0.tar.gz(以下简称kube1.9)到服务器 ``` # 1.三台服务机创建目录 mkdir -p /opt/package/k8s # 2.上传文件到指定目录 scp -r kube1.9.0.tar.gz root@192.168.238.20:/opt/package/k8s scp -r kube1.9.0.tar.gz root@192.168.238.21:/opt/package/k8s scp -r kube1.9.0.tar.gz root@192.168.238.22:/opt/package/k8s ``` ## 4.2 、解压安装主从 ```sh # 1.master下,进入/opt/package/k8s目录下解压,执行脚本 tar -zxvf kube1.19.0.tar.gz cd kube/shell chmod +x init.sh ./init.sh # 2.等待init.sh执行完毕,之后执行 chmod +x master.sh ./master.sh # 3. node1和node2执行下面的命令 cd /opt/package/k8s tar -zxvf kube1.9.0.tar.gz cd kube/shell chmod +x init.sh ./init.sh # 4. 进入master节点将/etc/kubernetes/admin.conf文件复制到node1节点和node2节点 scp -r /etc/kubernetes/admin.conf node1:/etc/kubernetes/ scp -r /etc/kubernetes/admin.conf node2:/etc/kubernetes/ ``` ## 4.3 、从节点加入主节点 1. 在master节点生成token ```sh kubeadm token create --print-join-command ``` 效果如下 ```sh [root@master shell]# kubeadm token create --print-join-command W1009 17:15:31.782757 37720 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] kubeadm join 192.168.0.90:6443 --token ul68zs.dkkvpwfex9rpzo0d --discovery-token-ca-cert-hash sha256:3e3ee481f5603621f216e707321aa26a68834939e440be91322c62eb8540ffce ``` 2. 在node1和node2中执行下面的命令------注意这里要上面生产的命令 ```sh kubeadm join 192.168.0.90:6443 --token ul68zs.dkkvpwfex9rpzo0d --discovery-token-ca-cert-hash sha256:3e3ee481f5603621f216e707321aa26a68834939e440be91322c62eb8540ffce ``` 结果如下 ```sh [root@node1 shell]# kubeadm join 192.168.0.90:6443 --token ul68zs.dkkvpwfex9rpzo0d --discovery-token-ca-cert-hash sha256:3e3ee481f5603621f216e707321aa26a68834939e440be91322c62eb8540ffce [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING FileExisting-socat]: socat not found in system path [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` 3. 给node1和node2添加执行权限 ```sh # 在node1和node2执行一下命令 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config kubectl get nodes # 注意上面的admin.conf是从master节点复制过来的 ``` ## 4.4、 验证集群pod是否running,各节点是否ready watch kubectl get pod -n kube-system -o wide 效果如下 ```sh [root@master shell]# watch kubectl get pod -n kube-system -o wide Every 2.0s: kubectl get pod -n kube-system -o wide Fri Oct 9 17:45:03 2020 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-5d7686f694-94fcc 1/1 Running 0 48m 100.89.161.131 master calico-node-42bwj 1/1 Running 0 48m 192.168.0.90 master calico-node-k6k6d 1/1 Running 0 27m 192.168.0.189 node2 calico-node-lgwwj 1/1 Running 0 29m 192.168.0.68 node1 coredns-f9fd979d6-2ncmm 1/1 Running 0 48m 100.89.161.130 master coredns-f9fd979d6-5s4nw 1/1 Running 0 48m 100.89.161.129 master etcd-master 1/1 Running 0 48m 192.168.0.90 master kube-apiserver-master 1/1 Running 0 48m 192.168.0.90 master kube-controller-manager-master 1/1 Running 0 48m 192.168.0.90 master kube-proxy-5g2ht 1/1 Running 0 29m 192.168.0.68 node1 kube-proxy-wpf76 1/1 Running 0 27m 192.168.0.189 node2 kube-proxy-zgcft 1/1 Running 0 48m 192.168.0.90 master kube-scheduler-master 1/1 Running 0 48m 192.168.0.90 master ``` kubectl get nodes 效果如下 ```sh [root@master shell]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 22m v1.19.0 node1 Ready 2m17s v1.19.0 node2 Ready 24s v1.19.0 ``` ## 4.5、配置NFS | master | NFS服务端+NFS客户端 | | ------ | ------------------- | | node1 | NFS客户端 | | node2 | NFS客户端 | 1. 将本地nfs离线包上传至服务器及各个节点的/opt/package/nfs文件夹下 ```sh scp -r nfs master:/opt/package scp -r nfs node1:/opt/package scp -r nfs n:/opt/package ``` 2. 安装服务端(master节点操作) ```sh # 进入master节点内/opt/package/nfs文件夹内执行以下命令 # 进入/opt/package/nfs cd /opt/package/nfs # 安装nfs rpm -ivh *.rpm --force --nodeps # 注意如果不能安装选择 nfs.zip文件 # 执行命令 vi /etc/exports,创建 exports 文件,文件内容如下: echo "/nfs/data/ *(insecure,rw,sync,no_root_squash)" > /etc/exports # 执行以下命令,启动 nfs 服务 # 创建共享目录 mkdir -p /nfs/data systemctl enable rpcbind systemctl enable nfs-server systemctl start rpcbind systemctl start nfs-server exportfs -r # 检查配置是否生效 exportfs # 输出结果如下所示 /nfs/data ``` 3. 安装NFS客户端(三台服务机操作) ```sh # 进入node1,node2节点内/opt/package/nfs文件夹内执行以下命令 cd /opt/package/nfs rpm -ivh *.rpm --force --nodeps # 注意如果不能安装选择 nfs.zip文件 systemctl start nfs && systemctl enable nfs ``` 4. K8S中安装NFS(任意K8S节点,这里选择master节点) ```sh # 1.进入/opt/package/nfs目录 cd /opt/package/nfs # 2.载入docker镜像 docker load < nfs-client-provisioner.tar.gz # 3.修改deployment.yaml文件 vim /opt/package/nfs/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest ##默认是latest版本 imagePullPolicy: Never ## 这里选择从本地拉取 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.238.20 ##这里写NFS服务器的IP地址 - name: NFS_PATH value: /nfs/data ##这里写NFS服务器中的共享挂载目录(强调:这里的路径必须是目录中最后一层的文件夹,否则部署的应用将无权限创建目录导致Pending) volumes: - name: nfs-client-root nfs: server: 192.168.238.20 ##这里写NFS服务器的IP地址 path: /nfs/data ##NFS服务器中的共享挂载目录(强调:这里的路径必须是目录中最后一层的文件夹,否则部署的应用将无权限创建目录导致Pending) # 4、部署yaml文件 [root@k8s-client nfs]# kubectl apply -f . # 5、查看服务 [root@k8s-client nfs]# kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-78697f488-5p52r 1/1 Running 0 16h # 6、列出你的集群中的StorageClass [root@k8s-client nfs]# kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE managed-nfs-storage fuseim.pri/ifs Delete Immediate false 16h # 7、标记一个StorageClass为默认的 (是storageclass的名字也就是你部署的StorageClass名字是啥就写啥) [root@k8s-client nfs]# kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' storageclass.storage.k8s.io/managed-nfs-storage patched # 8、验证你选用为默认的StorageClass [root@k8s-client nfs]# kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE managed-nfs-storage (default) fuseim.pri/ifs Delete Immediate false ``` # 5、安装kubesphere及其相应的插件 ## 5.1、将离线包上传至harbor仓库(该操作只需在master节点进行) 将安装包目录下的kubesphere文件夹上传至服务机master节点的 ```sh /opt/package/ ``` 目录下 进入harbor仓库 ![image-20211110230053303](assets/image-20211110230053303.png) 新建一个kubesphere项目 ![image-20211111101212415](assets/image-20211111101212415.png) ```sh # 在服务机master节点中执行命令 # 重新启动harbor,因为docker重启但是harbor不能自动重启 cd /opt/package/harbor docker-compose start # 进入该路径 cd /opt/package/kubesphere/ # 上传安装包 这里的最后一行改成自己harbor仓库的ip+端口号+项目名称 chmod +x offline-installation-tool.sh ./offline-installation-tool.sh -l images-list.txt -d ./kubesphere-images -r server.harbor.com:80/kubesphere # 等待上传完毕 ``` ![image-20211110230429326](assets/image-20211110230429326.png) ## 5.2、最小化安装kubesphere ```sh # 执行以下命令 # 1.编辑cluster-configuration.yaml添加您的私有镜像仓库 vim cluster-configuration.yaml spec: persistence: storageClass: "" authentication: jwtSecret: "" local_registry: server.harbor.com:80/kubesphere #添加内容 # 2.编辑完成后保存 cluster-configuration.yaml,使用以下命令将 ks-installer 替换为您自己仓库的地址---(本文的harbor安装地址server.harbor.com:80/kubesphere) sed -i "s#^\s*image: kubesphere.*/ks-installer:.*# image: server.harbor.com:80/kubesphere/kubesphere/ks-installer:v3.1.1#" kubesphere-installer.yaml # 3.请按照如下先后顺序安装(必须) kubectl apply -f kubesphere-installer.yaml kubectl get pods -A ``` ![image-20211110231108730](assets/image-20211110231108730-16365570741011.png) ![image-20211110231233626](assets/image-20211110231233626.png) ```sh # 4.等待ks-installer容器运行完毕,执行 kubectl apply -f cluster-configuration.yaml ``` ## 5.3、检查安装日志 ```sh # 检查安装日志等待安装成功 kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f ``` 如下图所示,则表示安装成功 ![image-20211111101337952](assets/image-20211111101337952.png) ## 5.4、安装必要的插件 本次需要安装额外的几个插件 1. KubeSphere日志系统 2. KubeSphere DevOps 3. KubeSphere kubeedge ```sh # 1.编辑cluster-configuration.yaml vim cluster-configuration.yaml devops: enabled: true # 将“false”更改为“true”。 kubeedge: enabled: true # 将“false”更改为“true”。 logging: enabled: true # 将“false”更改为“true”。 # 2.执行以下命令开始安装 kubectl apply -f cluster-configuration.yaml # 3.监控安装过程 kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f ``` 如下图所示,表示安装成功 ![image-20211111102817757](assets/image-20211111102817757.png) ![image-20211111102801129](assets/image-20211111102801129.png) ## 5.5、开放30881端口(重要--master节点操作) ```sh kubectl -n kubesphere-system patch svc ks-apiserver -p '{"spec":{"type":"NodePort","ports":[{"port":80,"protocal":"TCP","targetPort":9090,"nodePort":30881}]}}' ``` ![image-20211115213927460](assets/image-20211115213927460.png) ## 5.6、创建云平台的企业空间 1. 登录kubesphere,输入http://10.168.59.60:30880/,端口号默认为30880,第一次进入需要修改初始密码为: MKcloud123 ![image-20211115214619362](assets/image-20211115214619362.png) 2. 进入企业空间,创建新的企业空间为mkcloud ![image-20211115214723424](assets/image-20211115214723424.png) 3. 进入企业空间mkcloud,创建项目mkcloud ![image-20211115214817599](assets/image-20211115214817599.png) 4. 进入项目选择默认设置即可 ![image-20211115214846424](assets/image-20211115214846424.png) ![image-20211115214912342](assets/image-20211115214912342.png) ![image-20211115214937388](assets/image-20211115214937388.png) ## 5.7、kubesphere添加Harbor仓库 1. 进入当前项目 ![image-20211118211440018](assets/image-20211118211440018.png) 2. 选择配置中心的密钥,选择添加密钥 ![image-20211118211533239](assets/image-20211118211533239.png) 3. 密钥名称填写为harbor,仓库地址填写域名加端口号 ![image-20211118211659229](assets/image-20211118211659229.png) 4. 添加成功会显示私有密钥 ![image-20211118211733125](assets/image-20211118211733125.png) # 6、卸载云平台环境 *** ## 6.1、卸载K8S ```shell kubeadm reset -f modprobe -r ipip lsmod rm -rf ~/.kube/ rm -rf /etc/kubernetes/ rm -rf /etc/systemd/system/kubelet.service.d rm -rf /etc/systemd/system/kubelet.service rm -rf /usr/bin/kube* rm -rf /etc/cni rm -rf /opt/cni rm -rf /var/lib/etcd rm -rf /var/etcd yum clean all yum remove kube* ``` ## 6.2、卸载docker ```shell # 1.停止docker sudo systemctl stop docker # 2.查询安装过的包 yum list installed | grep docker # 3.删除安装的软件包 yum -y remove docker* # 4.删除镜像/容器等 rm -rf /var/lib/docker ``` # 7、目前遇见的BUG与解决方案 *** ## 7.1、harbor仓库登录显示400错误 ![image-20211123200112617](assets/image-20211123200112617.png) ```sh Error response from daemon: login attempt to https://server.harbor.com/v2/ failed with status: 400 Bad Request ``` 查询相关文档发现是nginx的配置出现了问题 [Docker 登录返回 400 错误请求](https://github.com/goharbor/harbor/issues/7159) ```sh #修改方式如下 #1.进入harbor仓库位置 cd /opt/package/software/harbor #2.修改配置文件 vim ./common/config/portal/nginx.conf #3.增加一个字段 http { client_body_temp_path /tmp/client_body_temp; proxy_temp_path /tmp/proxy_temp; fastcgi_temp_path /tmp/fastcgi_temp; uwsgi_temp_path /tmp/uwsgi_temp; scgi_temp_path /tmp/scgi_temp; large_client_header_buffers 4 1k; #增加的 #4.重启nginx即可 docker ps #执行的结果e939b007b6ca goharbor/harbor-core:v2.3.2 "/harbor/entrypoint.…" docker restart e39629e9c2c4 ``` ## 7.2、单节点安装pod报错Pending ![image-20220317164430599](assets/image-20220317164430599.png) ```sh # 当创建单机版的 k8s 时,这个时候 master 节点是默认不允许调度 pod 的,需要执行 kubectl taint nodes --all node-role.kubernetes.io/master- ``` ## 7.3、kubesphere的api-server无法启动 api-server显示报错---无法申请资源,原因是docker的进程数不够,增加一下的配置重启docker就行 ```shell echo " [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com # BindsTo=containerd.service # After=network-online.target firewalld.service containerd.service After=network-online.target firewalld.service Wants=network-online.target # Requires=docker.socket [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker # ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ExecStart=/usr/local/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID TasksMax=infinity TimeoutSec=0 RestartSec=2 Restart=always # Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229. # Both the old, and new location are accepted by systemd 229 and up, so using the old location # to make them work for either version of systemd. StartLimitBurst=3 # Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230. # Both the old, and new name are accepted by systemd 230 and up, so using the old name to make # this option work for either version of systemd. StartLimitInterval=60s # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not support it. # Only systemd 226 and above support this option. # TasksMax=infinity # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process [Install] WantedBy=multi-user.target " > /etc/systemd/system/docker.service # 重启docker systemctl daemon-reload && systemctl restart docker ``` ## 7.4、k8s cgroup 内存泄漏 **centos7.6及其以下版本,造成内存泄漏** ```shell vim /boot/grub2/grub.cfg # 在 ro crashkernel=auto 后面添加 cgroup.memory=nokmem # 重启电脑 ``` ![img](assets/20210509204813641.png) # 附录----安装软件的版本 | 软件 | 版本 | | -------------- | ------- | | centos | 7.7 | | docker | 19.03.7 | | docker-compose | 2.1.0 | | Harbor | 2.1.5 | | kubernetes | 1.19.0 | | kubesphere | 3.1.1 |