|
@@ -0,0 +1,785 @@
|
|
|
+> [TOC]
|
|
|
+
|
|
|
+# 1、安装前的准备工作
|
|
|
+
|
|
|
+## 1.1、准备三台服务器
|
|
|
+
|
|
|
+虚拟机的最低标准
|
|
|
+**master**
|
|
|
+至少4核心
|
|
|
+内存至少8G
|
|
|
+存储空间80G以上
|
|
|
+**node**
|
|
|
+至少2核心
|
|
|
+内存至少4G
|
|
|
+存储空间60G以上
|
|
|
+
|
|
|
+
|
|
|
+需要准备三台服务器并设置静态IP,这里不再赘述。本文档的配置如下
|
|
|
+
|
|
|
+| 节点名称 | ip |
|
|
|
+| -------- | -------------- |
|
|
|
+| master | 192.168.238.20 |
|
|
|
+| node1 | 192.168.238.21 |
|
|
|
+| node2 | 192.168.238.22 |
|
|
|
+
|
|
|
+## 1.2、安装前的准备工作
|
|
|
+
|
|
|
+```sh
|
|
|
+# 关闭防火墙
|
|
|
+systemctl stop firewalld && systemctl disable firewalld
|
|
|
+
|
|
|
+# 关闭selinux
|
|
|
+sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
|
|
|
+setenforce 0 # 临时
|
|
|
+cat /etc/selinux/config
|
|
|
+
|
|
|
+# 关闭swap
|
|
|
+swapoff -a # 临时
|
|
|
+sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
|
|
|
+free -m
|
|
|
+
|
|
|
+# 根据规划设置主机名
|
|
|
+hostnamectl set-hostname <hostname>
|
|
|
+
|
|
|
+# 在master添加hosts
|
|
|
+cat >> /etc/hosts << EOF
|
|
|
+192.168.238.20 master
|
|
|
+192.168.238.21 node1
|
|
|
+192.168.238.22 node2
|
|
|
+EOF
|
|
|
+
|
|
|
+# 将桥接的IPv4流量传递到iptables的链
|
|
|
+cat > /etc/sysctl.d/k8s.conf << EOF
|
|
|
+net.bridge.bridge-nf-call-ip6tables = 1
|
|
|
+net.bridge.bridge-nf-call-iptables = 1
|
|
|
+EOF
|
|
|
+sysctl --system # 生效
|
|
|
+
|
|
|
+# 修改当前时间为北京时间
|
|
|
+# 查看当前系统时间
|
|
|
+date
|
|
|
+# 修改当前系统时间
|
|
|
+date -s "2018-2-22 19:10:30
|
|
|
+# 查看硬件时间
|
|
|
+hwclock --show
|
|
|
+# 修改硬件时间
|
|
|
+hwclock --set --date "2018-2-22 19:10:30"
|
|
|
+# 同步系统时间和硬件时间
|
|
|
+hwclock --hctosys
|
|
|
+# 保存时钟
|
|
|
+clock -w
|
|
|
+```
|
|
|
+
|
|
|
+上述操作执行完毕后重启
|
|
|
+```sh
|
|
|
+reboot
|
|
|
+```
|
|
|
+三台服务器,开始下一步的安装
|
|
|
+
|
|
|
+
|
|
|
+# 2、离线安装Docker及其相应组件(三台服务机都需要安装)
|
|
|
+
|
|
|
+## 2.1、离线安装Docker
|
|
|
+
|
|
|
+```sh
|
|
|
+# 1.创建目录存放相应的安装包
|
|
|
+mkdir -p /opt/package/docker
|
|
|
+
|
|
|
+# 2.上传安装包下的docker文件夹到上述的目录中
|
|
|
+
|
|
|
+# 3.进入目录,进行解压缩操作
|
|
|
+cd /opt/package/docker
|
|
|
+unzip docker19-rpm.zip
|
|
|
+
|
|
|
+# 4.安装docker
|
|
|
+rpm -ivh *.rpm --force --nodeps
|
|
|
+systemctl enable docker && systemctl start docker
|
|
|
+
|
|
|
+# 5.查看是否安装成功
|
|
|
+docker --version
|
|
|
+```
|
|
|
+
|
|
|
+输出docker版本号表示安装成功
|
|
|
+![image-20220313152413610](assets/image-20220313152413610.png)
|
|
|
+
|
|
|
+## 2.2、离线安装Docker-compose
|
|
|
+
|
|
|
+将安装包目录下的docker-compose-linux-x86_64文件上传到服务机的```/opt/package```目录下使用命令
|
|
|
+```sh
|
|
|
+// 1.将下载好的文件传入linux系统中,并重命名未docker-compose
|
|
|
+mv docker-compose-linux-x86_64 docker-compose
|
|
|
+// 2.给予docker-compose文件可执行权限
|
|
|
+chmod u+x docker-compose
|
|
|
+// 3.将docker-compose文件移至/usr/local/bin
|
|
|
+mv docker-compose /usr/local/bin
|
|
|
+// 4.查看版本
|
|
|
+docker-compose --version
|
|
|
+```
|
|
|
+输出docker-compose版本号表示安装成功
|
|
|
+![image-20211108154405864](assets/image-20211108154405864.png)
|
|
|
+
|
|
|
+# 3、安装Harbor及相应的配置(只需要一台服务机安装)
|
|
|
+
|
|
|
+## 3.0、安装注意事项
|
|
|
+
|
|
|
+本文档Harbor安装在单独一台虚拟机上面,如果只有三台机器,请将下面的server.harbor.com换成固定的IP(这里的固定IP意思是安装harbor的虚拟机的IP),例如将harbor仓库安装在master节点就将server.harbor.com换为192.168.238.20。
|
|
|
+
|
|
|
+## 3.1、安装Harbor
|
|
|
+
|
|
|
+1. 将压缩包harbor-offline-installer-v2.1.5.tgz上传到
|
|
|
+```sh
|
|
|
+/opt/package/
|
|
|
+```
|
|
|
+目录下
|
|
|
+2. 解压该压缩包
|
|
|
+```sh
|
|
|
+tar xf harbor-offline-installer-v2.1.5.tgz
|
|
|
+```
|
|
|
+3. 修改harbor安装的配置文件
|
|
|
+
|
|
|
+首先备份一份压缩包
|
|
|
+```sh
|
|
|
+# 复制配置文件内容到harbor.yml 中(安装时只识别harbor.yml)
|
|
|
+cp harbor.yml.tmpl harbor.yml
|
|
|
+# 用于存放harbor的持久化数据
|
|
|
+mkdir -p /opt/package/harbor/data
|
|
|
+
|
|
|
+# 用于存放harbor的日志
|
|
|
+mkdir -p /opt/package/harbor/log
|
|
|
+```
|
|
|
+
|
|
|
+其次对harbor.yml文件进行修改配置
|
|
|
+
|
|
|
+```
|
|
|
+# 需要修改为域名,如果没有单独的机器,请修改为IP
|
|
|
+hostname: server.harbor.com
|
|
|
+
|
|
|
+# http related config
|
|
|
+http:
|
|
|
+ port: 80
|
|
|
+# 需要全部注释
|
|
|
+# https related config
|
|
|
+# https:
|
|
|
+ # https port for harbor, default is 443
|
|
|
+ # port: 443
|
|
|
+ # The path of cert and key files for nginx
|
|
|
+ # certificate: /your/certificate/path
|
|
|
+ # private_key: /your/private/key/path
|
|
|
+
|
|
|
+data_volume: /opt/package/harbor/data # 需要添加一个自己的目录
|
|
|
+
|
|
|
+log:
|
|
|
+ location: /opt/package/harbor/log # 需要添加一个自己的目录
|
|
|
+```
|
|
|
+
|
|
|
+4. 安装并启动Harbor
|
|
|
+
|
|
|
+保证此时在harbor安装文件中,执行install.sh文件进行安装,命令为:
|
|
|
+```
|
|
|
+./install.sh
|
|
|
+```
|
|
|
+
|
|
|
+5. 访问harbor Web界面
|
|
|
+
|
|
|
+通过自己的ip+端口访问
|
|
|
+
|
|
|
+![image-20211207125209937](assets/image-20211207125209937.png)
|
|
|
+
|
|
|
+## 3.2、配置Docker访问Harbor(三台服务机都要配置)
|
|
|
+
|
|
|
+首先修改服务机的hosts,如果没有单独的harbor服务器,请忽略这一步
|
|
|
+```
|
|
|
+# 将下面的ip缓存harbor的ip
|
|
|
+echo "10.168.59.60 server.harbor.com">> /etc/hosts
|
|
|
+```
|
|
|
+
|
|
|
+docker添加harbor配置-----注意这里要加harbor的端口号,这里配置的端口号为上述harbor配置文件的端口号
|
|
|
+```sh
|
|
|
+mkdir -p /etc/docker
|
|
|
+
|
|
|
+tee /etc/docker/daemon.json <<-'EOF'
|
|
|
+{
|
|
|
+ "insecure-registries": ["server.harbor.com:80"]
|
|
|
+}
|
|
|
+EOF
|
|
|
+
|
|
|
+systemctl daemon-reload && systemctl restart docker
|
|
|
+
|
|
|
+# 如果是ip方式
|
|
|
+mkdir -p /etc/docker
|
|
|
+
|
|
|
+tee /etc/docker/daemon.json <<-'EOF'
|
|
|
+{
|
|
|
+ "insecure-registries": ["192.168.238.20:80", "192.168.238.20", "lab3", "la"]
|
|
|
+}
|
|
|
+EOF
|
|
|
+
|
|
|
+systemctl daemon-reload && systemctl restart docker
|
|
|
+```
|
|
|
+
|
|
|
+输入命令
|
|
|
+```sh
|
|
|
+# 重新启动harbor,因为docker重启但是harbor不能自动重启
|
|
|
+cd /opt/package/
|
|
|
+docker-compose start
|
|
|
+# 登录harbor
|
|
|
+docker login server.harbor.com:80
|
|
|
+```
|
|
|
+输入用户名:admin
|
|
|
+密码:Harbor12345
|
|
|
+![image-20211110155332792](assets/image-20211110155332792.png)
|
|
|
+至此,harbor配置完成
|
|
|
+
|
|
|
+# 4、K8S 离线安装及NFS配置(三台服务机都需要安装)
|
|
|
+
|
|
|
+
|
|
|
+## 4.1 、上传kube1.9.0.tar.gz(以下简称kube1.9)到服务器
|
|
|
+
|
|
|
+```
|
|
|
+# 1.三台服务机创建目录
|
|
|
+mkdir -p /opt/package/k8s
|
|
|
+# 2.上传文件到指定目录
|
|
|
+scp -r kube1.9.0.tar.gz root@192.168.238.20:/opt/package/k8s
|
|
|
+scp -r kube1.9.0.tar.gz root@192.168.238.21:/opt/package/k8s
|
|
|
+scp -r kube1.9.0.tar.gz root@192.168.238.22:/opt/package/k8s
|
|
|
+```
|
|
|
+
|
|
|
+## 4.2 、解压安装主从
|
|
|
+
|
|
|
+```sh
|
|
|
+# 1.master下,进入/opt/package/k8s目录下解压,执行脚本
|
|
|
+tar -zxvf kube1.19.0.tar.gz
|
|
|
+cd kube/shell
|
|
|
+chmod +x init.sh
|
|
|
+./init.sh
|
|
|
+
|
|
|
+# 2.等待init.sh执行完毕,之后执行
|
|
|
+chmod +x master.sh
|
|
|
+./master.sh
|
|
|
+
|
|
|
+# 3. node1和node2执行下面的命令
|
|
|
+cd /opt/package/k8s
|
|
|
+tar -zxvf kube1.9.0.tar.gz
|
|
|
+cd kube/shell
|
|
|
+chmod +x init.sh
|
|
|
+./init.sh
|
|
|
+
|
|
|
+# 4. 进入master节点将/etc/kubernetes/admin.conf文件复制到node1节点和node2节点
|
|
|
+scp -r /etc/kubernetes/admin.conf node1:/etc/kubernetes/
|
|
|
+scp -r /etc/kubernetes/admin.conf node2:/etc/kubernetes/
|
|
|
+
|
|
|
+```
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+## 4.3 、从节点加入主节点
|
|
|
+
|
|
|
+ 1. 在master节点生成token
|
|
|
+
|
|
|
+ ```sh
|
|
|
+ kubeadm token create --print-join-command
|
|
|
+ ```
|
|
|
+
|
|
|
+ 效果如下
|
|
|
+
|
|
|
+ ```sh
|
|
|
+ [root@master shell]# kubeadm token create --print-join-command
|
|
|
+ W1009 17:15:31.782757 37720 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
|
|
|
+ kubeadm join 192.168.0.90:6443 --token ul68zs.dkkvpwfex9rpzo0d --discovery-token-ca-cert-hash sha256:3e3ee481f5603621f216e707321aa26a68834939e440be91322c62eb8540ffce
|
|
|
+ ```
|
|
|
+
|
|
|
+ 2. 在node1和node2中执行下面的命令------注意这里要上面生产的命令
|
|
|
+
|
|
|
+ ```sh
|
|
|
+ kubeadm join 192.168.0.90:6443 --token ul68zs.dkkvpwfex9rpzo0d --discovery-token-ca-cert-hash sha256:3e3ee481f5603621f216e707321aa26a68834939e440be91322c62eb8540ffce
|
|
|
+ ```
|
|
|
+
|
|
|
+ 结果如下
|
|
|
+
|
|
|
+ ```sh
|
|
|
+ [root@node1 shell]# kubeadm join 192.168.0.90:6443 --token ul68zs.dkkvpwfex9rpzo0d --discovery-token-ca-cert-hash sha256:3e3ee481f5603621f216e707321aa26a68834939e440be91322c62eb8540ffce
|
|
|
+ [preflight] Running pre-flight checks
|
|
|
+ [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
|
|
|
+ [WARNING FileExisting-socat]: socat not found in system path
|
|
|
+ [preflight] Reading configuration from the cluster...
|
|
|
+ [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
|
|
|
+ [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
|
|
|
+ [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
|
|
|
+ [kubelet-start] Starting the kubelet
|
|
|
+ [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
|
|
|
+
|
|
|
+ This node has joined the cluster:
|
|
|
+ * Certificate signing request was sent to apiserver and a response was received.
|
|
|
+ * The Kubelet was informed of the new secure connection details.
|
|
|
+
|
|
|
+ Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
|
|
|
+ ```
|
|
|
+
|
|
|
+3. 给node1和node2添加执行权限
|
|
|
+
|
|
|
+```sh
|
|
|
+# 在node1和node2执行一下命令
|
|
|
+mkdir -p $HOME/.kube
|
|
|
+sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
|
|
|
+sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
|
|
+kubectl get nodes
|
|
|
+# 注意上面的admin.conf是从master节点复制过来的
|
|
|
+```
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+## 4.4、 验证集群pod是否running,各节点是否ready
|
|
|
+
|
|
|
+watch kubectl get pod -n kube-system -o wide
|
|
|
+效果如下
|
|
|
+
|
|
|
+```sh
|
|
|
+ [root@master shell]# watch kubectl get pod -n kube-system -o wide
|
|
|
+ Every 2.0s: kubectl get pod -n kube-system -o wide Fri Oct 9 17:45:03 2020
|
|
|
+ NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
|
|
+ calico-kube-controllers-5d7686f694-94fcc 1/1 Running 0 48m 100.89.161.131 master <none> <none>
|
|
|
+ calico-node-42bwj 1/1 Running 0 48m 192.168.0.90 master <none> <none>
|
|
|
+ calico-node-k6k6d 1/1 Running 0 27m 192.168.0.189 node2 <none> <none>
|
|
|
+ calico-node-lgwwj 1/1 Running 0 29m 192.168.0.68 node1 <none> <none>
|
|
|
+ coredns-f9fd979d6-2ncmm 1/1 Running 0 48m 100.89.161.130 master <none> <none>
|
|
|
+ coredns-f9fd979d6-5s4nw 1/1 Running 0 48m 100.89.161.129 master <none> <none>
|
|
|
+ etcd-master 1/1 Running 0 48m 192.168.0.90 master <none> <none>
|
|
|
+ kube-apiserver-master 1/1 Running 0 48m 192.168.0.90 master <none> <none>
|
|
|
+ kube-controller-manager-master 1/1 Running 0 48m 192.168.0.90 master <none> <none>
|
|
|
+ kube-proxy-5g2ht 1/1 Running 0 29m 192.168.0.68 node1 <none> <none>
|
|
|
+ kube-proxy-wpf76 1/1 Running 0 27m 192.168.0.189 node2 <none> <none>
|
|
|
+ kube-proxy-zgcft 1/1 Running 0 48m 192.168.0.90 master <none> <none>
|
|
|
+ kube-scheduler-master 1/1 Running 0 48m 192.168.0.90 master <none> <none>
|
|
|
+```
|
|
|
+kubectl get nodes
|
|
|
+效果如下
|
|
|
+```sh
|
|
|
+ [root@master shell]# kubectl get nodes
|
|
|
+ NAME STATUS ROLES AGE VERSION
|
|
|
+ master Ready master 22m v1.19.0
|
|
|
+ node1 Ready <none> 2m17s v1.19.0
|
|
|
+ node2 Ready <none> 24s v1.19.0
|
|
|
+```
|
|
|
+
|
|
|
+## 4.5、配置NFS
|
|
|
+
|
|
|
+| master | NFS服务端+NFS客户端 |
|
|
|
+| ------ | ------------------- |
|
|
|
+| node1 | NFS客户端 |
|
|
|
+| node2 | NFS客户端 |
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+1. 将本地nfs离线包上传至服务器及各个节点的/opt/package/nfs文件夹下
|
|
|
+
|
|
|
+ ```sh
|
|
|
+ scp -r nfs master:/opt/package
|
|
|
+ scp -r nfs node1:/opt/package
|
|
|
+ scp -r nfs n:/opt/package
|
|
|
+ ```
|
|
|
+
|
|
|
+2. 安装服务端(master节点操作)
|
|
|
+
|
|
|
+ ```sh
|
|
|
+ # 进入master节点内/opt/package/nfs文件夹内执行以下命令
|
|
|
+ # 进入/opt/package/nfs
|
|
|
+ cd /opt/package/nfs
|
|
|
+ # 安装nfs
|
|
|
+ rpm -ivh *.rpm --force --nodeps # 注意如果不能安装选择 nfs.zip文件
|
|
|
+ # 执行命令 vi /etc/exports,创建 exports 文件,文件内容如下:
|
|
|
+ echo "/nfs/data/ *(insecure,rw,sync,no_root_squash)" > /etc/exports
|
|
|
+ # 执行以下命令,启动 nfs 服务
|
|
|
+ # 创建共享目录
|
|
|
+ mkdir -p /nfs/data
|
|
|
+ systemctl enable rpcbind
|
|
|
+ systemctl enable nfs-server
|
|
|
+ systemctl start rpcbind
|
|
|
+ systemctl start nfs-server
|
|
|
+ exportfs -r
|
|
|
+ # 检查配置是否生效
|
|
|
+ exportfs
|
|
|
+ # 输出结果如下所示
|
|
|
+ /nfs/data <world>
|
|
|
+ ```
|
|
|
+3. 安装NFS客户端(三台服务机操作)
|
|
|
+```sh
|
|
|
+# 进入node1,node2节点内/opt/package/nfs文件夹内执行以下命令
|
|
|
+cd /opt/package/nfs
|
|
|
+rpm -ivh *.rpm --force --nodeps # 注意如果不能安装选择 nfs.zip文件
|
|
|
+systemctl start nfs && systemctl enable nfs
|
|
|
+```
|
|
|
+
|
|
|
+4. K8S中安装NFS(任意K8S节点,这里选择master节点)
|
|
|
+```sh
|
|
|
+# 1.进入/opt/package/nfs目录
|
|
|
+cd /opt/package/nfs
|
|
|
+# 2.载入docker镜像
|
|
|
+docker load < nfs-client-provisioner.tar.gz
|
|
|
+# 3.修改deployment.yaml文件
|
|
|
+vim /opt/package/nfs/deployment.yaml
|
|
|
+
|
|
|
+apiVersion: apps/v1
|
|
|
+kind: Deployment
|
|
|
+metadata:
|
|
|
+ name: nfs-client-provisioner
|
|
|
+ labels:
|
|
|
+ app: nfs-client-provisioner
|
|
|
+ # replace with namespace where provisioner is deployed
|
|
|
+ namespace: default
|
|
|
+spec:
|
|
|
+ replicas: 1
|
|
|
+ strategy:
|
|
|
+ type: Recreate
|
|
|
+ selector:
|
|
|
+ matchLabels:
|
|
|
+ app: nfs-client-provisioner
|
|
|
+ template:
|
|
|
+ metadata:
|
|
|
+ labels:
|
|
|
+ app: nfs-client-provisioner
|
|
|
+ spec:
|
|
|
+ serviceAccountName: nfs-client-provisioner
|
|
|
+ containers:
|
|
|
+ - name: nfs-client-provisioner
|
|
|
+ image: quay.io/external_storage/nfs-client-provisioner:latest ##默认是latest版本
|
|
|
+ imagePullPolicy: Never ## 这里选择从本地拉取
|
|
|
+ volumeMounts:
|
|
|
+ - name: nfs-client-root
|
|
|
+ mountPath: /persistentvolumes
|
|
|
+ env:
|
|
|
+ - name: PROVISIONER_NAME
|
|
|
+ value: fuseim.pri/ifs
|
|
|
+ - name: NFS_SERVER
|
|
|
+ value: 192.168.238.20 ##这里写NFS服务器的IP地址
|
|
|
+ - name: NFS_PATH
|
|
|
+ value: /nfs/data ##这里写NFS服务器中的共享挂载目录(强调:这里的路径必须是目录中最后一层的文件夹,否则部署的应用将无权限创建目录导致Pending)
|
|
|
+ volumes:
|
|
|
+ - name: nfs-client-root
|
|
|
+ nfs:
|
|
|
+ server: 192.168.238.20 ##这里写NFS服务器的IP地址
|
|
|
+ path: /nfs/data ##NFS服务器中的共享挂载目录(强调:这里的路径必须是目录中最后一层的文件夹,否则部署的应用将无权限创建目录导致Pending)
|
|
|
+
|
|
|
+# 4、部署yaml文件
|
|
|
+[root@k8s-client nfs]# kubectl apply -f .
|
|
|
+
|
|
|
+# 5、查看服务
|
|
|
+[root@k8s-client nfs]# kubectl get pods
|
|
|
+NAME READY STATUS RESTARTS AGE
|
|
|
+nfs-client-provisioner-78697f488-5p52r 1/1 Running 0 16h
|
|
|
+
|
|
|
+# 6、列出你的集群中的StorageClass
|
|
|
+[root@k8s-client nfs]# kubectl get storageclass
|
|
|
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
|
|
|
+managed-nfs-storage fuseim.pri/ifs Delete Immediate false 16h
|
|
|
+
|
|
|
+# 7、标记一个StorageClass为默认的 (是storageclass的名字也就是你部署的StorageClass名字是啥就写啥)
|
|
|
+[root@k8s-client nfs]# kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
|
|
|
+storageclass.storage.k8s.io/managed-nfs-storage patched
|
|
|
+
|
|
|
+# 8、验证你选用为默认的StorageClass
|
|
|
+[root@k8s-client nfs]# kubectl get storageclass
|
|
|
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
|
|
|
+managed-nfs-storage (default) fuseim.pri/ifs Delete Immediate false
|
|
|
+```
|
|
|
+
|
|
|
+# 5、安装kubesphere及其相应的插件
|
|
|
+
|
|
|
+## 5.1、将离线包上传至harbor仓库(该操作只需在master节点进行)
|
|
|
+
|
|
|
+将安装包目录下的kubesphere文件夹上传至服务机master节点的
|
|
|
+```sh
|
|
|
+/opt/package/
|
|
|
+```
|
|
|
+目录下
|
|
|
+
|
|
|
+进入harbor仓库
|
|
|
+![image-20211110230053303](assets/image-20211110230053303.png)
|
|
|
+新建一个kubesphere项目
|
|
|
+![image-20211111101212415](assets/image-20211111101212415.png)
|
|
|
+
|
|
|
+```sh
|
|
|
+# 在服务机master节点中执行命令
|
|
|
+# 重新启动harbor,因为docker重启但是harbor不能自动重启
|
|
|
+cd /opt/package/harbor
|
|
|
+docker-compose start
|
|
|
+# 进入该路径
|
|
|
+cd /opt/package/kubesphere/
|
|
|
+
|
|
|
+# 上传安装包 这里的最后一行改成自己harbor仓库的ip+端口号+项目名称
|
|
|
+chmod +x offline-installation-tool.sh
|
|
|
+./offline-installation-tool.sh -l images-list.txt -d ./kubesphere-images -r server.harbor.com:80/kubesphere
|
|
|
+# 等待上传完毕
|
|
|
+```
|
|
|
+![image-20211110230429326](assets/image-20211110230429326.png)
|
|
|
+
|
|
|
+## 5.2、最小化安装kubesphere
|
|
|
+
|
|
|
+```sh
|
|
|
+# 执行以下命令
|
|
|
+# 1.编辑cluster-configuration.yaml添加您的私有镜像仓库
|
|
|
+vim cluster-configuration.yaml
|
|
|
+spec:
|
|
|
+ persistence:
|
|
|
+ storageClass: ""
|
|
|
+ authentication:
|
|
|
+ jwtSecret: ""
|
|
|
+ local_registry: server.harbor.com:80/kubesphere #添加内容
|
|
|
+
|
|
|
+# 2.编辑完成后保存 cluster-configuration.yaml,使用以下命令将 ks-installer 替换为您自己仓库的地址---(本文的harbor安装地址server.harbor.com:80/kubesphere)
|
|
|
+sed -i "s#^\s*image: kubesphere.*/ks-installer:.*# image: server.harbor.com:80/kubesphere/kubesphere/ks-installer:v3.1.1#" kubesphere-installer.yaml
|
|
|
+
|
|
|
+# 3.请按照如下先后顺序安装(必须)
|
|
|
+kubectl apply -f kubesphere-installer.yaml
|
|
|
+kubectl get pods -A
|
|
|
+```
|
|
|
+![image-20211110231108730](assets/image-20211110231108730-16365570741011.png)
|
|
|
+![image-20211110231233626](assets/image-20211110231233626.png)
|
|
|
+
|
|
|
+```sh
|
|
|
+# 4.等待ks-installer容器运行完毕,执行
|
|
|
+kubectl apply -f cluster-configuration.yaml
|
|
|
+```
|
|
|
+
|
|
|
+## 5.3、检查安装日志
|
|
|
+
|
|
|
+```sh
|
|
|
+# 检查安装日志等待安装成功
|
|
|
+kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
|
|
|
+```
|
|
|
+如下图所示,则表示安装成功
|
|
|
+![image-20211111101337952](assets/image-20211111101337952.png)
|
|
|
+
|
|
|
+
|
|
|
+## 5.4、安装必要的插件
|
|
|
+
|
|
|
+本次需要安装额外的几个插件
|
|
|
+1. KubeSphere日志系统
|
|
|
+2. KubeSphere DevOps
|
|
|
+3. KubeSphere kubeedge
|
|
|
+
|
|
|
+```sh
|
|
|
+# 1.编辑cluster-configuration.yaml
|
|
|
+vim cluster-configuration.yaml
|
|
|
+
|
|
|
+devops:
|
|
|
+ enabled: true # 将“false”更改为“true”。
|
|
|
+
|
|
|
+kubeedge:
|
|
|
+ enabled: true # 将“false”更改为“true”。
|
|
|
+
|
|
|
+logging:
|
|
|
+ enabled: true # 将“false”更改为“true”。
|
|
|
+
|
|
|
+# 2.执行以下命令开始安装
|
|
|
+kubectl apply -f cluster-configuration.yaml
|
|
|
+
|
|
|
+# 3.监控安装过程
|
|
|
+kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
|
|
|
+```
|
|
|
+如下图所示,表示安装成功
|
|
|
+![image-20211111102817757](assets/image-20211111102817757.png)
|
|
|
+![image-20211111102801129](assets/image-20211111102801129.png)
|
|
|
+
|
|
|
+## 5.5、开放30881端口(重要--master节点操作)
|
|
|
+
|
|
|
+```sh
|
|
|
+kubectl -n kubesphere-system patch svc ks-apiserver -p '{"spec":{"type":"NodePort","ports":[{"port":80,"protocal":"TCP","targetPort":9090,"nodePort":30881}]}}'
|
|
|
+```
|
|
|
+
|
|
|
+![image-20211115213927460](assets/image-20211115213927460.png)
|
|
|
+
|
|
|
+## 5.6、创建云平台的企业空间
|
|
|
+
|
|
|
+1. 登录kubesphere,输入http://10.168.59.60:30880/,端口号默认为30880,第一次进入需要修改初始密码为: MKcloud123
|
|
|
+![image-20211115214619362](assets/image-20211115214619362.png)
|
|
|
+2. 进入企业空间,创建新的企业空间为mkcloud
|
|
|
+![image-20211115214723424](assets/image-20211115214723424.png)
|
|
|
+3. 进入企业空间mkcloud,创建项目mkcloud
|
|
|
+![image-20211115214817599](assets/image-20211115214817599.png)
|
|
|
+4. 进入项目选择默认设置即可
|
|
|
+![image-20211115214846424](assets/image-20211115214846424.png)
|
|
|
+![image-20211115214912342](assets/image-20211115214912342.png)
|
|
|
+![image-20211115214937388](assets/image-20211115214937388.png)
|
|
|
+
|
|
|
+## 5.7、kubesphere添加Harbor仓库
|
|
|
+
|
|
|
+1. 进入当前项目
|
|
|
+![image-20211118211440018](assets/image-20211118211440018.png)
|
|
|
+2. 选择配置中心的密钥,选择添加密钥
|
|
|
+![image-20211118211533239](assets/image-20211118211533239.png)
|
|
|
+3. 密钥名称填写为harbor,仓库地址填写域名加端口号
|
|
|
+![image-20211118211659229](assets/image-20211118211659229.png)
|
|
|
+4. 添加成功会显示私有密钥
|
|
|
+![image-20211118211733125](assets/image-20211118211733125.png)
|
|
|
+
|
|
|
+# 6、卸载云平台环境
|
|
|
+
|
|
|
+***
|
|
|
+
|
|
|
+## 6.1、卸载K8S
|
|
|
+
|
|
|
+```shell
|
|
|
+kubeadm reset -f
|
|
|
+modprobe -r ipip
|
|
|
+lsmod
|
|
|
+rm -rf ~/.kube/
|
|
|
+rm -rf /etc/kubernetes/
|
|
|
+rm -rf /etc/systemd/system/kubelet.service.d
|
|
|
+rm -rf /etc/systemd/system/kubelet.service
|
|
|
+rm -rf /usr/bin/kube*
|
|
|
+rm -rf /etc/cni
|
|
|
+rm -rf /opt/cni
|
|
|
+rm -rf /var/lib/etcd
|
|
|
+rm -rf /var/etcd
|
|
|
+yum clean all
|
|
|
+yum remove kube*
|
|
|
+```
|
|
|
+
|
|
|
+## 6.2、卸载docker
|
|
|
+
|
|
|
+```shell
|
|
|
+# 1.停止docker
|
|
|
+sudo systemctl stop docker
|
|
|
+# 2.查询安装过的包
|
|
|
+yum list installed | grep docker
|
|
|
+# 3.删除安装的软件包
|
|
|
+yum -y remove docker*
|
|
|
+# 4.删除镜像/容器等
|
|
|
+rm -rf /var/lib/docker
|
|
|
+```
|
|
|
+
|
|
|
+# 7、目前遇见的BUG与解决方案
|
|
|
+
|
|
|
+***
|
|
|
+
|
|
|
+## 7.1、harbor仓库登录显示400错误
|
|
|
+
|
|
|
+![image-20211123200112617](assets/image-20211123200112617.png)
|
|
|
+```sh
|
|
|
+Error response from daemon: login attempt to https://server.harbor.com/v2/ failed with status: 400 Bad Request
|
|
|
+```
|
|
|
+
|
|
|
+查询相关文档发现是nginx的配置出现了问题
|
|
|
+[Docker 登录返回 400 错误请求](https://github.com/goharbor/harbor/issues/7159)
|
|
|
+
|
|
|
+```sh
|
|
|
+#修改方式如下
|
|
|
+#1.进入harbor仓库位置
|
|
|
+cd /opt/package/software/harbor
|
|
|
+#2.修改配置文件
|
|
|
+vim ./common/config/portal/nginx.conf
|
|
|
+#3.增加一个字段
|
|
|
+http {
|
|
|
+ client_body_temp_path /tmp/client_body_temp;
|
|
|
+ proxy_temp_path /tmp/proxy_temp;
|
|
|
+ fastcgi_temp_path /tmp/fastcgi_temp;
|
|
|
+ uwsgi_temp_path /tmp/uwsgi_temp;
|
|
|
+ scgi_temp_path /tmp/scgi_temp;
|
|
|
+ large_client_header_buffers 4 1k; #增加的
|
|
|
+
|
|
|
+#4.重启nginx即可
|
|
|
+docker ps
|
|
|
+#执行的结果e939b007b6ca goharbor/harbor-core:v2.3.2 "/harbor/entrypoint.…"
|
|
|
+
|
|
|
+docker restart e39629e9c2c4
|
|
|
+```
|
|
|
+
|
|
|
+## 7.2、单节点安装pod报错Pending
|
|
|
+
|
|
|
+![image-20220317164430599](assets/image-20220317164430599.png)
|
|
|
+
|
|
|
+```sh
|
|
|
+# 当创建单机版的 k8s 时,这个时候 master 节点是默认不允许调度 pod 的,需要执行
|
|
|
+kubectl taint nodes --all node-role.kubernetes.io/master-
|
|
|
+```
|
|
|
+
|
|
|
+## 7.3、kubesphere的api-server无法启动
|
|
|
+
|
|
|
+api-server显示报错---无法申请资源,原因是docker的进程数不够,增加一下的配置重启docker就行
|
|
|
+
|
|
|
+```shell
|
|
|
+echo "
|
|
|
+[Unit]
|
|
|
+Description=Docker Application Container Engine
|
|
|
+Documentation=https://docs.docker.com
|
|
|
+# BindsTo=containerd.service
|
|
|
+# After=network-online.target firewalld.service containerd.service
|
|
|
+After=network-online.target firewalld.service
|
|
|
+Wants=network-online.target
|
|
|
+# Requires=docker.socket
|
|
|
+
|
|
|
+[Service]
|
|
|
+Type=notify
|
|
|
+# the default is not to use systemd for cgroups because the delegate issues still
|
|
|
+# exists and systemd currently does not support the cgroup feature set required
|
|
|
+# for containers run by docker
|
|
|
+# ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
|
|
|
+ExecStart=/usr/local/bin/dockerd
|
|
|
+ExecReload=/bin/kill -s HUP $MAINPID
|
|
|
+TasksMax=infinity
|
|
|
+TimeoutSec=0
|
|
|
+RestartSec=2
|
|
|
+Restart=always
|
|
|
+
|
|
|
+# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
|
|
|
+# Both the old, and new location are accepted by systemd 229 and up, so using the old location
|
|
|
+# to make them work for either version of systemd.
|
|
|
+StartLimitBurst=3
|
|
|
+
|
|
|
+# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
|
|
|
+# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
|
|
|
+# this option work for either version of systemd.
|
|
|
+StartLimitInterval=60s
|
|
|
+
|
|
|
+# Having non-zero Limit*s causes performance problems due to accounting overhead
|
|
|
+# in the kernel. We recommend using cgroups to do container-local accounting.
|
|
|
+LimitNOFILE=infinity
|
|
|
+LimitNPROC=infinity
|
|
|
+LimitCORE=infinity
|
|
|
+
|
|
|
+# Comment TasksMax if your systemd version does not support it.
|
|
|
+# Only systemd 226 and above support this option.
|
|
|
+# TasksMax=infinity
|
|
|
+
|
|
|
+# set delegate yes so that systemd does not reset the cgroups of docker containers
|
|
|
+Delegate=yes
|
|
|
+
|
|
|
+# kill only the docker process, not all processes in the cgroup
|
|
|
+KillMode=process
|
|
|
+
|
|
|
+[Install]
|
|
|
+WantedBy=multi-user.target
|
|
|
+" > /etc/systemd/system/docker.service
|
|
|
+
|
|
|
+# 重启docker
|
|
|
+systemctl daemon-reload && systemctl restart docker
|
|
|
+```
|
|
|
+
|
|
|
+## 7.4、k8s cgroup 内存泄漏
|
|
|
+
|
|
|
+**centos7.6及其以下版本,造成内存泄漏**
|
|
|
+
|
|
|
+```shell
|
|
|
+vim /boot/grub2/grub.cfg
|
|
|
+# 在 ro crashkernel=auto 后面添加 cgroup.memory=nokmem
|
|
|
+# 重启电脑
|
|
|
+
|
|
|
+```
|
|
|
+
|
|
|
+![img](assets/20210509204813641.png)
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+# 附录----安装软件的版本
|
|
|
+
|
|
|
+| 软件 | 版本 |
|
|
|
+| -------------- | ------- |
|
|
|
+| centos | 7.7 |
|
|
|
+| docker | 19.03.7 |
|
|
|
+| docker-compose | 2.1.0 |
|
|
|
+| Harbor | 2.1.5 |
|
|
|
+| kubernetes | 1.19.0 |
|
|
|
+| kubesphere | 3.1.1 |
|
|
|
+
|