wordpress区块链导航类网站网店代运营公司

wordpress区块链导航类网站,网店代运营公司,医院网站建设方案需要多少钱,外观设计公司在k8s中为什么要做持久化存储#xff1f; 在k8s中部署的应用都是以pod容器的形式运行的#xff0c;假如我们部署MySQL、Redis等数据库#xff0c;需要对这些数据库产生的数据做备份。因为Pod是有生命周期的#xff0c;如果pod不挂载数据卷#xff0c;那pod被删除或重启后…在k8s中为什么要做持久化存储在k8s中部署的应用都是以pod容器的形式运行的假如我们部署MySQL、Redis等数据库需要对这些数据库产生的数据做备份。因为Pod是有生命周期的如果pod不挂载数据卷那pod被删除或重启后这些数据会随之消失如果想要长久的保留这些数据就要用到pod数据持久化存储。1、k8s持久化存储emptyDir查看k8s支持哪些存储[rootk8s-master](mailto:rootk8s-master) ~]# kubectl explain pods.spec.volumes FIELDS: awsElasticBlockStore Object azureDisk Object azureFile Object cephfs Object cinder Object configMapObject csi Object downwardAPI Object emptyDir Object ephemeral Object fc Object flexVolume Object flocker Object gcePersistentDisk Object gitRepo Object glusterfsObject hostPath Object iscsi Object name string -required- nfs Object persistentVolumeClaimObject photonPersistentDisk Object portworxVolume Object projected Object quobyte Object rbd Object scaleIO Object secret Object storageos Object vsphereVolume Object常用的如下emptyDir hostPath nfs persistentVolumeClaim glusterfs cephfs configMap secret我们想要使用存储卷需要经历如下步骤1、定义pod的volume这个volume指明它要关联到哪个存储上的2、在容器中要使用volumemounts挂载对应的存储经过以上两步才能正确的使用存储卷emptyDir类型的Volume是在Pod分配到Node上时被创建Kubernetes会在Node上自动分配一个目录因此无需指定宿主机Node上对应的目录文件。 这个目录的初始内容为空当Pod从Node上移除时emptyDir中的数据会被永久删除。emptyDir Volume主要用于某些应用程序无需永久保存的临时目录多个容器的共享目录等。创建一个pod挂载临时目录emptyDirEmptydir的官方网址https://kubernetes.io/docs/concepts/storage/volumes#emptydir[[rootk8s-master](mailto:rootk8s-master) ~]# cat emptydir.yaml ​ apiVersion: v1 kind: Pod metadata: name: pod-empty spec: containers: - name: container-empty image: nginx imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /cache name: cache-volume ##与volumes中的name保持一致 volumes: - emptyDir: {} name: cache-volume更新资源清单文件[[rootk8s-master](mailto:rootk8s-master) ~]# kubectl apply -f emptydir.yaml ​ pod/pod-empty created查看本机临时目录存在的位置可用如下方法查看pod调度到哪个节点[rootk8s-master ~]# kubectl get pod pod-empty -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-empty 1/1 Running 0 27s 172.16.69.230 k8s-node2 none none查看pod的uid[rootk8s-master ~]# kubectl get pod pod-empty -o yaml | grep uid uid: 45a10614-495b-4745-be0f-c7492b90e2b7登录到k8s-node2上[rootk8s-node2 ~]# tree /var/lib/kubelet/pods/45a10614-495b-4745-be0f-c7492b90e2b7/ /var/lib/kubelet/pods/45a10614-495b-4745-be0f-c7492b90e2b7/ ├── containers │ └── container-empty │ └── 43f2b9b9 ├── etc-hosts ├── plugins │ └── kubernetes.io~empty-dir │ ├── cache-volume │ │ └── ready │ └── wrapped_kube-api-access-njjrv │ └── ready └── volumes ├── kubernetes.io~empty-dir │ └── cache-volume └── kubernetes.io~projected └── kube-api-access-njjrv ├── ca.crt - ..data/ca.crt ├── namespace - ..data/namespace └── token - ..data/token ​ 11 directories, 7 files测试###模拟产生数据测试 rootpod-empty:/cache# touch file{1..10} rootpod-empty:/cache# ls 10}file{1. aaa file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 rootpod-empty:/cache# exit exit ​ ###node查看 [rootk8s-node2 ~]# tree /var/lib/kubelet/pods/45a10614-495b-4745-be0f-c7492b90e2b7/ /var/lib/kubelet/pods/45a10614-495b-4745-be0f-c7492b90e2b7/ ├── containers │ └── container-empty │ └── 43f2b9b9 ├── etc-hosts ├── plugins │ └── kubernetes.io~empty-dir │ ├── cache-volume │ │ └── ready │ └── wrapped_kube-api-access-njjrv │ └── ready └── volumes ├── kubernetes.io~empty-dir │ └── cache-volume │ ├── 10}file{1. │ ├── aaa │ ├── file1 │ ├── file10 │ ├── file2 │ ├── file3 │ ├── file4 │ ├── file5 │ ├── file6 │ ├── file7 │ ├── file8 │ └── file9 └── kubernetes.io~projected └── kube-api-access-njjrv ├── ca.crt - ..data/ca.crt ├── namespace - ..data/namespace └── token - ..data/token ​ 12 directories, 18 files ###模拟删除pod测试 ​ [rootk8s-master ~]# kubectl delete pod pod-empty pod pod-empty deleted [rootk8s-node2 ~]# tree /var/lib/kubelet/pods/45a10614-495b-4745-be0f-c7492b90e2b7/ /var/lib/kubelet/pods/45a10614-495b-4745-be0f-c7492b90e2b7/ [error opening dir] ​ 0 directories, 0 files ​2、k8s持久化存储hostPathhostPath Volume是指Pod挂载宿主机上的目录或文件。 hostPath Volume使得容器可以使用宿主机的文件系统进行存储hostpath宿主机路径节点级别的存储卷在pod被删除这个存储卷还是存在的不会被删除所以只要同一个pod被调度到同一个节点上来在pod被删除重新被调度到这个节点之后对应的数据依然是存在的。查看hostPath存储卷的用法[[rootk8s-master](mailto:rootk8s-master) ~]# kubectl explain pods.spec.volumes.hostPath KIND: Pod VERSION: v1 RESOURCE: hostPath Object DESCRIPTION: HostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: [https://kubernetes.io/docs/concepts/storage/volumes#hostpath](https://kubernetes.io/docs/concepts/storage/volumes#hostpath) Represents a host path mapped into a pod. Host path volumes do not support ownership management or SELinux relabeling. FIELDS: path string -required- type string创建一个pod挂载hostPath存储卷[rootk8s-master~]# cat hostpath.yaml ​ apiVersion: v1 kind: Pod metadata: name: test-hostpath spec: containers: - image: nginx imagePullPolicy: IfNotPresent name: test-nginx volumeMounts: - mountPath: /usr/share/nginx/html name: test-volume volumes: - name: test-volume hostPath: path: /data type: DirectoryOrCreate注意DirectoryOrCreate表示本地有/data目录就用本地的本地没有就会在pod调度到的节点自动创建一个更新资源清单文件并查看pod调度到了哪个物理节点[rootk8s-master ~]# kubectl apply -f test-hostpath.yaml pod/test-hostpath created ​ [rootk8s-master ~]# kubectl get pod test-hostpath -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-hostpath 1/1 Running 0 80s 172.16.69.248 k8s-node2 none none测试###没有首页 [rootk8s-master ~]# curl 172.16.69.248 html headtitle403 Forbidden/title/head body centerh1403 Forbidden/h1/center hrcenternginx/1.25.3/center /body /html ###生成首页 [rootk8s-node2 ~]# cd /data/ [rootk8s-node2 data]# ls [rootk8s-node2 data]# echo 1111 index.html ​ [rootk8s-master ~]# curl 172.16.69.248 1111 ​hostpath存储卷缺点单节点pod删除之后重新创建必须调度到同一个node节点数据才不会丢失如何调度到同一个nodeName呢 需要我们再yaml文件中进行指定就可以啦apiVersion: v1 kind: Pod metadata: name: test-hostpath spec: nodeName: k8s-node2 containers: - image: nginx name: test-nginx volumeMounts: - mountPath: /usr/share/nginx/html name: test-volume volumes: - name: test-volume hostPath: path: /data type: DirectoryOrCreate测试[rootk8s-master ~]# kubectl apply -f test-hostpath.yaml pod/test-hostpath configured [rootk8s-master ~]# kubectl delete pod test-hostpath pod test-hostpath deleted [rootk8s-master ~]# kubectl apply -f test-hostpath.yaml pod/test-hostpath created [rootk8s-master ~]# kubectl get pod test-hostpath -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-hostpath 1/1 Running 0 9s 172.16.69.219 k8s-node2 none none [rootk8s-master ~]# curl 172.16.69.219 11113、k8s持久化存储nfs上节说的hostPath存储存在单点故障pod挂载hostPath时只有调度到同一个节点数据才不会丢失。那可以使用nfs作为持久化存储。搭建nfs服务以k8s的控制节点作为NFS服务端[rootk8s-master ~]# yum install -y nfs-utils在宿主机创建NFS需要的共享目录[rootk8s-master ~]# mkdir /data -pv mkdir: 已创建目录 /data mkdir: 已创建目录 /data配置nfs共享服务器上的/data目录[rootk8s-master] ~]# systemctl enable --now nfs [rootk8s-master] ~]# vim /etc/exports /data 192.168.166.0/24(rw,sync,no_root_squash,no_subtree_check)#使NFS配置生效[rootk8s-master ~]# exportfs -avr exporting 192.168.166.0/24:/data所有的worker节点安装nfs-utilsyum install nfs-utils -y systemctl enable --now nfs#在k8s-node1和k8s-node2上手动挂载试试 [rootk8s-node1 ~]# mount 192.168.166.7:/data /mnt [rootk8s-node1 ~]# df -Th | grep nfs 192.168.166.7:/data nfs4 116G 6.7G 109G 6% /mnt #nfs可以被正常挂载 #手动卸载 [rootk8s-node1 ~]# umount /mnt创建Pod挂载NFS共享出来的目录Pod挂载nfs的官方地址https://kubernetes.io/zh/docs/concepts/storage/volumes/[rootk8s-master ~]# cat nfs.yaml ​ apiVersion: v1 kind: Pod metadata: name: test-nfs spec: containers: - name: test-nfs image: nginx imagePullPolicy: IfNotPresent ports: - containerPort: 80 protocol: TCP volumeMounts: - name: nfs-volumes mountPath: /usr/share/nginx/html volumes: - name: nfs-volumes nfs: path: /data #共享目录 server: 192.168.166.7 ##nfs服务器地址更新资源清单文件[rootk8s-master ~]# kubectl apply -f test-nfs.yaml pod/test-nfs created查看pod是否创建成功[rootk8s-master ~]# kubectl get pods -o wide | grep nfs test-nfs 1/1 Running 0 55s 172.16.79.68 k8s-node1 none none测试[rootk8s-master ~]# curl 172.16.79.68 html headtitle403 Forbidden/title/head body centerh1403 Forbidden/h1/center hrcenternginx/1.25.3/center /body /html #登录到nfs服务器在共享目录创建一个index.html [rootk8s-master ~]# cd /data/ [rootk8s-master volumes]# echo nfs index.html [rootk8s-master ~]# curl 172.16.79.68 nfs上面说明挂载nfs存储卷成功了nfs支持多个客户端挂载可以创建多个pod挂载同一个nfs服务器共享出来的目录但是nfs如果宕机了数据也就丢失了所以需要使用分布式存储常见的分布式存储有glusterfs和cephfs4、k8s持久化存储 PVC参考官网:https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes4.1.1 k8s PV是什么PersistentVolumePV是群集中的一块存储由管理员配置或使用存储类动态配置。 它是集群中的资源就像pod是k8s集群资源一样。 PV是容量插件如Volumes其生命周期独立于使用PV的任何单个pod。4.1.2 k8s PVC是什么PersistentVolumeClaimPVC是用户使用存储的请求。 它类似于pod。Pod消耗节点资源PVC消耗存储资源。 pod可以请求特定级别的资源CPU和内存。 pvc在申请pv的时候也可以请求特定的大小和访问模式例如可以一次读写或多次只读。4.1.3 k8s PVC和PV工作原理PV是群集中的资源。 PVC是对这些资源的请求。PV和PVC之间的相互作用遵循以下生命周期1pv的供应方式可以通过两种方式配置PV静态或动态。静态的集群管理员创建了许多PV。它们包含可供群集用户使用的实际存储的详细信息。它们存在于Kubernetes API中可供使用。动态的当管理员创建的静态PV都不匹配用户的PersistentVolumeClaim时群集可能会尝试为PVC专门动态配置卷。此配置基于StorageClassesPVC必须请求存储类管理员必须创建并配置该类以便进行动态配置。2绑定用户创建pvc并指定需要的资源和访问模式。在找到可用pv之前pvc会保持未绑定状态3使用需要找一个存储服务器把它划分成多个存储空间k8s管理员可以把这些存储空间定义成多个pv在pod中使用pvc类型的存储卷之前需要先创建pvc通过定义需要使用的pv的大小和对应的访问模式找到合适的pvpvc被创建之后就可以当成存储卷来使用了我们在定义pod时就可以使用这个pvc的存储卷pvc和pv它们是一一对应的关系pv如果被pvc绑定了就不能被其他pvc使用了我们在创建pvc的时候应该确保和底下的pv能绑定如果没有合适的pv那么pvc就会处于pending状态。4回收策略当我们创建pod时如果使用pvc做为存储卷那么它会和pv绑定当删除podpvc和pv绑定就会解除解除之后和pvc绑定的pv卷里的数据需要怎么处理目前卷可以保留、回收或删除1、Retain 当删除pvc的时候pv仍然存在处于released状态但是它不能被其他pvc绑定使用里面的数据还是存在的当我们下次再使用的时候数据还是存在的这个是默认的回收策略。 2、Delete 删除pvc时即会从Kubernetes中移除PV也会从相关的外部设施中删除存储资产。4.1.4 创建pod使用pvc作为持久化存储卷1、创建nfs共享目录#在宿主机创建NFS需要的共享目录 ​ [rootk8s-master] ~]# mkdir /data/v{1..10} -p ​ #配置nfs共享宿主机上的/data/v1..v10目录 [rootk8s-master ~]# cat /etc/exports /data 192.168.166.0/24(rw,no_root_squash) /data/v1 192.168.166.0/24(rw,no_root_squash) /data/v2 192.168.166.0/24(rw,no_root_squash) /data/v3 192.168.166.0/24(rw,no_root_squash) /data/v4 192.168.166.0/24(rw,no_root_squash) /data/v5 192.168.166.0/24(rw,no_root_squash) /data/v6 192.168.166.0/24(rw,no_root_squash) /data/v7 192.168.166.0/24(rw,no_root_squash) /data/v8 192.168.166.0/24(rw,no_root_squash) /data/v9 192.168.166.0/24(rw,no_root_squash) /data/v10 192.168.166.0/24(rw,no_root_squash) #重新加载配置使配置成效 [rootk8s-master ~]# exportfs -arv ​2、如何编写pv的资源清单文件#查看定义pv需要的字段 [rootk8s-master ~]# kubectl explain pv KIND: PersistentVolume VERSION: v1 DESCRIPTION: PersistentVolume (PV) is a storage resource provisioned by an administrator. It is analogous to a node. More info: [https://kubernetes.io/docs/concepts/storage/persistent-volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes) FIELDS: apiVersion strings kindstring metadataObject specObject #查看定义nfs类型的pv需要的字段 [rootk8s-master] ~]# kubectl explain pv.spec.nfs KIND: PersistentVolume VERSION: v1 RESOURCE: nfs Object FIELDS: pathstring -required- readOnlyboolean server string -required-3、创建pv[rootk8s-master ~]# cat pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: v1 spec: capacity: storage: 1Gi #pv的存储空间容量 accessModes: [ReadWriteOnce] nfs: path: /data/v1 #把nfs的存储空间创建成pv server: 192.168.166.7 #nfs服务器的地址 --- apiVersion: v1 kind: PersistentVolume metadata: name: v2 spec: capacity: storage: 2Gi accessModes: [ReadWriteMany] nfs: path: /data/v2 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v3 spec: capacity: storage: 3Gi accessModes: [ReadOnlyMany] nfs: path: /data/v3 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v4 spec: capacity: storage: 4Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v4 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v5 spec: capacity: storage: 5Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v5 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v6 spec: capacity: storage: 6Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v6 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v7 spec: capacity: storage: 7Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v7 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v8 spec: capacity: storage: 8Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v8 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v9 spec: capacity: storage: 9Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v9 server: 192.168.166.7 --- apiVersion: v1 kind: PersistentVolume metadata: name: v10 spec: capacity: storage: 10Gi accessModes: [ReadWriteOnce,ReadWriteMany] nfs: path: /data/v10 server: 192.168.166.7更新资源清单文件[rootk8s-master ~]# kubectl apply -f pv.yaml persistentvolume/v1 created persistentvolume/v2 created persistentvolume/v3 created persistentvolume/v4 created persistentvolume/v5 created persistentvolume/v6 created persistentvolume/v7 created persistentvolume/v8 created persistentvolume/v9 created persistentvolume/v10 created查看pv资源[rootk8s-master ~]# kubectl get pv ​ NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE v1 1Gi RWO Retain Available 88s v10 10Gi RWO,RWX Retain Available 88s v2 2Gi RWX Retain Available 88s v3 3Gi ROX Retain Available 88s v4 4Gi RWO,RWX Retain Available 88s v5 5Gi RWO,RWX Retain Available 88s v6 6Gi RWO,RWX Retain Available 88s v7 7Gi RWO,RWX Retain Available 88s v8 8Gi RWO,RWX Retain Available 88s v9 9Gi RWO,RWX Retain Available 88s ​ #STATUS是Available表示pv是可用的 NAME名称 通常用于标识某个特定的对象在这里可能是指存储相关的实体比如存储卷等的具体名字方便对其进行区分和引用。 CAPACITY容量 代表该对象所具备的存储容量大小一般会以特定的存储单位如字节、KB、MB、GB、TB 等来衡量表示其能够容纳的数据量多少。 ACCESS MODES访问模式 指的是允许对相应存储资源进行访问的方式例如可以是只读、读写、可追加等不同模式决定了用户或系统对其存储内容操作的权限范围。 RECLAIM POLICY回收策略 关乎当存储资源不再被使用或者释放时对其所占用空间等资源的处理方式常见的策略比如立即回收、延迟回收、按特定条件回收等确保资源能合理地被再次利用。 STATUS状态 描述该存储相关对象当前所处的情况例如可能是可用、不可用、正在初始化、已损坏等不同状态便于了解其是否能正常工作。 CLAIM声明 在存储语境中往往涉及对存储资源的一种请求或者占用声明表明某个主体对相应存储资源有着相关权益或者正在使用它等情况。 STORAGECLASS存储类别 用于区分不同特性的存储分类比如可以根据存储的性能高速、低速、存储介质磁盘、磁带等、存储成本昂贵、廉价等因素划分出不同的存储类以满足不同场景的需求。 VOLUMEATTRIBUTESCLASS卷属性类别 主要是针对存储卷这一特定存储对象而言涵盖了该卷在诸如容量属性、性能属性、安全属性等多方面的类别划分体现其具备的各类特性。 REASON原因 如果存储对象处于某种特定状态比如异常状态等此处用于说明造成该状态的具体缘由方便排查问题、分析情况。 AGE时长 一般是指从该存储对象创建开始到当前所经历的时间长度可用于衡量其存在的时间阶段辅助判断其使用情况等。4、创建pvc和符合条件的pv绑定[rootk8s-master ~]# cat pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc spec: accessModes: [ReadWriteMany] resources: requests: storage: 2Gi更新资源清单文件[rootk8s-master ~]# kubectl apply -f my-pvc.yaml ​ persistentvolumeclaim/my-pvc created查看pv和pvcSTATUS是Bound表示这个pv已经被my-pvc绑定了[rootk8s-master ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE v1 1Gi RWO Retain Available 17m v10 10Gi RWO,RWX Retain Available 17m v2 2Gi RWX Retain Bound default/my-pvc 17m v3 3Gi ROX Retain Available 17m v4 4Gi RWO,RWX Retain Bound default/my-pvc1 17m v5 5Gi RWO,RWX Retain Available 17m v6 6Gi RWO,RWX Retain Available 17m v7 7Gi RWO,RWX Retain Available 17m v8 8Gi RWO,RWX Retain Available 17m v9 9Gi RWO,RWX Retain Available 17m [rootk8s-master ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE my-pvc Bound v2 2Gi RWX 12m my-pvc1 Bound v4 4Gi RWO,RWX 4m11s ​5、创建pod挂载pvc[roothd1 volume]# cat pod_pvc.yaml ​ apiVersion: v1 kind: Pod metadata: name: pod-pvc spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent volumeMounts: - name: nginx-html mountPath: /usr/share/nginx/html volumes: - name: nginx-html persistentVolumeClaim: claimName: my-pvc更新资源清单文件[rootk8s-master ~]# kubectl apply -f pod-pvc.yaml pod/pod-pvc created ​查看pod状态[rootk8s-master ~]# kubectl get pod -o wide | grep pvc pod-pvc 1/1 Running 0 16s 172.16.79.127 k8s-node1 none none #通过上面可以看到pod处于running状态正常运行测试[rootk8s-master ~]# curl 172.16.79.127 html headtitle403 Forbidden/title/head body centerh1403 Forbidden/h1/center hrcenternginx/1.25.3/center /body /html ​ ####在nfs服务器端写入index.html [rootk8s-master ~]# cd /data/v2 [rootk8s-master v2]# echo pvc index.html [rootk8s-master ~]# curl 172.16.79.127 pvc ###创建另一个pod测试 ​删除pod-pvc 这个pod发现pvc还是存在的[rootk8s-master ~]# kubectl delete pod pod-pvc [rootk8s-master ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE my-pvc Bound v2 2Gi RWX 36s my-pvc1 Bound v4 4Gi RWO,RWX 20m ​ [rootk8s-master ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE v1 1Gi RWO Retain Available 54s v10 10Gi RWO,RWX Retain Available 33m v2 2Gi RWX Retain Bound default/my-pvc 54s v3 3Gi ROX Retain Available 33m v4 4Gi RWO,RWX Retain Bound default/my-pvc1 33m v5 5Gi RWO,RWX Retain Available 33m v6 6Gi RWO,RWX Retain Available 33m v7 7Gi RWO,RWX Retain Available 33m v8 8Gi RWO,RWX Retain Available 33m v9 9Gi RWO,RWX Retain Available 33m删除pvc[rootk8s-master ~]# kubectl delete pvc my-pvc persistentvolumeclaim my-pvc deleted [rootk8s-master ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE my-pvc1 Bound v4 4Gi RWO,RWX 20m #发现pv的状态发生了变化 [rootk8s-master ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE v1 1Gi RWO Retain Available 2m9s v10 10Gi RWO,RWX Retain Available 34m v2 2Gi RWX Retain Released default/my-pvc 2m9s v3 3Gi ROX Retain Available 34m v4 4Gi RWO,RWX Retain Bound default/my-pvc1 34m v5 5Gi RWO,RWX Retain Available 34m v6 6Gi RWO,RWX Retain Available 34m v7 7Gi RWO,RWX Retain Available 34m v8 8Gi RWO,RWX Retain Available 34m v9 9Gi RWO,RWX Retain Available 34m ​此时我们进入到nfs的共享目录发现我们建立的index.html还是存在的[rootk8s-master v2]# ls index.html注使用pvc和pv的注意事项1、我们每次创建pvc的时候需要事先有划分好的pv这样可能不方便那么可以在创建pvc的时候直接动态创建一个pv这个存储类pv事先是不存在的2、pvc和pv绑定如果使用默认的回收策略retain那么删除pvc之后pv会处于released状态我们想要继续使用这个pv需要手动删除pvkubectl delete pv pv_name删除pv不会删除pv里的数据当我们重新创建pvc时还会和这个最匹配的pv绑定数据还是原来数据不会丢失。5、 k8s存储类storageclass5.1 概述上面介绍的PV和PVC模式都是需要先创建好PV然后定义好PVC和pv进行一对一的Bond但是如果PVC请求成千上万那么就需要创建成千上万的PV对于运维人员来说维护成本很高Kubernetes提供一种自动创建PV的机制叫StorageClass它的作用就是创建PV的模板。k8s集群管理员通过创建storageclass可以动态生成一个存储卷pv供k8s pvc使用。每个StorageClass都包含字段provisionerparameters和reclaimPolicy。具体来说StorageClass会定义以下两部分1、PV的属性 比如存储的大小、类型等2、创建这种PV需要使用到的存储插件比如Ceph、NFS等有了这两部分信息Kubernetes就能够根据用户提交的PVC找到对应的StorageClass然后Kubernetes就会调用 StorageClass声明的存储插件创建出需要的PV。StorageClass运行原理volumeClaimTemplates实现了pvc的自动化StorageClass实现了pv的自动化每个 StorageClass 都包含 provisioner、parameters 和 reclaimPolicy 字段 这些字段会在 StorageClass 需要动态分配 PersistentVolume 时会使用到。StorageClass 对象的命名很重要用户使用这个命名来请求生成一个特定的类。 当创建 StorageClass 对象时管理员设置 StorageClass 对象的命名和其他参数一旦创建了对象就不能再对其更新。管理员可以为没有申请绑定到特定 StorageClass 的 PVC 指定一个默认的存储类要使用 StorageClass我们就得安装对应的自动配置程序比如我们这里存储后端使用的是 nfs那么我们就需要使用到一个 nfs-client 的自动配置程序我们也叫它 Provisioner制备器这个程序使我们已经配置好的 nfs 服务器来自动创建持久卷也就是自动帮我们创建 PV。搭建StorageClassNFS,大致有以下几个步骤:1.创建一个可用的NFS Serve实际存储文件的空间 2.创建Service Account 这是用来管控NFS provisioner在k8s集群中运行的权限 3.创建StorageClass 负责建立PVC并调用NFS provisioner进行预定的工作,并让PV与PVC建立管理 4.创建NFS provisioner存储制备器 1每个 StorageClass 都有一个制备器Provisioner用来决定使用哪个卷插件制备 PV。 该字段必须指定。 2主要有两个功能 一个是在NFS共享目录下创建挂载点(volume) 另一个则是建了PV并将PV与NFS的挂载点建立关联storageclass字段#查看定义的storageclass需要的字段 [rootk8s-master~]# kubectl explain storageclass KIND: StorageClass VERSION: storage.k8s.io/v1 DESCRIPTION: StorageClass describes the parameters for a class of storage for which PersistentVolumes can be dynamically provisioned. StorageClasses are non-namespaced; the name of the storage class according to etcd is in ObjectMeta.Name. FIELDS: allowVolumeExpansion boolean allowedTopologies []Object apiVersion string kindstring metadataObject mountOptions []string parameters map[string]string provisionerstring -required- reclaimPolicy string volumeBindingMode string ​provisioner供应商也称作制备器storageclass需要有一个供应者用来确定我们使用什么样的存储来创建pv常见的provisionerprovisioner既可以由内部供应商提供也可以由外部供应商提供如果是外部供应商可以参考https://github.com/kubernetes-incubator/external-storage/下提供的方法创建。https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner以NFS为例要想使用NFS我们需要一个nfs-client的自动装载程序称之为provisioner这个程序会使我们已经配置好的NFS服务器自动创建持久卷也就是自动帮我们创建PV。allowVolumeExpansion允许卷扩展PersistentVolume 可以配置成可扩展。将此功能设置为true时允许用户通过编辑相应的 PVC 对象来调整卷大小。当基础存储类的allowVolumeExpansion字段设置为 true 时以下类型的卷支持卷扩展。注意此功能仅用于扩容卷不能用于缩小卷。5.2 安装nfs provisioner用于配合存储类动态生成pv1、创建运行nfs-provisioner需要的sa账号[rootk8s-master newnfs]# cat sa.yaml apiVersion: v1 kind: Namespace metadata: name: newnfs --- apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner namespace: newnfs --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [] resources: [persistentvolumes] verbs: [get, list, watch, create, delete] - apiGroups: [] resources: [persistentvolumeclaims] verbs: [get, list, watch, update] - apiGroups: [storage.k8s.io] resources: [storageclasses] verbs: [get, list, watch] - apiGroups: [] resources: [events] verbs: [create, update, patch] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: newnfs roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner namespace: newnfs rules: - apiGroups: [] resources: [endpoints] verbs: [get, list, watch, create, update, patch] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner namespace: newnfs subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: newnfs roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io ​ ###################################### [rootk8s-master newnfs]# kubectl apply -f sa.yaml namespace/newnfs created serviceaccount/nfs-client-provisioner created clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created ​2、安装nfs-provisioner程序[rootk8s-master ~]# mkdir /data -p ​ #把/data变成nfs共享的目录 ​ [rootk8s-master ~]# cat /etc/exports /data 192.168.166.0/24(rw,no_root_squash) /data/v1 192.168.166.0/24(rw,no_root_squash) /data/v2 192.168.166.0/24(rw,no_root_squash) /data/v3 192.168.166.0/24(rw,no_root_squash) /data/v4 192.168.166.0/24(rw,no_root_squash) /data/v5 192.168.166.0/24(rw,no_root_squash) /data/v6 192.168.166.0/24(rw,no_root_squash) /data/v7 192.168.166.0/24(rw,no_root_squash) /data/v8 192.168.166.0/24(rw,no_root_squash) /data/v9 192.168.166.0/24(rw,no_root_squash) /data/v10 192.168.166.0/24(rw,no_root_squash) ​ ############################### [rootk8s-master newnfs]# cat nfs.yaml kind: Deployment apiVersion: apps/v1 metadata: name: nfs-client-provisioner namespace: newnfs spec: replicas: 1 selector: matchLabels: app: nfs-client-provisioner strategy: type: Recreate #设置升级策略为删除再创建(默认为滚动更新) template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner #上一步创建的ServiceAccount名称 containers: - name: nfs-client-provisioner image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0 imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME # Provisioner的名称,以后设置的storageclass要和这个保持一致 value: storage-nfs - name: NFS_SERVER # NFS服务器地址,需和valumes参数中配置的保持一致 value: 192.168.166.7 - name: NFS_PATH # NFS服务器数据存储目录,需和volumes参数中配置的保持一致 value: /data - name: ENABLE_LEADER_ELECTION value: true volumes: - name: nfs-client-root nfs: server: 192.168.166.7 # NFS服务器地址 path: /data # NFS共享目录更新资源清单文件[rootk8s-master ~]# kubectl apply -f nfs.yaml ​ deployment.apps/nfs-client-provisioner created查看nfs-provisioner是否正常运行[rootk8s-master newnfs]# kubectl -n newnfs get pods | grep nfs nfs-client-provisioner-5486f75d5-qjjnh 1/1 Running 0 8m1s5.3 创建storageclass动态供给pv[rootk8s-master newnfs]# cat sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: namespace: newnfs name: nfs-storage annotations: storageclass.kubernetes.io/is-default-class: false ## 是否设置为默认的storageclass provisioner: storage-nfs ## 动态卷分配者名称必须和上面创建的deploy中环境变量“PROVISIONER_NAME”变量值一致 parameters: archiveOnDelete: true ## 设置为false时删除PVC不会保留数据,true则保留数据 mountOptions: - hard ## 指定为硬挂载方式 - nfsvers4 ## 指定NFS版本,这个需要根据NFS Server版本号设置nfs ​ [rootk8s-master ~]# kubectl apply -f sc.yaml storageclass.storage.k8s.io/nfs-storage created ​查看storageclass是否创建成功[rootk8s-master nfs]# kubectl -n newnfs get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-storage storage-nfs Delete Immediate false 8m30s显示内容如上说明storageclass创建成功了注意provisioner处写的example.com/nfs应该跟安装nfs provisioner时候的env下的PROVISIONER_NAME的value值保持一致如下env: name: PROVISIONER_NAME value: storage-nfs5.4 创建pvc通过storageclass动态生成pv[rootk8s-master newnfs]# cat pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: storage-pvc namespace: newnfs spec: storageClassName: nfs-storage ## 需要与上面创建的storageclass的名称一致 accessModes: - ReadWriteOnce resources: requests: storage: 1Mi ​ [rootk8s-master newnfs]# kubectl apply -f pvc.yaml persistentvolumeclaim/storage-pvc created查看是否动态生成了pvpvc是否创建成功并和pv绑定[rootk8s-master newnfs]# kubectl -n newnfs get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE storage-pvc Bound pvc-a56945be-6150-4591-b3e0-5e4a698b8e3a 1Mi RWO nfs-storage 8m8s [rootk8s-master newnfs]# kubectl -n newnfs get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-a56945be-6150-4591-b3e0-5e4a698b8e3a 1Mi RWO Delete Bound newnfs/storage-pvc nfs-storage 26m ​通过上面可以看到test-claim1的pvc已经成功创建了绑定的pv是pvc-da737fb7-3ffb-43c4-a86a-2bdfa7f201e2这个pv是由storageclass调用nfs provisioner自动生成的。步骤总结1、供应商创建一个nfs provisioner2、创建storageclassstorageclass指定刚才创建的供应商3、创建pvc这个pvc指定storageclass报错修复[rootk8s-master manifests]# kubectl -n newnfs get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE storage-pvc pending [rootk8s-master manifests]# kubectl -n newnfs logs nfs-client-provisioner-5486f75d5-qjjnh E0131 09:17:58.845719 1 controller.go:766] Unexpected error getting claim reference to claim default/test-claim1: selfLink was empty, cant make reference[rootk8s-master manifests]# cd /etc/kubernetes/manifests [rootk8s-master manifests]# cat kube-apiserver.yaml apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.166.161:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address192.168.166.161 - --allow-privilegedtrue - --authorization-modeNode,RBAC - --client-ca-file/etc/kubernetes/pki/ca.crt - --enable-admission-pluginsNodeRestriction - --enable-bootstrap-token-authtrue - --etcd-cafile/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servershttps://127.0.0.1:2379 - --kubelet-client-certificate/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key/etc/kubernetes/pki/apiserver-kubelet-client.key - --kubelet-preferred-address-typesInternalIP,ExternalIP,Hostname - --proxy-client-cert-file/etc/kubernetes/pki/front-proxy-client.crt - --proxy-client-key-file/etc/kubernetes/pki/front-proxy-client.key - --requestheader-allowed-namesfront-proxy-client - --requestheader-client-ca-file/etc/kubernetes/pki/front-proxy-ca.crt - --requestheader-extra-headers-prefixX-Remote-Extra- - --requestheader-group-headersX-Remote-Group - --requestheader-username-headersX-Remote-User - --secure-port6443 - --service-account-issuerhttps://kubernetes.default.svc.cluster.local - --service-account-key-file/etc/kubernetes/pki/sa.pub - --service-account-signing-key-file/etc/kubernetes/pki/sa.key - --service-cluster-ip-range10.10.0.0/16 - --tls-cert-file/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file/etc/kubernetes/pki/apiserver.key - --feature-gatesRemoveSelfLinkfalse ###增加该行1.20版本以后 image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 192.168.166.161 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 name: kube-apiserver readinessProbe: failureThreshold: 3 httpGet: host: 192.168.166.161 path: /readyz port: 6443 scheme: HTTPS periodSeconds: 1 timeoutSeconds: 15 resources: requests: cpu: 250m startupProbe: failureThreshold: 24 httpGet: host: 192.168.166.161 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true hostNetwork: true priorityClassName: system-node-critical securityContext: seccompProfile: type: RuntimeDefault volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs status: {} ​5.5 创建pod挂载storageclass动态生成的pvcstorage-pvc[rootk8s-master nfs]# cat read-pod.yaml ​ kind: Pod apiVersion: v1 metadata: name: read-pod namespace: newnfs spec: containers: - name: read-pod image: nginx imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-pvc mountPath: /usr/share/nginx/html restartPolicy: Never volumes: - name: nfs-pvc persistentVolumeClaim: claimName: storage-pvc #更新资源清单文件 ​ [rootk8s-master newnfs]# kubectl apply -f test-pod.yaml pod/read-pod created ​ ​ #查看pod是否创建成功 ​ [rootk8s-master nfs]# kubectl -n newnfs get pods | grep read ​ read-pod 1/1 Running 0 ##在NFS服务端写入首页 [rootk8s-master ~]# cd /data/newnfs-storage-pvc-pvc-a56945be-6150-4591-b3e0-5e4a698b8e3a/ [rootk8s-master newnfs-storage-pvc-pvc-a56945be-6150-4591-b3e0-5e4a698b8e3a]# echo nfs index.html ##测试访问 [rootk8s-master newnfs]# curl 172.16.79.96 nfs查看index.html的物理位置删除read-pod ,删除pvc 会自动删除pv 所以默认的回收策略是delete是可以验证出来的[rootk8s-master newnfs]# kubectl -n newnfs delete pod read-pod pod read-pod deleted ###pv没有变化 [rootk8s-master newnfs]# kubectl -n newnfs delete pvc storage-pvc persistentvolumeclaim storage-pvc deleted ###pv发生变化没有了 ##在NFS服务端发生如下变化会将原来的nfspv进行更改名称归档 [rootk8s-master volumes]# ls archived-pvc-a56945be-6150-4591-b3e0-5e4a698b8e3a ​重新更新pvc文件会发现物理nfs中多了一个目录。[rootk8s-master newnfs]# kubectl apply -f pvc.yaml persistentvolumeclaim/storage-pvc created #### [rootk8s-master volumes]# ls archived-pvc-a56945be-6150-4591-b3e0-5e4a698b8e3a newnfs-storage-pvc-pvc-397dfa84-4b11-4f96-8378-801f034806d6