Recovering Cloud Disk-Mounted Pod When Node Crashes

Warning: After the node fails, you cannot unmount the cloud disk on the node, so you cannot safely unmount the cloud disk. The following documents involve forcibly unbinding the cloud disk, which may cause data loss. If you can't accept this risk, don't perform the operations in this document (you should try to recover the node at this time).

If disk data loss is tolerable (such as storage log data), and you want the Pod to recover as soon as possible, you can refer to the operations in this document.

After the node fails, if the Pod has mounted a cloud disk, it cannot be automatically recovered. Because Kubernetes cannot determine whether the node is really down or temporarily lost due to network problems, so the cloud disk may still have data written at this time. If the Pod migration is hastily carried out, it may damage the cloud disk.

If it is a normal Pod, it will automatically be evicted after a period of toleration time, see our other document: Pod Tolerating Node Anomaly Time Adjustment.

If you confirm that the node has failed, and you want to recover the Pod that has mounted the cloud disk on the failed node, please refer to this document for operations.

Scenario Reproduction

Next, we create a cluster of 3 nodes:

$ kubectl get node
NAME           STATUS                     ROLES    AGE     VERSION
10.9.133.163   Ready,SchedulingDisabled   master   4m26s   v1.22.5
10.9.138.243   Ready                      <none>   4m16s   v1.22.5
10.9.187.33    Ready,SchedulingDisabled   master   4m41s   v1.22.5
10.9.188.56    Ready                      <none>   3m58s   v1.22.5
10.9.32.154    Ready,SchedulingDisabled   master   4m43s   v1.22.5
10.9.61.217    Ready                      <none>   3m49s   v1.22.5

Create a StatefulSet in it, start 6 pods, they all mount ssd cloud disk:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-web-ssd
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 6
  serviceName: "test-nginx-ssd"
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: uhub.surfercloud.com/ucloud/nginx:latest
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: test-nginx-ssd
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: test-nginx-ssd
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "ssd-csi-udisk"
      resources:
        requests:
          storage: 1Gi

On the console interface, manually power off a cloud host to simulate a crash scenario:

After a while, you will find that the node is in the NotReady state, the Pod is stuck at Terminating, and the StatefulSet controller will not start new replicas or perform cloud disk unloading operations for them:

$ kubectl get node 10.9.188.56
NAME          STATUS     ROLES    AGE   VERSION
10.9.188.56   NotReady   <none>   12m   v1.22.5

$ kubectl get po -o wide | grep '10.9.188.56'
test-web-ssd-3   1/1     Terminating   0          9m52s   10.9.48.46     10.9.188.56    <none>           <none>

Repair Pod

If the node cannot be recovered in a short time, we should consider recovering the interrupted Pod to ensure the normal operation of the business.

First, after confirming that the node has failed, manually unmount the cloud disk on the abnormal node on the console interface:

Warning: This step may cause partial data loss on the cloud disk. Please confirm that you can accept the risk before proceeding. If you cannot tolerate data loss, do not perform the operation.

Then, in Kubernetes, delete the abnormal node resource, so that the abnormal Pod can be released and rebuilt:

$ kubectl delete node 10.9.188.56
node "10.9.188.56" deleted

At this time, the scheduler does not know that the cloud disk has been unmounted. We still need to manually delete the corresponding volumeattachment resource:

$ kubectl get volumeattachment | grep '10.9.188.56'
csi-53bbdc72c17cc6ecc7c87b2b9b6526679175e53f0b88b52384f9773c0abbcded   udisk.csi.ucloud.cn   pvc-f6aa7b36-6d44-4521-ba8a-0086fd979e70   10.9.188.56    true       13m

$ kubectl delete volumeattachment csi-53bbdc72c17cc6ecc7c87b2b9b6526679175e53f0b88b52384f9773c0abbcded
volumeattachment.storage.k8s.io "csi-53bbdc72c17cc6ecc7c87b2b9b6526679175e53f0b88b52384f9773c0abbcded" deleted

Wait for the Pod to rebuild, and after the reconstruction, it will reuse the original cloud disk:

$ kubectl get po test-web-ssd-3
NAME             READY   STATUS    RESTARTS   AGE
test-web-ssd-3   1/1     Running   0          103s