xiaoqing blogAnalysis the Static Pod Removal Process in kubelet
The previous article discussed the interesting removal process of the mirror pod. This article will explore the removal process of static pods.
Static pods can originate from files and HTTP services, and static pods are only visible internally to the kubelet. The mirror pod is an image of the static pod that allows external components to capture the static state.
The previous article explained that removing the mirror pod does not delete the static pod. To delete a static pod, you need to either delete the files under the --pod-manifest-path directory or remove the pod by making the HTTP server specified in --manifest-url return a response body that excludes this pod.
Pod Removal Process Series Articles:
Kubelet Bug: Sandbox Not Cleaned Up - Unraveling Retention Issues
Exploring Mirror Pod Deletion in Kubernetes: Understanding its Impact on Static Pod
Now, let’s analyze the removal process of a static pod from a file. The Kubernetes version in this article is 1.23 and log level 4.
1 Analyzing kubelet logs
By analyzing kubelet logs and combining the relevant code, we can understand the removal process of static pods.
The complete log file is available at mirror pod kubelet log and watch pod output.
When the removal of the static pod configuration file is detected, the log indicates that the pod is disappearing, triggering a “SyncPodKill” event notification to the podWorker:
I1123 14:18:35.558172 315900 kubelet.go:2124] "SyncLoop REMOVE" source="file" pods=[default/nginx-static-pod-10.11.251.2]
I1123 14:18:35.558191 315900 kubelet.go:1969] "Pod has been deleted and must be killed" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.558206 315900 pod_workers.go:638] "Pod is being removed by the kubelet, begin teardown" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6This triggers the podWorker to execute syncTerminatingPod:
I1123 14:18:35.558234 315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=1
I1123 14:18:35.558244 315900 pod_workers.go:1005] "Pod worker has observed request to terminate" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.558259 315900 kubelet.go:1795] "syncTerminatingPod enter" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6The syncTerminatingPod function then stops the container and sandbox:
I1123 14:18:35.558456 315900 kubelet.go:1825] "Pod terminating with grace period" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 gracePeriod=30
I1123 14:18:35.558519 315900 kuberuntime_container.go:719] "Killing container with a grace period override" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 containerName="nginx-container" containerID="docker://b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc" gracePeriod=30
I1123 14:18:35.558528 315900 kuberuntime_container.go:723] "Killing container with a grace period" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 containerName="nginx-container"Once syncTerminatingPod is completed, the podWorker finishes its execution:
I1123 14:18:35.775202 315900 kubelet.go:1873] "Pod termination stopped all running containers" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.775212 315900 kubelet.go:1875] "syncTerminatingPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.775220 315900 pod_workers.go:1050] "Pod terminated all containers successfully" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.775232 315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=1
I1123 14:18:35.775237 315900 pod_workers.go:888] "Processing pod event"Then, the podWorker starts executing syncTerminatedPod:
I1123 14:18:35.775237 315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=2
I1123 14:18:35.938222 315900 kubelet.go:1883] "syncTerminatedPod enter" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6PLEG detects the stopping of the container and sandbox. Since the event type is “SyncPodKill,” the podWorker does not set the “delete” field to true. In the kl.containerDeletor.deleteContainersInPod function, the removeAll parameter is set to false (the cleanup strategy is to keep the last exited container), so the PLEG event triggers the execution of cleanUpContainersInPod return an error : “Container not found in pod’s containers”
Because there is only one exited container, and no cleanup action is triggered.
I1123 14:18:35.938230 315900 kubelet.go:2156] "SyncLoop (PLEG): pod does not exist, ignore irrelevant event" event=&{ID:a8712c005851ee6b29cff91b9ab4b9c6 Type:ContainerDied Data:b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc}
I1123 14:18:35.938232 315900 kubelet_pods.go:1441] "Generating pod status" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:18:35.938244 315900 kubelet.go:2156] "SyncLoop (PLEG): pod does not exist, ignore irrelevant event" event=&{ID:a8712c005851ee6b29cff91b9ab4b9c6 Type:ContainerDied Data:398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78}
I1123 14:18:35.938251 315900 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78"Once syncTerminatedPod is completed, the podWorker finishes its execution:
I1123 14:18:35.941374 315900 kubelet.go:1924] "syncTerminatedPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.941383 315900 pod_workers.go:1105] "Pod is complete and the worker can now stop" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.941395 315900 pod_workers.go:959] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=2The log then indicates the update of the mirror pod’s status:
I1123 14:18:35.949126 315900 kubelet.go:2127] "SyncLoop RECONCILE" source="api" pods=[default/nginx-static-pod-10.11.251.2]Housekeeping is triggered, leading to the deletion of the mirror pod (with a GracePeriodSeconds set to 0):
I1123 14:18:37.445960 315900 kubelet.go:2202] "SyncLoop (housekeeping)"
I1123 14:18:37.448122 315900 kubelet_pods.go:1082] "Clean up pod workers for terminated pods"
I1123 14:18:37.448136 315900 pod_workers.go:1258] "Pod has been terminated and is no longer known to the kubelet, remove all history" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:37.448143 315900 kubelet_pods.go:1111] "Clean up probes for terminated pods"
I1123 14:18:37.451130 315900 kubelet_pods.go:1148] "Clean up orphaned pod statuses"
I1123 14:18:37.453700 315900 kubelet_pods.go:1167] "Clean up orphaned pod directories"
I1123 14:18:37.453841 315900 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=a8712c005851ee6b29cff91b9ab4b9c6 path="/data/kubernetes/kubelet/pods/a8712c005851ee6b29cff91b9ab4b9c6/volumes"
I1123 14:18:37.453954 315900 kubelet_volumes.go:236] "Orphaned pod found, removing" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:37.453970 315900 kubelet_pods.go:1178] "Clean up orphaned mirror pods"
I1123 14:18:37.453983 315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-10.11.251.2" podUID=
I1123 14:18:37.463808 315900 config.go:278] "Setting pods for source" source="api"
I1123 14:18:37.466092 315900 config.go:278] "Setting pods for source" source="api"
I1123 14:18:37.466537 315900 kubelet_pods.go:1040] "Deleted pod" podName="nginx-static-pod-10.11.251.2_default"
I1123 14:18:37.466546 315900 kubelet_pods.go:1185] "Clean up orphaned pod cgroups"
I1123 14:18:37.466560 315900 kubelet.go:2210] "SyncLoop (housekeeping) end"The deletion of the mirror pod and its removal from the API server is detected, but since the pod has already been removed from the podManager, it does not trigger the podWorker to execute:
I1123 14:18:37.466578 315900 kubelet.go:2130] "SyncLoop DELETE" source="api" pods=[default/nginx-static-pod-10.11.251.2]
I1123 14:18:37.466589 315900 kubelet.go:2124] "SyncLoop REMOVE" source="api" pods=[default/nginx-static-pod-10.11.251.2]Finally, the garbage collector removes the container and sandbox, and removes the pod log directory:
I1123 14:19:21.542254 315900 kuberuntime_container.go:947] "Removing container" containerID="b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc"
I1123 14:19:21.542265 315900 scope.go:110] "RemoveContainer" containerID="b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc"
I1123 14:19:21.554998 315900 kuberuntime_gc.go:171] "Removing sandbox" sandboxID="398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78"
I1123 14:19:21.563426 315900 kuberuntime_gc.go:343] "Removing pod logs" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:19:21.566388 315900 kubelet.go:1333] "Container garbage collection succeeded"2 Removal Process for Static Pods
- The
podConfigdetects the removal of the static file, triggering aSyncLoop REMOVE. podWorkerexecutessyncTerminatingPod(stops the container and sandbox).- PLEG (Pod Lifecycle Event Generator) detects the exit of the sandbox and container.
podWorkerexecutessyncTerminatedPod(removes cgroup, updates the status of the mirror pod, and waits for the volume unmount of the pod to complete).- Detection of the status update for the mirror pod.
- Housekeeping is triggered, performing cleanup tasks (removing
podWorker, deleting the mirror pod, and removing the pod volume directory). - Detection of the deletion of the mirror pod and its removal from the API server.
- The garbage collector executes the cleanup of the sandbox and containers, and removes the pod log directory.
3 Summary
The removal of a regular pod requires two DELETE operations to be performed to remove it from the API server. In contrast, In contrast, the removal of a static pod is achieved by removing the corresponding files or by removing the response body from the HTTP server. The corresponding mirror pod for the static pod is deleted during housekeeping, and the cleanup of exited containers and sandboxes is handled by the garbage collector.