The previous article discussed the interesting removal process of the mirror pod. This article will explore the removal process of static pods.
Static pods can originate from files and HTTP services, and static pods are only visible internally to the kubelet. The mirror pod is an image of the static pod that allows external components to capture the static state.
The previous article explained that removing the mirror pod does not delete the static pod. To delete a static pod, you need to either delete the files under the --pod-manifest-path directory or remove the pod by making the HTTP server specified in --manifest-url return a response body that excludes this pod.
When the removal of the static pod configuration file is detected, the log indicates that the pod is disappearing, triggering a “SyncPodKill” event notification to the podWorker:
1
2
3
I1123 14:18:35.558172 315900 kubelet.go:2124] "SyncLoop REMOVE" source="file" pods=[default/nginx-static-pod-10.11.251.2]
I1123 14:18:35.558191 315900 kubelet.go:1969] "Pod has been deleted and must be killed" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.558206 315900 pod_workers.go:638] "Pod is being removed by the kubelet, begin teardown" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
This triggers the podWorker to execute syncTerminatingPod:
PLEG detects the stopping of the container and sandbox. Since the event type is “SyncPodKill,” the podWorker does not set the “delete” field to true. In the kl.containerDeletor.deleteContainersInPod function, the removeAll parameter is set to false (the cleanup strategy is to keep the last exited container), so the PLEG event triggers the execution of cleanUpContainersInPod return an error : “Container not found in pod’s containers”
Because there is only one exited container, and no cleanup action is triggered.
1
2
3
4
I1123 14:18:35.938230 315900 kubelet.go:2156] "SyncLoop (PLEG): pod does not exist, ignore irrelevant event" event=&{ID:a8712c005851ee6b29cff91b9ab4b9c6 Type:ContainerDied Data:b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc}
I1123 14:18:35.938232 315900 kubelet_pods.go:1441] "Generating pod status" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:18:35.938244 315900 kubelet.go:2156] "SyncLoop (PLEG): pod does not exist, ignore irrelevant event" event=&{ID:a8712c005851ee6b29cff91b9ab4b9c6 Type:ContainerDied Data:398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78}
I1123 14:18:35.938251 315900 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78"
Once syncTerminatedPod is completed, the podWorker finishes its execution:
1
2
3
I1123 14:18:35.941374 315900 kubelet.go:1924] "syncTerminatedPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.941383 315900 pod_workers.go:1105] "Pod is complete and the worker can now stop" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.941395 315900 pod_workers.go:959] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=2
The log then indicates the update of the mirror pod’s status:
Housekeeping is triggered, leading to the deletion of the mirror pod (with a GracePeriodSeconds set to 0):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
I1123 14:18:37.445960 315900 kubelet.go:2202] "SyncLoop (housekeeping)"
I1123 14:18:37.448122 315900 kubelet_pods.go:1082] "Clean up pod workers for terminated pods"
I1123 14:18:37.448136 315900 pod_workers.go:1258] "Pod has been terminated and is no longer known to the kubelet, remove all history" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:37.448143 315900 kubelet_pods.go:1111] "Clean up probes for terminated pods"
I1123 14:18:37.451130 315900 kubelet_pods.go:1148] "Clean up orphaned pod statuses"
I1123 14:18:37.453700 315900 kubelet_pods.go:1167] "Clean up orphaned pod directories"
I1123 14:18:37.453841 315900 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=a8712c005851ee6b29cff91b9ab4b9c6 path="/data/kubernetes/kubelet/pods/a8712c005851ee6b29cff91b9ab4b9c6/volumes"
I1123 14:18:37.453954 315900 kubelet_volumes.go:236] "Orphaned pod found, removing" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:37.453970 315900 kubelet_pods.go:1178] "Clean up orphaned mirror pods"
I1123 14:18:37.453983 315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-10.11.251.2" podUID=
I1123 14:18:37.463808 315900 config.go:278] "Setting pods for source" source="api"
I1123 14:18:37.466092 315900 config.go:278] "Setting pods for source" source="api"
I1123 14:18:37.466537 315900 kubelet_pods.go:1040] "Deleted pod" podName="nginx-static-pod-10.11.251.2_default"
I1123 14:18:37.466546 315900 kubelet_pods.go:1185] "Clean up orphaned pod cgroups"
I1123 14:18:37.466560 315900 kubelet.go:2210] "SyncLoop (housekeeping) end"
The deletion of the mirror pod and its removal from the API server is detected, but since the pod has already been removed from the podManager, it does not trigger the podWorker to execute:
The podConfig detects the removal of the static file, triggering a SyncLoop REMOVE.
podWorker executes syncTerminatingPod (stops the container and sandbox).
PLEG (Pod Lifecycle Event Generator) detects the exit of the sandbox and container.
podWorker executes syncTerminatedPod (removes cgroup, updates the status of the mirror pod, and waits for the volume unmount of the pod to complete).
Detection of the status update for the mirror pod.
Housekeeping is triggered, performing cleanup tasks (removing podWorker, deleting the mirror pod, and removing the pod volume directory).
Detection of the deletion of the mirror pod and its removal from the API server.
The garbage collector executes the cleanup of the sandbox and containers, and removes the pod log directory.
3 Summary
The removal of a regular pod requires two DELETE operations to be performed to remove it from the API server. In contrast, In contrast, the removal of a static pod is achieved by removing the corresponding files or by removing the response body from the HTTP server. The corresponding mirror pod for the static pod is deleted during housekeeping, and the cleanup of exited containers and sandboxes is handled by the garbage collector.
