Analysis the Static Pod Removal Process in kubelet

The previous article discussed the interesting removal process of the mirror pod. This article will explore the removal process of static pods.

Static pods can originate from files and HTTP services, and static pods are only visible internally to the kubelet. The mirror pod is an image of the static pod that allows external components to capture the static state.

The previous article explained that removing the mirror pod does not delete the static pod. To delete a static pod, you need to either delete the files under the --pod-manifest-path directory or remove the pod by making the HTTP server specified in --manifest-url return a response body that excludes this pod.

Pod Removal Process Series Articles:

Now, let’s analyze the removal process of a static pod from a file. The Kubernetes version in this article is 1.23 and log level 4.

By analyzing kubelet logs and combining the relevant code, we can understand the removal process of static pods.

The complete log file is available at mirror pod kubelet log and watch pod output.

When the removal of the static pod configuration file is detected, the log indicates that the pod is disappearing, triggering a “SyncPodKill” event notification to the podWorker:

I1123 14:18:35.558172  315900 kubelet.go:2124] "SyncLoop REMOVE" source="file" pods=[default/nginx-static-pod-]
I1123 14:18:35.558191  315900 kubelet.go:1969] "Pod has been deleted and must be killed" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.558206  315900 pod_workers.go:638] "Pod is being removed by the kubelet, begin teardown" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6

This triggers the podWorker to execute syncTerminatingPod:

I1123 14:18:35.558234  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=1
I1123 14:18:35.558244  315900 pod_workers.go:1005] "Pod worker has observed request to terminate" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.558259  315900 kubelet.go:1795] "syncTerminatingPod enter" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6

The syncTerminatingPod function then stops the container and sandbox:

I1123 14:18:35.558456  315900 kubelet.go:1825] "Pod terminating with grace period" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 gracePeriod=30
I1123 14:18:35.558519  315900 kuberuntime_container.go:719] "Killing container with a grace period override" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 containerName="nginx-container" containerID="docker://b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc" gracePeriod=30
I1123 14:18:35.558528  315900 kuberuntime_container.go:723] "Killing container with a grace period" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 containerName="nginx-container"

Once syncTerminatingPod is completed, the podWorker finishes its execution:

I1123 14:18:35.775202  315900 kubelet.go:1873] "Pod termination stopped all running containers" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.775212  315900 kubelet.go:1875] "syncTerminatingPod exit" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.775220  315900 pod_workers.go:1050] "Pod terminated all containers successfully" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.775232  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=1
I1123 14:18:35.775237  315900 pod_workers.go:888] "Processing pod event"

Then, the podWorker starts executing syncTerminatedPod:

I1123 14:18:35.775237  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=2
I1123 14:18:35.938222  315900 kubelet.go:1883] "syncTerminatedPod enter" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6

PLEG detects the stopping of the container and sandbox. Since the event type is “SyncPodKill,” the podWorker does not set the “delete” field to true. In the kl.containerDeletor.deleteContainersInPod function, the removeAll parameter is set to false (the cleanup strategy is to keep the last exited container), so the PLEG event triggers the execution of cleanUpContainersInPod return an error : “Container not found in pod’s containers”

Because there is only one exited container, and no cleanup action is triggered.

I1123 14:18:35.938230  315900 kubelet.go:2156] "SyncLoop (PLEG): pod does not exist, ignore irrelevant event" event=&{ID:a8712c005851ee6b29cff91b9ab4b9c6 Type:ContainerDied Data:b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc}
I1123 14:18:35.938232  315900 kubelet_pods.go:1441] "Generating pod status" pod="default/nginx-static-pod-"
I1123 14:18:35.938244  315900 kubelet.go:2156] "SyncLoop (PLEG): pod does not exist, ignore irrelevant event" event=&{ID:a8712c005851ee6b29cff91b9ab4b9c6 Type:ContainerDied Data:398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78}
I1123 14:18:35.938251  315900 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78"

Once syncTerminatedPod is completed, the podWorker finishes its execution:

I1123 14:18:35.941374  315900 kubelet.go:1924] "syncTerminatedPod exit" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.941383  315900 pod_workers.go:1105] "Pod is complete and the worker can now stop" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:35.941395  315900 pod_workers.go:959] "Processing pod event done" pod="default/nginx-static-pod-" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=2

The log then indicates the update of the mirror pod’s status:

I1123 14:18:35.949126  315900 kubelet.go:2127] "SyncLoop RECONCILE" source="api" pods=[default/nginx-static-pod-]

Housekeeping is triggered, leading to the deletion of the mirror pod (with a GracePeriodSeconds set to 0):

I1123 14:18:37.445960  315900 kubelet.go:2202] "SyncLoop (housekeeping)"
I1123 14:18:37.448122  315900 kubelet_pods.go:1082] "Clean up pod workers for terminated pods"
I1123 14:18:37.448136  315900 pod_workers.go:1258] "Pod has been terminated and is no longer known to the kubelet, remove all history" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:37.448143  315900 kubelet_pods.go:1111] "Clean up probes for terminated pods"
I1123 14:18:37.451130  315900 kubelet_pods.go:1148] "Clean up orphaned pod statuses"
I1123 14:18:37.453700  315900 kubelet_pods.go:1167] "Clean up orphaned pod directories"
I1123 14:18:37.453841  315900 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=a8712c005851ee6b29cff91b9ab4b9c6 path="/data/kubernetes/kubelet/pods/a8712c005851ee6b29cff91b9ab4b9c6/volumes"
I1123 14:18:37.453954  315900 kubelet_volumes.go:236] "Orphaned pod found, removing" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:18:37.453970  315900 kubelet_pods.go:1178] "Clean up orphaned mirror pods"
I1123 14:18:37.453983  315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-" podUID=
I1123 14:18:37.463808  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:18:37.466092  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:18:37.466537  315900 kubelet_pods.go:1040] "Deleted pod" podName="nginx-static-pod-"
I1123 14:18:37.466546  315900 kubelet_pods.go:1185] "Clean up orphaned pod cgroups"
I1123 14:18:37.466560  315900 kubelet.go:2210] "SyncLoop (housekeeping) end"

The deletion of the mirror pod and its removal from the API server is detected, but since the pod has already been removed from the podManager, it does not trigger the podWorker to execute:

I1123 14:18:37.466578  315900 kubelet.go:2130] "SyncLoop DELETE" source="api" pods=[default/nginx-static-pod-]
I1123 14:18:37.466589  315900 kubelet.go:2124] "SyncLoop REMOVE" source="api" pods=[default/nginx-static-pod-]

Finally, the garbage collector removes the container and sandbox, and removes the pod log directory:

I1123 14:19:21.542254  315900 kuberuntime_container.go:947] "Removing container" containerID="b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc"
I1123 14:19:21.542265  315900 scope.go:110] "RemoveContainer" containerID="b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc"
I1123 14:19:21.554998  315900 kuberuntime_gc.go:171] "Removing sandbox" sandboxID="398445f28f116ed45394c18d7697a64dceeef739379d5ac920bbf3fd6cc1bb78"
I1123 14:19:21.563426  315900 kuberuntime_gc.go:343] "Removing pod logs" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:19:21.566388  315900 kubelet.go:1333] "Container garbage collection succeeded"
  1. The podConfig detects the removal of the static file, triggering a SyncLoop REMOVE.
  2. podWorker executes syncTerminatingPod (stops the container and sandbox).
  3. PLEG (Pod Lifecycle Event Generator) detects the exit of the sandbox and container.
  4. podWorker executes syncTerminatedPod (removes cgroup, updates the status of the mirror pod, and waits for the volume unmount of the pod to complete).
  5. Detection of the status update for the mirror pod.
  6. Housekeeping is triggered, performing cleanup tasks (removing podWorker, deleting the mirror pod, and removing the pod volume directory).
  7. Detection of the deletion of the mirror pod and its removal from the API server.
  8. The garbage collector executes the cleanup of the sandbox and containers, and removes the pod log directory.


The removal of a regular pod requires two DELETE operations to be performed to remove it from the API server. In contrast, In contrast, the removal of a static pod is achieved by removing the corresponding files or by removing the response body from the HTTP server. The corresponding mirror pod for the static pod is deleted during housekeeping, and the cleanup of exited containers and sandboxes is handled by the garbage collector.

Related Content