This is also an article about the research on the process of removing pods, focusing on the removal of mirror pods. The term “mirror pod” may sound unfamiliar, it is a type of pod within Kubernetes.
Let’s first introduce the classification of pods. Pods come from file, http, and apiserver sources. Pods from the apiserver are called ordinary pods, while pods from other sources are called static pods (the control plane installed using kubeadm runs with static pods). To manage pods conveniently, kubelet generates corresponding pods for static pods on the apiserver. These types of pods are called mirror pods, essentially mirroring the static pod (almost identical, except for a different UID and the addition of “kubernetes.io/config.mirror” in annotations).
So, what happens when you delete a mirror pod? Will it remove static pods on the node?
This article is based on Kubernetes version 1.23 and log level 4.
1 Understanding Static Pods and Mirror Pods
The kubelet’s mechanism generates static pods with a name that is the pod name in the configuration plus the node’s name. In this example, the defined pod name is “nginx-static-pod,” and the generated pod name is “nginx-static-pod-10.11.251.2.”
# kubectl get pod nginx-static-pod-10.11.251.2 -o yaml -wapiVersion:v1kind:Podmetadata:annotations:kubernetes.io/config.hash:a8712c005851ee6b29cff91b9ab4b9c6kubernetes.io/config.mirror:a8712c005851ee6b29cff91b9ab4b9c6kubernetes.io/config.seen:"2023-11-23T14:05:21.400551188+08:00"kubernetes.io/config.source:filecreationTimestamp:"2023-11-23T06:05:21Z"name:nginx-static-pod-10.11.251.2namespace:defaultownerReferences:- apiVersion:v1controller:truekind:Nodename:10.11.251.2uid:d2e3ebf6-2a70-434f-9142-0f74a1dd16bcresourceVersion:"292107092"uid:1facfed2-ba67-40f8-9982-ab86a5e7fd33
The pod is deleted; the deletionTimestamp field has a value, and deletionGracePeriodSeconds is set to 30.
A new mirror pod is created; the UID is 7bdace8a-8cc5-4cb4-886b-f3961f07d3b8 (the pod’s UID has changed, and only the phase and qosClass fields are present in the status).
podWorker executes syncPod (as static pod is used to determine status in podWorker, syncTerminatingPod is not executed since the static pod is not deleted).
Deletion is completed, and a new mirror pod is created.
1
2
3
I1123 14:11:15.675108 315900 kubelet.go:1729] "Deleted mirror pod because it is outdated" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:11:15.675121 315900 kubelet.go:1740] "Creating a mirror pod for static pod" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:11:15.683985 315900 config.go:383] "Receiving a new pod"
Based on the observations above, deleting a mirror pod does not remove the static pod. Furthermore, when the kubelet recreates a mirror pod, it lacks status conditions and container status. Later, the pod status is synchronized. In reality, operations on mirror pods do not impact static pods because static pods are only visible within the kubelet, and external queries for static pod status can be made through the mirror pod on the apiserver or the /runningpods/ interface on the kubelet’s HTTP server.
From the source code analysis, every time the syncLoop triggers, podWorker executes syncPod.
In the syncPod function, the logic for handling static pods is as follows:
If the pod is a static pod and a mirror pod exists, check if the mirror pod is a valid image pod for this static pod. If it is not a valid mirror pod, delete the mirror pod.
If the mirror pod has already been deleted, perform the final deletion action (setting deletionGracePeriodSeconds to 0).
If the static pod has no mirror pod, create a mirror pod.
// Create Mirror Pod for Static Pod if it doesn't already exist
// If it's a static pod, check if the mirror pod exists, and create one when it doesn't
ifkubetypes.IsStaticPod(pod){deleted:=false// If the mirror pod exists
ifmirrorPod!=nil{// If the mirror pod is deleted or not the mirror pod of the pod
ifmirrorPod.DeletionTimestamp!=nil||!kl.podManager.IsMirrorPodOf(mirrorPod,pod){// The mirror pod is semantically different from the static pod. Remove
// it. The mirror pod will get recreated later.
klog.InfoS("Trying to delete pod","pod",klog.KObj(pod),"podUID",mirrorPod.ObjectMeta.UID)// pod.Name + "_" + pod.Namespace
podFullName:=kubecontainer.GetPodFullName(pod)varerrerror// Call API to delete mirror pod (parse name and namespace from podFullName)
deleted,err=kl.podManager.DeleteMirrorPod(podFullName,&mirrorPod.ObjectMeta.UID)ifdeleted{klog.InfoS("Deleted mirror pod because it is outdated","pod",klog.KObj(mirrorPod))}elseiferr!=nil{klog.ErrorS(err,"Failed deleting mirror pod","pod",klog.KObj(mirrorPod))}}}// If there is no mirror pod for static pod or the mirror pod is deleted,
// then create a new mirror pod as long as there is no error when getting the node
// and the node is not deleted
ifmirrorPod==nil||deleted{node,err:=kl.GetNode()// If there is an error getting the node or the node is deleted, no need to create a mirror pod
iferr!=nil||node.DeletionTimestamp!=nil{klog.V(4).InfoS("No need to create a mirror pod, since the node has been removed from the cluster","node",klog.KRef("",string(kl.nodeName)))}else{klog.V(4).InfoS("Creating a mirror pod for a static pod","pod",klog.KObj(pod))// If the node is not deleted, create a mirror pod for the static pod
iferr:=kl.podManager.CreateMirrorPod(pod);err!=nil{klog.ErrorS(err,"Failed creating a mirror pod for","pod",klog.KObj(pod))}}}}
