xiaoqing blogxiaoqing blog
Posts Videos Tags Categories About Resume
xiaoqing blogxiaoqing blog

Exploring Mirror Pod Deletion in Kubernetes: Understanding its Impact on Static Pod

xiaoqing  included in  category Kubernetes
     1714 words   8 minutes 

This is also an article about the research on the process of removing pods, focusing on the removal of mirror pods. The term “mirror pod” may sound unfamiliar, it is a type of pod within Kubernetes.

Let’s first introduce the classification of pods. Pods come from file, http, and apiserver sources. Pods from the apiserver are called ordinary pods, while pods from other sources are called static pods (the control plane installed using kubeadm runs with static pods). To manage pods conveniently, kubelet generates corresponding pods for static pods on the apiserver. These types of pods are called mirror pods, essentially mirroring the static pod (almost identical, except for a different UID and the addition of “kubernetes.io/config.mirror” in annotations).

So, what happens when you delete a mirror pod? Will it remove static pods on the node?

Pod Removal Process Series Articles:

This article is based on Kubernetes version 1.23 and log level 4.

The kubelet’s mechanism generates static pods with a name that is the pod name in the configuration plus the node’s name. In this example, the defined pod name is “nginx-static-pod,” and the generated pod name is “nginx-static-pod-10.11.251.2.”

Here is the static pod configuration on kubelet:

yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-static-pod
  namespace: default
spec:
  containers:
  - name: nginx-container
    image: nginx:latest
    ports:
    - containerPort: 80

Corresponding mirror pod:

yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292107092"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33
  spec:
    containers:
    - image: nginx:latest
      imagePullPolicy: Always
      name: nginx-container
      ports:
      - containerPort: 80
        protocol: TCP
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: 10.11.251.2
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      operator: Exists
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2023-11-23T06:05:21Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2023-11-23T06:05:29Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2023-11-23T06:05:29Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2023-11-23T06:05:21Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc
      image: nginx:latest
      imageID: docker-pullable://nginx@sha256:10d1f5b58f74683ad34eb29287e07dab1e90f10af243f151bb50aa5dbb4d62ee
      lastState: {}
      name: nginx-container
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2023-11-23T06:05:29Z"
    hostIP: 10.11.251.2
    phase: Running
    podIP: 10.26.124.96
    podIPs:
    - ip: 10.26.124.96
    qosClass: BestEffort
    startTime: "2023-11-23T06:05:21Z"

Let’s conduct an experiment by directly deleting the mirror pod using kubectl and then observing the changes in pod status.

The complete kubelet logs and watch records are available at The log for mirror pod in kubelet.

The UID of the mirror pod is 1facfed2-ba67-40f8-9982-ab86a5e7fd33.

yaml

# kubectl get pod nginx-static-pod-10.11.251.2 -o yaml -w
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292107092"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

The pod is deleted; the deletionTimestamp field has a value, and deletionGracePeriodSeconds is set to 30.

yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2023-11-23T06:11:45Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110805"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

Deleted again, and deletionGracePeriodSeconds is set to 0.

yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-11-23T06:11:15Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110806"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

The mirror pod is removed from the apiserver (only the resourceVersion has changed).

yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-11-23T06:11:15Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110807"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

A new mirror pod is created; the UID is 7bdace8a-8cc5-4cb4-886b-f3961f07d3b8 (the pod’s UID has changed, and only the phase and qosClass fields are present in the status).

yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:11:15Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1


    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110808"
  uid: 7bdace8a-8cc5-4cb4-886b-f3961f07d3b8
  ...
status:
  phase: Pending
  qosClass: BestEffort

The status of the mirror pod is updated.

yaml

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:21Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:29Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:29Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:21Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc
    image: nginx:latest
    imageID: docker-pullable://nginx@sha256:10d1f5b58f74683ad34eb29287e07dab1e90f10af243f151bb50aa5dbb4d62ee
    lastState: {}
    name: nginx-container
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-11-23T06:05:29Z"
  hostIP: 10.11.251.2
  phase: Running
  podIP: 10.26.124.96
  podIPs:
  - ip: 10.26.124.96
  qosClass: BestEffort
  startTime: "2023-11-23T06:05:21Z"

The mirror pod is deleted.

text

I1123 14:11:15.664065  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:11:15.664544  315900 kubelet.go:2130] "SyncLoop DELETE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

podWorker executes syncPod (as static pod is used to determine status in podWorker, syncTerminatingPod is not executed since the static pod is not deleted).

text

I1123 14:11:15.664591  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0
I1123 14:11:15.664610  315900 kubelet.go:1554] "syncPod enter"

In syncPod, mirror pod is deleted again.

text

I1123 14:11:15.664893  315900 kubelet.go:1724] "Trying to delete pod" pod="default/nginx-static-pod-10.11.251.2" podUID=1facfed2-ba67-40f8-9982-ab86a5e7fd33
I1123 14:11:15.664900  315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-10.11.251.2" podUID=1facfed2-ba67-40f8-9982-ab86a5e7fd33

Perceiving the deletion of the pod again, then removing it from the apiserver.

text

I1123 14:11:15.671615  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:11:15.671989  315900 kubelet.go:2130] "SyncLoop DELETE" source="api" pods=[default/nginx-static-pod-10.11.251.2]
I1123 14:11:15.674620  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:11:15.675067  315900 kubelet.go:2124] "SyncLoop REMOVE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

Deletion is completed, and a new mirror pod is created.

text

I1123 14:11:15.675108  315900 kubelet.go:1729] "Deleted mirror pod because it is outdated" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:11:15.675121  315900 kubelet.go:1740] "Creating a mirror pod for static pod" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:11:15.683985  315900 config.go:383] "Receiving a new pod"

syncPod execution is completed.

text

I1123 14:11:15.684056  315900 kubelet.go:1556] "syncPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 isTerminal=false
I1123 14:11:15.684085  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0

podWorker continues to execute syncPod (this is triggered by the previous sync REMOVE).

text

I1123 14:11:15.684093  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0
I1123 14:11:16.053579  315900 kubelet.go:1554] "syncPod enter" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6

Perceiving the newly created mirror pod.

text

I1123 14:11:15.684281  315900 kubelet.go:2114] "SyncLoop ADD" source="api" pods=[default/nginx-static-pod-10.11.251.2]

Executing deletion of the mirror pod again; syncPod execution is completed, and podWorker execution is completed.

text

I1123 14:11:16.053912  315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-10.11.251.2" podUID=1facfed2-ba67-40f8-9982-ab86a5e7fd33
I1123 14:11:16.058466  315900 kubelet.go:1556] "syncPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 isTerminal=false
I1123 14:11:16.058484  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0

podWorker continues to execute syncPod (triggered by SyncLoop ADD), and then execution is completed.

text

I1123 14:11:16.058491  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0
I1123 14:11:17.057755  315900 kubelet.go:1554] "syncPod enter" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:11:17.058187  315900 kubelet.go:1556] "syncPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b

9c6 isTerminal=false
I1123 14:11:17.058208  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0

Mirror pod status update (this is triggered by the syncBatch in statusManager).

text

I1123 14:11:21.465237  315900 kubelet.go:2127] "SyncLoop RECONCILE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

Perceiving the update of the mirror pod status.

text

I1123 14:11:21.465237  315900 kubelet.go:2127] "SyncLoop RECONCILE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

kubelet-mirror-pod-delete

Based on the observations above, deleting a mirror pod does not remove the static pod. Furthermore, when the kubelet recreates a mirror pod, it lacks status conditions and container status. Later, the pod status is synchronized. In reality, operations on mirror pods do not impact static pods because static pods are only visible within the kubelet, and external queries for static pod status can be made through the mirror pod on the apiserver or the /runningpods/ interface on the kubelet’s HTTP server.

From the source code analysis, every time the syncLoop triggers, podWorker executes syncPod.

In the syncPod function, the logic for handling static pods is as follows:

  • If the pod is a static pod and a mirror pod exists, check if the mirror pod is a valid image pod for this static pod. If it is not a valid mirror pod, delete the mirror pod.
  • If the mirror pod has already been deleted, perform the final deletion action (setting deletionGracePeriodSeconds to 0).
  • If the static pod has no mirror pod, create a mirror pod.

The code is located in pkg/kubelet/kubelet.go#L1761-L1798:

go

// Create Mirror Pod for Static Pod if it doesn't already exist
// If it's a static pod, check if the mirror pod exists, and create one when it doesn't
if kubetypes.IsStaticPod(pod) {
    deleted := false
    // If the mirror pod exists
    if mirrorPod != nil {
        // If the mirror pod is deleted or not the mirror pod of the pod
        if mirrorPod.DeletionTimestamp != nil || !kl.podManager.IsMirrorPodOf(mirrorPod, pod) {
            // The mirror pod is semantically different from the static pod. Remove
            // it. The mirror pod will get recreated later.
            klog.InfoS("Trying to delete pod", "pod", klog.KObj(pod), "podUID", mirrorPod.ObjectMeta.UID)
            // pod.Name + "_" + pod.Namespace
            podFullName := kubecontainer.GetPodFullName(pod)
            var err error
            // Call API to delete mirror pod (parse name and namespace from podFullName)
            deleted, err = kl.podManager.DeleteMirrorPod(podFullName, &mirrorPod.ObjectMeta.UID)
            if deleted {
                klog.InfoS("Deleted mirror pod because it is outdated", "pod", klog.KObj(mirrorPod))
            } else if err != nil {
                klog.ErrorS(err, "Failed deleting mirror pod", "pod", klog.KObj(mirrorPod))
            }
        }
    }
    // If there is no mirror pod for static pod or the mirror pod is deleted,
    // then create a new mirror pod as long as there is no error when getting the node
    // and the node is not deleted
    if mirrorPod == nil || deleted {
        node, err := kl.GetNode()
        // If there is an error getting the node or the node is deleted, no need to create a mirror pod
        if err != nil || node.DeletionTimestamp != nil {
            klog.V(4).InfoS("No need to create a mirror pod, since the node has been removed from the cluster", "node", klog.KRef("", string(kl.nodeName)))
        } else {
            klog.V(4).InfoS("Creating a mirror pod for a static pod", "pod", klog.KObj(pod))
            // If the node is not deleted, create a mirror pod for the static pod
            if err := kl.podManager.CreateMirrorPod(pod); err != nil {
                klog.ErrorS(err, "Failed creating a mirror pod for", "pod", klog.KObj(pod))
            }
        }
    }
}

Related Content

MigrationController Plugin of Koordinator Descheduler: Ensuring Successful Scheduling of Pods After Eviction

Koordinator Descheduler: LowNodeLoad Plugin Enhancing Node Balance and Application Stability

Best Practices for Cost Optimization in Kubernetes

Why Evicted Pods are not deleted and How to cleanup

Updated on 2023-12-01
 Kubernetes, Kubelet
 | Home