所有文章 标签 分类 关于 简历 简体中文

深入探索Kubernetes中的Mirror Pod删除过程及其对Static Pod 的影响

xiaoqing  收录于  类别 kubernetes
     约 3474 字  预计阅读 7 分钟 

这也是一篇研究pod移除流程的文章，这个文章专注mirror pod的移除。mirror pod这个名词可能很陌生，中文直译就是镜像pod，它是pod里的一种类型。

那我们先介绍pod的分类，pod来源分为file、http、apiserver。其中来自apiserver中的pod称为普通pod，其他来源的pod称为static pod（使用kubeadm安装集群的控制平面就是用static pod运行的）。为了管理pod方便，kubelet会在apiserver上为static pod生成对应的pod，这类型pod称为mirror pod，即可以理解是这个pod的替身（基本上跟static pod一模一样，只有uid不一样和annotations里多了"kubernetes.io/config.mirror"）。

那么删除mirror pod会发生什么？对节点上static pod有没有影响，会不会移除节点上的static pod？

pod移除流程系列文章

本文使用kubernete版本是1.23和日志级别为4。

kubelet的static pod的机制生成的pod name为配置里的pod name加上node节点名。在这个例子中，定义的pod name为nginx-static-pod，而最终生成的pod名字为nginx-static-pod-10.11.251.2。

下面是kubelet上的static pod配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: Pod
metadata:
  name: nginx-static-pod
  namespace: default
spec:
  containers:
  - name: nginx-container
    image: nginx:latest
    ports:
    - containerPort: 80

对应的mirror pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292107092"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33
spec:
  containers:
  - image: nginx:latest
    imagePullPolicy: Always
    name: nginx-container
    ports:
    - containerPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: 10.11.251.2
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    operator: Exists
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:21Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:29Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:29Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:21Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc
    image: nginx:latest
    imageID: docker-pullable://nginx@sha256:10d1f5b58f74683ad34eb29287e07dab1e90f10af243f151bb50aa5dbb4d62ee
    lastState: {}
    name: nginx-container
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-11-23T06:05:29Z"
  hostIP: 10.11.251.2
  phase: Running
  podIP: 10.26.124.96
  podIPs:
  - ip: 10.26.124.96
  qosClass: BestEffort
  startTime: "2023-11-23T06:05:21Z"

我们直接做个实验，使用kubectl删除mirror pod，然后watch pod变化

完整的kubelet日志和watch记录在 The log for mirror pod in kubelet

mirror pod的uid为1facfed2-ba67-40f8-9982-ab86a5e7fd33

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# kubectl get pod nginx-static-pod-10.11.251.2 -o yaml -w
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292107092"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

pod被删除，deletionTimestamp字段有值，且deletionGracePeriodSeconds为30

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2023-11-23T06:11:45Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110805"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

再次被删除，deletionGracePeriodSeconds设置为0

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-11-23T06:11:15Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110806"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

mirror pod从apiserver中移除（这里只有resourceVersion发生了变化）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:05:21Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-11-23T06:11:15Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110807"
  uid: 1facfed2-ba67-40f8-9982-ab86a5e7fd33

新的mirror pod创建，uid为7bdace8a-8cc5-4cb4-886b-f3961f07d3b8（pod的uid发生变化，status里只有phase和qosClass字段）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.mirror: a8712c005851ee6b29cff91b9ab4b9c6
    kubernetes.io/config.seen: "2023-11-23T14:05:21.400551188+08:00"
    kubernetes.io/config.source: file
  creationTimestamp: "2023-11-23T06:11:15Z"
  name: nginx-static-pod-10.11.251.2
  namespace: default
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: 10.11.251.2
    uid: d2e3ebf6-2a70-434f-9142-0f74a1dd16bc
  resourceVersion: "292110808"
  uid: 7bdace8a-8cc5-4cb4-886b-f3961f07d3b8
  ...
status:
  phase: Pending
  qosClass: BestEffort

mirror pod的status更新

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:21Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:29Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:29Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-11-23T06:05:21Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://b6ca55d329230c8f5776eb1160fe161d6fefa01f2b31e55dbb820add90aadccc
    image: nginx:latest
    imageID: docker-pullable://nginx@sha256:10d1f5b58f74683ad34eb29287e07dab1e90f10af243f151bb50aa5dbb4d62ee
    lastState: {}
    name: nginx-container
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-11-23T06:05:29Z"
  hostIP: 10.11.251.2
  phase: Running
  podIP: 10.26.124.96
  podIPs:
  - ip: 10.26.124.96
  qosClass: BestEffort
  startTime: "2023-11-23T06:05:21Z"

mirror pod被删除

1
2

I1123 14:11:15.664065  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:11:15.664544  315900 kubelet.go:2130] "SyncLoop DELETE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

podWorker执行syncPod（由于在podWorker中使用static pod来判断状态，而static pod未被删除，所以不执行syncTerminatingPod）

1
2

I1123 14:11:15.664591  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0
I1123 14:11:15.664610  315900 kubelet.go:1554] "syncPod enter"

在syncPod中执行再次删除mirror pod

1
2

I1123 14:11:15.664893  315900 kubelet.go:1724] "Trying to delete pod" pod="default/nginx-static-pod-10.11.251.2" podUID=1facfed2-ba67-40f8-9982-ab86a5e7fd33
I1123 14:11:15.664900  315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-10.11.251.2" podUID=1facfed2-ba67-40f8-9982-ab86a5e7fd33

感知到pod再次删除，随后从apiserver中移除

1
2
3
4

I1123 14:11:15.671615  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:11:15.671989  315900 kubelet.go:2130] "SyncLoop DELETE" source="api" pods=[default/nginx-static-pod-10.11.251.2]
I1123 14:11:15.674620  315900 config.go:278] "Setting pods for source" source="api"
I1123 14:11:15.675067  315900 kubelet.go:2124] "SyncLoop REMOVE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

删除完成并创建新的mirror pod

1
2
3

I1123 14:11:15.675108  315900 kubelet.go:1729] "Deleted mirror pod because it is outdated" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:11:15.675121  315900 kubelet.go:1740] "Creating a mirror pod for static pod" pod="default/nginx-static-pod-10.11.251.2"
I1123 14:11:15.683985  315900 config.go:383] "Receiving a new pod"

syncPod执行完成

1
2

I1123 14:11:15.684056  315900 kubelet.go:1556] "syncPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 isTerminal=false
I1123 14:11:15.684085  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0

podWorker继续执行syncPod（这个是之前sync REMOVE触发）

1
2

I1123 14:11:15.684093  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0
I1123 14:11:16.053579  315900 kubelet.go:1554] "syncPod enter" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6

感知到新创建的mirror pod

1

I1123 14:11:15.684281  315900 kubelet.go:2114] "SyncLoop ADD" source="api" pods=[default/nginx-static-pod-10.11.251.2]

重新执行删除mirror pod，syncPod执行完成，podWorker执行完成

1
2
3

I1123 14:11:16.053912  315900 mirror_client.go:130] "Deleting a mirror pod" pod="default/nginx-static-pod-10.11.251.2" podUID=1facfed2-ba67-40f8-9982-ab86a5e7fd33
I1123 14:11:16.058466  315900 kubelet.go:1556] "syncPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 isTerminal=false
I1123 14:11:16.058484  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0

podWorker继续执行syncPod（SyncLoop ADD触发的），然后执行完成

1
2
3
4

I1123 14:11:16.058491  315900 pod_workers.go:888] "Processing pod event" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0
I1123 14:11:17.057755  315900 kubelet.go:1554] "syncPod enter" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6
I1123 14:11:17.058187  315900 kubelet.go:1556] "syncPod exit" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 isTerminal=false
I1123 14:11:17.058208  315900 pod_workers.go:988] "Processing pod event done" pod="default/nginx-static-pod-10.11.251.2" podUID=a8712c005851ee6b29cff91b9ab4b9c6 updateType=0

mirror pod status更新（这个是由statusManager里的syncBatch触发更新）

1

I1123 14:11:21.465237  315900 kubelet.go:2127] "SyncLoop RECONCILE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

感知到mirror pod status更新

1

I1123 14:11:21.465237  315900 kubelet.go:2127] "SyncLoop RECONCILE" source="api" pods=[default/nginx-static-pod-10.11.251.2]

kubelet-mirror-pod-delete

根据上面的现象删除mirror pod并不会移除static pod，而且kubelet还会重新创建一个新的mirror pod，创建出来的mirror pod并没有status condition和container status，最后kubelet进行mirror pod的status更新。实际上对mirror pod的操作并不会影响static pod，因为static pod只在kubelet内部可见，外部可以通过apiserver上的mirror pod或kubelet的HTTP server上的/runningpods/接口进行查询static pod状态。

从源码方面分析，每次syncLoop触发podWorker都执行syncPod。

在syncPod里处理static pod逻辑：

  • 如果该pod是static pod且mirror pod存在，检测这个mirror pod是否是这个static pod的合法镜像pod。如果不是合法mirror pod，则把这个mirror pod删除。
  • 如果这个mirror pod已经被删除，则执行最后删除动作（设置deletionGracePeriodSeconds为0）。
  • 如果static pod没有mirror pod，则创建mirror pod。

代码在 pkg/kubelet/kubelet.go#L1761-L1798

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

	// Create Mirror Pod for Static Pod if it doesn't already exist
	// 如果是static pod，则检查是否存在mirror pod，当mirror pod不存在时候进行创建
	if kubetypes.IsStaticPod(pod) {
		deleted := false
		// 存在mirror pod
		if mirrorPod != nil {
			// mirrorPod被删除或mirrorPod不是pod的mirror pod
			if mirrorPod.DeletionTimestamp != nil || !kl.podManager.IsMirrorPodOf(mirrorPod, pod) {
				// The mirror pod is semantically different from the static pod. Remove
				// it. The mirror pod will get recreated later.
				klog.InfoS("Trying to delete pod", "pod", klog.KObj(pod), "podUID", mirrorPod.ObjectMeta.UID)
				// pod.Name + "_" + pod.Namespace
				podFullName := kubecontainer.GetPodFullName(pod)
				var err error
				// 调用api删除mirror pod（从podFullName解析出name和namespace）
				deleted, err = kl.podManager.DeleteMirrorPod(podFullName, &mirrorPod.ObjectMeta.UID)
				if deleted {
					klog.InfoS("Deleted mirror pod because it is outdated", "pod", klog.KObj(mirrorPod))
				} else if err != nil {
					klog.ErrorS(err, "Failed deleting mirror pod", "pod", klog.KObj(mirrorPod))
				}
			}
		}
		// static pod没有mirror pod或mirror pod被删除，则在获取node未发生错误且node未被删除条件下，创建mirror pod
		if mirrorPod == nil || deleted {
			node, err := kl.GetNode()
			// 当获取node发生错误或node被删除，则无需创建mirror pod
			if err != nil || node.DeletionTimestamp != nil {
				klog.V(4).InfoS("No need to create a mirror pod, since node has been removed from the cluster", "node", klog.KRef("", string(kl.nodeName)))
			} else {
				klog.V(4).InfoS("Creating a mirror pod for static pod", "pod", klog.KObj(pod))
				// node没有被删除，则创建static pod的mirror pod
				if err := kl.podManager.CreateMirrorPod(pod); err != nil {
					klog.ErrorS(err, "Failed creating a mirror pod for", "pod", klog.KObj(pod))
				}
			}
		}
	}

相关内容

Koordinator Descheduler的MigrationController插件：确保驱逐后Pod成功调度

koordinator Descheduler的LowNodeLoad插件助力节点平衡与应用稳定性的提升

Kubernetes成本优化最佳实践

为什么被驱逐evicted pod未被删除以及如何清理

深度解析descheduler的highNodeUtilization和lowNodeUtilization插件的原理

更新于 2023-12-01
 kubernetes, kubelet, pod移除流程
返回 | 主页