About 4 months ago, while troubleshooting a bug in the CNI plugin that caused pod removal failures, I examined the deletion process of Kubernetes 1.23 version pods. In the kubelet logs, I frequently encountered the error “an error occurred when try to find container,” which I had previously ignored. This time, I decided to analyze the root cause of this error.

This article will analyze the following aspects:

1. Introduction to the core components of pod lifecycle management in kubelet.
2. Analysis of the actual pod removal process based on the logs output during the pod removal process in kubelet.

Before we begin, if you ask me how serious this error is and what impact it has, my answer is that it doesn’t matter. This issue is caused by inconsistencies in asynchronous and cached information, and it does not affect the execution of the pod deletion and cleanup process. If you want to know the reason, continue reading; if not, you can close this article directly because it is lengthy and not suitable for troubleshooting.

In the previous article “Why Does HPA Scale Slowly”, an analysis of the principles and algorithms of Horizontal Pod Autoscaler (HPA) scaling was conducted, along with some solutions. One interesting topic related to HPA is “Can HPA scale down to 0”, a requirement aimed at cost reduction, and it is a common need in serverless environments.

Recently, during a surge in business activity, we encountered a problem where the availability of our services decreased due to a sudden increase in resource usage caused by incoming traffic. There are various reasons for the unavailability of our services, and one of the direct causes is the sharp increase in resource usage when traffic surges, and the HPA does not scale in a timely manner.

This article aims to investigate this issue and primarily addresses it from three aspects:

1. How slow is the scaling process?
2. Why is the scaling process slow?
3. What are the solutions to this problem?

1 kubectl apply Cannot Remove Annotations

I usually prefer deploying applications using YAML, and recently, when using kubectl apply, I encountered an issue. When I use kubectl rollout restart to restart an application, kubectl adds kubectl.kubernetes.io/restartedAt: <current time> to spec.template.metadata.annotations. Then, when I update the YAML file and use kubectl apply, it doesn’t remove the kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00" from the annotations.

First of all, I would like to thank the karmada community for providing the tickets to KubeCon, and I met with core contributors and maintainers of karmada such as zhen chang, hongcai Ren, and Wei jiang.

In previous years, attending the technology conference left me with no deep impression and no harvest. This time I forced myself to record it, deepen my impression and summarize my gains.

I am interested in colocation, so the sharing I listened to is basically related to this.

updated: all of the kubeCon china 2023 videos is released. youtube video list, WeChat subscription account

slides: https://kccncosschn2023.sched.com/?iframe=no

istioCon china 2023 slides: https://istioconchina2023.sched.com/ https://github.com/cloudnativeto/academy/tree/master/istiocon-china-2023