Recently, during a surge in business activity, we encountered a problem where the availability of our services decreased due to a sudden increase in resource usage caused by incoming traffic. There are various reasons for the unavailability of our services, and one of the direct causes is the sharp increase in resource usage when traffic surges, and the HPA does not scale in a timely manner.

This article aims to investigate this issue and primarily addresses it from three aspects:

1. How slow is the scaling process?
2. Why is the scaling process slow?
3. What are the solutions to this problem?

1 kubectl apply Cannot Remove Annotations

I usually prefer deploying applications using YAML, and recently, when using kubectl apply, I encountered an issue. When I use kubectl rollout restart to restart an application, kubectl adds kubectl.kubernetes.io/restartedAt: <current time> to spec.template.metadata.annotations. Then, when I update the YAML file and use kubectl apply, it doesn’t remove the kubectl.kubernetes.io/restartedAt: "2022-07-26T11:44:32+08:00" from the annotations.

First of all, I would like to thank the karmada community for providing the tickets to KubeCon, and I met with core contributors and maintainers of karmada such as zhen chang, hongcai Ren, and Wei jiang.

In previous years, attending the technology conference left me with no deep impression and no harvest. This time I forced myself to record it, deepen my impression and summarize my gains.

I am interested in colocation, so the sharing I listened to is basically related to this.

updated: all of the kubeCon china 2023 videos is released. youtube video list, WeChat subscription account

slides: https://kccncosschn2023.sched.com/?iframe=no

istioCon china 2023 slides: https://istioconchina2023.sched.com/ https://github.com/cloudnativeto/academy/tree/master/istiocon-china-2023

DNS servers are typically stable components of infrastructure and are rarely changed. However, if the IP address of a DNS server needs to be updated, here’s how to change the DNS configuration of Kubernetes nodes.

1 Standard Procedure

To change the node’s DNS configuration, follow these steps:

1. Replace the DNS server IP address directly in the /etc/resolv.conf file on node.

2. The above step updates only the /etc/resolv.conf file on the node. It does not update the /etc/resolv.conf inside the Node Local DNS and CoreDNS pods.

1 Karmada: What is it?

Karmada is a Kubernetes multi-cluster management system. It allows resources to be distributed across multiple clusters while maintaining the original way of using the API server. It offers features such as cross-cloud multi-cluster management, multi-policy multi-cluster scheduling, cross-cluster fault tolerance for applications, a global unified resource view, multi-cluster service discovery, and Federated Horizontal Pod Autoscaling (FederatedHPA) capabilities. Its design philosophy inherits from Cluster Federation v2 and is currently an open-source CNCF sandbox project. For more details, visit the official website at https://karmada.io/.

1 Background

Use kubebuilder to develop a vpa-related operator. This operator will watch all vpa creation, deletion and update in the cluster. controller-runtime provides predict to filter out unnecessary events, and use predicate.GenerationChangedPredicate to filter out vpa update status. However, it was found that the status update of vpa (recommended value updated by vpa-recommender) also triggered Reconcile.

Encountered Strange Phenomenon: Spark-generated job pods are consistently scheduled on the same node, meaning that pods from different jobs are all being scheduled to the same node. This results in an uneven distribution of pods, even though the nodes have no taints, and their resource availability is similar. The jobs do not have any nodeSelector, nodeAffinity, nodeName, or PodTopologySpread.