Gracefully Changing the DNS Server IP for node on a Kubernetes Cluster Without Impacting Applications

xiaoqing published on 2023-08-23 included in category kubernetes

DNS servers are typically stable components of infrastructure and are rarely changed. However, if the IP address of a DNS server needs to be updated, here’s how to change the DNS configuration of Kubernetes nodes. 1 Standard ProcedureTo change the node’s DNS configuration, follow these steps: Replace the DNS server IP address directly in the /etc/resolv.conf file on node. The above step updates only the /etc/resolv.conf file on the node. It does not update the /etc/resolv.

practice of Karmada as cluster resource synchronization in disaster recovery systems

xiaoqing published on 2023-08-19 included in category kubernetes

1 Karmada: What is it?Karmada is a Kubernetes multi-cluster management system. It allows resources to be distributed across multiple clusters while maintaining the original way of using the API server. It offers features such as cross-cloud multi-cluster management, multi-policy multi-cluster scheduling, cross-cluster fault tolerance for applications, a global unified resource view, multi-cluster service discovery, and Federated Horizontal Pod Autoscaling (FederatedHPA) capabilities. Its design philosophy inherits from Cluster Federation v2 and is currently an open-source CNCF sandbox project.

Research the principle of metadata.generation value increase

xiaoqing published on 2023-04-22 included in category kubernetes

1 Background

Use kubebuilder to develop a vpa-related operator. This operator will watch all vpa creation, deletion and update in the cluster. controller-runtime provides predict to filter out unnecessary events, and use predicate.GenerationChangedPredicate to filter out vpa update status. However, it was found that the status update of vpa (recommended value updated by vpa-recommender) also triggered Reconcile.

The pod always scheduling to the same node

xiaoqing published on 2023-03-22 included in category kubernetes

Encountered Strange Phenomenon: Spark-generated job pods are consistently scheduled on the same node, meaning that pods from different jobs are all being scheduled to the same node. This results in an uneven distribution of pods, even though the nodes have no taints, and their resource availability is similar. The jobs do not have any nodeSelector, nodeAffinity, nodeName, or PodTopologySpread.

Resource Recommendation Algorithms for Crane and VPA

xiaoqing published on 2023-02-18 included in category kubernetes

Introduction to VPAVPA, short for Vertical Pod Autoscaler, is an open-source implementation based on the Google paper Autopilot: Workload Autoscaling at Google Scale. It recommends container resource requests based on historical monitoring data from the containers within pods. In other words, VPA scales by directly modifying the resource requests (and limits, if configured in VPA resources) within the pod. Key Benefits: Increases node resource utilization. Suitable for long-running, homogeneous applications.

modify the contents of /etc/resolv.conf when the pod is running

xiaoqing published on 2022-12-02 included in category kubernetes

Kubernetes provides a method to modify the configuration of the /etc/resolv.conf file for pods using the spec.dnsConfig and spec.dnsPolicy fields. You can find specific information on this in the Customizing DNS Service documentation. However, this approach leads to the recreation of pods.

In our specific business scenario, we need pods to use local DNS instead of the centralized CoreDNS, even for pods created before the change in cluster DNS configuration. We need to update the nameserver for these existing pods to point to the local DNS server. However, we cannot actively delete pods or restart containers. This practice is not considered ideal in container usage, but it aligns with our company culture, as the business application doesn’t support graceful termination.

many pod probes are failing after upgrading Kubernetes

xiaoqing published on 2022-09-17 included in category kubernetes

BackgroundAfter upgrading the Kubernetes cluster from version 1.18 to 1.23, many previously running pods are experiencing restarts due to liveness probe failures, and some are failing readiness probes as well. Initially, it was suspected that the pod restarts were not caused by the Kubernetes upgrade, as the container hash algorithm remained unchanged between versions 1.18 and 1.23. Therefore, upgrading kubelet should not result in the regeneration of already running containers.