Home avatar

Don't forget why you set off

Troubleshooting kubelet no container metric issues

Background: There are two versions of Kubernetes, 1.18.20 and 1.21.8. The kubelet is started using systemd with the same startup parameters. In Kubernetes v1.21, there is no container-related data, while in v1.18, there is monitoring data related to containers. The operating system is CentOS 7.

Observation: In the v1.21 cluster, the Grafana monitoring graphs for pods do not display any data related to containers, such as CPU and memory usage.

Exploring Path-Based and Header-Based Routing Solutions in Knative

Currently, in Knative version 1.1.0, traffic routing is primarily based on domain names.

However, in many use cases:

  1. Services often have fixed external domain names, and there may be multiple of them.
  2. Services are typically organized under specific paths within a domain, meaning one domain consists of multiple services.
  3. Grey releases are based on multiple header values with and and or relationships.

Let’s discuss how to address these requirements.

Apart from using Knative’s default domain, Knative provides domainMapping to bind additional domains.

kubelet podConfig - Providing Pods Running on kubelet

The podConfig within the kubelet is used to aggregate information from multiple sources related to pods. It processes and aggregates information from various sources of pods (deduplication, analysis of changes in pods). If there are changes in pods from these sources, podConfig updates and aggregates the changes, generating various types of PodUpdate messages and sending them to the PodUpdate channel.

Event Generation Mechanism in Kubernetes

When troubleshooting an issue in a cluster, there are generally two ways to investigate problems. One is to check the logs of various components to identify the issue, while the other is to retrieve event events through the API server and analyze what actions a component has taken based on those events.

The event event mechanism makes it easy for us to troubleshoot problems. An event is an API object that records what action a component took at a certain time. Each component acts as a client, sending requests to the API server to create or update event events. By default, the API server retains event events for one hour.

Smooth Rolling Update of Ingress Controller Using AWS NLB - No Service Interruption

In the cloud, to allow external traffic to access your Kubernetes cluster, the typical approach is to use a LoadBalancer-type service, which directs external traffic through a load balancer into the Ingress Controller and then distributes it to various pods.

Ingress controllers often need updates or configuration changes, which usually require the program to be restarted. So, what happens when an Ingress Controller needs a rolling update? Will Ingress access be interrupted?

Kubelet Startup - Command Execution

In the previous analysis, we explored the initialization of kubelet’s command-line. Now, we will delve into the execution of kubelet’s Cobra Command. The Execute() function in Cobra essentially involves command-line parsing (unless DisableFlagParsing: true is set), followed by passing the remaining arguments to the Command.Run function for execution.

This analysis is based on Kubernetes version 1.18.6. For the source code and related readings, please visit the source code analysis repository.

kubenet IP Leak

Recently, after upgrading the Docker version, some pods remained in the ‘pending’ state, and it was found that the reason for the pods’ pending status was the inability to obtain an IP address. After investigation, it was discovered that the Docker version upgrade was performed incorrectly, leading to an IP leak in kubenet, which resulted in no available IPs for allocation.

The Kubernetes version used was 1.18.8, with the network mode set to kubenet, a maximum of 125 pods per node, and a pod CIDR of 25.