Recently, while developing a CNI network plugin, I encountered an issue where containers couldn’t access the LoadBalancer IP of a service. However, access to pod, service, and node addresses was functioning correctly.

1 Background

1. The network plugin is based on the cloud provider’s Elastic Network Interface (ENI), with multiple pods sharing a single ENI in a policy routing mode. This setup directs pod traffic in and out through the ENI.
2. The primary network interface is bound to an IP, while the secondary network interfaces (ENIs) do not have IP bindings. The IPs of the pods are assigned to the secondary network interfaces.
3. kube-proxy binds the LoadBalancer IP to the kube-ipvs0 network interface, adds IPVS rules for LoadBalancer IP forwarding, and sets up SNAT rules in iptables to allow pod IPs to access the LoadBalancer IP.

2 Symptom

When attempting to access the LoadBalancer address 10.12.115.101:80 from within a container, connectivity issues occur:

Background: There are two versions of Kubernetes, 1.18.20 and 1.21.8. The kubelet is started using systemd with the same startup parameters. In Kubernetes v1.21, there is no container-related data, while in v1.18, there is monitoring data related to containers. The operating system is CentOS 7.

Observation: In the v1.21 cluster, the Grafana monitoring graphs for pods do not display any data related to containers, such as CPU and memory usage.

Currently, in Knative version 1.1.0, traffic routing is primarily based on domain names.

However, in many use cases:

1. Services often have fixed external domain names, and there may be multiple of them.
2. Services are typically organized under specific paths within a domain, meaning one domain consists of multiple services.
3. Grey releases are based on multiple header values with and and or relationships.

Let’s discuss how to address these requirements.

1 Domain Binding

Apart from using Knative’s default domain, Knative provides domainMapping to bind additional domains.

The podConfig within the kubelet is used to aggregate information from multiple sources related to pods. It processes and aggregates information from various sources of pods (deduplication, analysis of changes in pods). If there are changes in pods from these sources, podConfig updates and aggregates the changes, generating various types of PodUpdate messages and sending them to the PodUpdate channel.

When troubleshooting an issue in a cluster, there are generally two ways to investigate problems. One is to check the logs of various components to identify the issue, while the other is to retrieve event events through the API server and analyze what actions a component has taken based on those events.

The event event mechanism makes it easy for us to troubleshoot problems. An event is an API object that records what action a component took at a certain time. Each component acts as a client, sending requests to the API server to create or update event events. By default, the API server retains event events for one hour.

1 Overview

In the cloud, to allow external traffic to access your Kubernetes cluster, the typical approach is to use a LoadBalancer-type service, which directs external traffic through a load balancer into the Ingress Controller and then distributes it to various pods.

Ingress controllers often need updates or configuration changes, which usually require the program to be restarted. So, what happens when an Ingress Controller needs a rolling update? Will Ingress access be interrupted?

In the previous analysis, we explored the initialization of kubelet’s command-line. Now, we will delve into the execution of kubelet’s Cobra Command. The Execute() function in Cobra essentially involves command-line parsing (unless DisableFlagParsing: true is set), followed by passing the remaining arguments to the Command.Run function for execution.

This analysis is based on Kubernetes version 1.18.6. For the source code and related readings, please visit the source code analysis repository.