Posts Tags Categories About Resume English

modify the contents of /etc/resolv.conf when the pod is running

xiaoqing  included in  category kubernetes
     1335 words  7 minutes 

Kubernetes provides a method to modify the configuration of the /etc/resolv.conf file for pods using the spec.dnsConfig and spec.dnsPolicy fields. You can find specific information on this in the Customizing DNS Service documentation. However, this approach leads to the recreation of pods.

In our specific business scenario, we need pods to use local DNS instead of the centralized CoreDNS, even for pods created before the change in cluster DNS configuration. We need to update the nameserver for these existing pods to point to the local DNS server. However, we cannot actively delete pods or restart containers. This practice is not considered ideal in container usage, but it aligns with our company culture, as the business application doesn’t support graceful termination.

The first method involves directly modifying the /etc/resolv.conf file inside containers. Since containers run using Docker, all containers within a pod share the same mounted /etc/resolv.conf file. Therefore, modifying the /etc/resolv.conf in one container achieves the goal of changing the nameserver for the entire pod.

All containers within the pod use the same resolv.conf file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# docker ps
a15fc9af0da0   75c6ae8cb401                                                         "/app/push-netstat -…"   5 seconds ago   Up 4 seconds             k8s_main-container_push-netstat-2pwbn_ops_b90f9867-6c13-4242-9904-26354f2b65df_0
9007620aa2cb   k8s-google-containers/pause-amd64:3.2      "/pause"                 6 seconds ago   Up 5 seconds             k8s_POD_push-netstat-2pwbn_ops_b90f9867-6c13-4242-9904-26354f2b65df_0
# docker inspect 9007620aa2cb 
[
    {
        "Id": "9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0",
        "Created": "2022-12-01T04:51:14.862746022Z",
        "Path": "/pause",
        "Image": "sha256:80d28bedfe5dec59da9ebf8e6260224ac9008ab5c11dbbe16ee3ba3e4439ac2c",
        "ResolvConfPath": "/data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf",

# docker inspect a15fc9af0da0
[
    {
        "Id": "a15fc9af0da02a93a9c84f3528cc3970a44157c3b63d1d2efa17e8694cded9b5",
        "Created": "2022-12-01T04:51:15.425130718Z",
        "ResolvConfPath": "/data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf",

#cat /proc/1/mountinfo 

952 703 253:1 /usr/share/zoneinfo/Asia/Shanghai /usr/share/zoneinfo/UCT ro,noatime - ext4 /dev/vda1 rw,stripe=32411,data=ordered
953 705 253:17 /kubernetes/kubelet/pods/b90f9867-6c13-4242-9904-26354f2b65df/containers/main-container/7adc41b0 /dev/termination-log rw,relatime - ext4 /dev/vdb1 rw,data=ordered
954 703 253:17 /kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/vdb1 rw,data=ordered
955 703 253:17 /kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/hostname /etc/hostname rw,relatime - ext4 /dev/vdb1 rw,data=ordered
956 703 253:17 /kubernetes/kubelet/pods/b90f9867-6c13-4242-9904-26354f2b65df/etc-hosts /etc/hosts rw,relatime - ext4 /dev/vdb1 rw,data=ordered
957 705 0:122 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
958 703 0:120 / /run/secrets/kubernetes.io/serviceaccount ro,relatime - tmpfs tmpfs rw,size=1048576k

Drawback: Requires a program inside the container that can modify files, like echo (you cannot use vim or sed directly because they create a new file and overwrite the source file). Also, it requires the container to run as the root user (or it will result in “Permission denied” errors).

Advantage: Simple and straightforward.

Using sed inside a container run as root results in an error.

1
2

# sed -i '$a \sdasd' /etc/resolv.conf 
sed: cannot rename /etc/sedz6ITFW: Device or resource busy

Modifying the file inside a non-root user container results in a “Permission denied” error.

1
2

$ echo a >>  /etc/resolv.conf 
bash: /etc/resolv.conf: Permission denied

The second method involves modifying the source file mounted on the host corresponding to /etc/resolv.conf inside Docker containers. Similar to the first method, you cannot use vim or sed directly for this, but you can use the echo command to make changes.

The original idea for this method comes from dockershim, which modifies /etc/resolv.conf inside the sandbox.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

	// Rewrite resolv.conf file generated by docker.
	// NOTE: cluster dns settings aren't passed anymore to docker api in all cases,
	// not only for pods with host network: the resolver conf will be overwritten
	// after sandbox creation to override docker's behaviour. This resolv.conf
	// file is shared by all containers of the same pod, and needs to be modified
	// only once per pod.
	if dnsConfig := config.GetDnsConfig(); dnsConfig != nil {
		containerInfo, err := ds.client.InspectContainer(createResp.ID)
		if err != nil {
			return nil, fmt.Errorf("failed to inspect sandbox container for pod %q: %v", config.Metadata.Name, err)
		}

		if err := rewriteResolvFile(containerInfo.ResolvConfPath, dnsConfig.Servers, dnsConfig.Searches, dnsConfig.Options); err != nil {
			return nil, fmt.Errorf("rewrite resolv.conf failed for pod %q: %v", config.Metadata.Name, err)
		}
	}

https://github.com/kubernetes/kubernetes/blob/e17755904ae82971e954820187961636c350c6ff/pkg/kubelet/dockershim/docker_sandbox.go#L147-L162

Drawback: Requires root permissions on the host machine and is only applicable when Docker is the container runtime.

Advantage: It’s a simple method.

Modifying the file using vim on the host doesn’t take effect inside the container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#options ndots:5 --> options ndots:4
# vim /data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf
nameserver 169.254.20.10
search xxx-ops.svc.cluster.local svc.cluster.local cluster.local internal.xxx.com
options ndots:4

# doesn't take effect inside the container
# kubectl exec -it -n ops push-netstat-2pwbn cat /etc/resolv.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
nameserver 169.254.20.10
search xxx-ops.svc.cluster.local svc.cluster.local cluster.local internal.xxx.com
options ndots:5

# cat /data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf
nameserver 169.254.20.10
search xxx-ops.svc.cluster.local svc.cluster.local cluster.local internal.xxx.com
options ndots:4

# run a container with same network namespace, but /etc/resolv.conf is changed
#docker run -it --name test-resolv-conf --network container:9007620aa2cb  ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
e96e057aae67: Pull complete 
Digest: sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca4005c2
Status: Downloaded newer image for ubuntu:latest

root@push-netstat-5pxlr:/# cat /etc/resolv.conf 
nameserver 169.254.20.10
search xxx-ops.svc.cluster.local svc.cluster.local cluster.local internal.xxx.com
options ndots:4

Using the echo command to modify the mounted source file on the host allows the changes to be reflected inside the container. This method enables the container to detect and use the modified /etc/resolv.conf file on the host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# echo 11 >> /data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf
# ll -i /data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf
3276908 -rw-r--r-- 1 root root 132 Dec  1 13:01 /data/kubernetes/docker/containers/9007620aa2cb23f2d91a3fb194771c39be45d75e73ef36a3f78b99a4574a1cb0/resolv.conf

# kubectl exec -it -n ops push-netstat-2pwbn /bin/cat /etc/resolv.conf 
nameserver 169.254.20.10
search xxx-ops.svc.cluster.local svc.cluster.local cluster.local internal.xxx.com
options ndots:5
11
# kubectl exec -it -n ops push-netstat-2pwbn /bin/ls /etc/resolv.conf -i
3276908 /etc/resolv.conf

Even though the content of the /etc/resolv.conf file has been updated, does the application re-read the /etc/resolv.conf file?

Different applications exhibit different behaviors:

Java JVM: By default, it only reads the file at startup and does not re-read /etc/resolv.conf once it’s running.

As a program (any process, JVM included) has its very first DNS request it reads and caches forever the entire contents of /etc/resolv.conf by default. It never refreshes that info later, even when it encounters a total DNS failure. The program would need to have some specific system calls programmed to behave in more user-friendly manner. This SO question explains the details.

https://serverfault.com/a/901790/405334

networkaddress.cache.ttl

Specified in java.security to indicate the caching policy for successful name lookups from the name service.. The value is specified as integer to indicate the number of seconds to cache the successful lookup.

A value of -1 indicates “cache forever”. The default behavior is to cache forever when a security manager is installed, and to cache for an implementation specific period of time, when a security manager is not installed.

https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html

Golang: When using the net/http library in Go 1.17 for testing, Go will re-read /etc/resolv.conf.

The behavior of other applications may vary. Some applications periodically check the DNS configuration file for updates, while others read it once at startup and keep it unchanged. To understand the behavior of a specific application, you may need to consult the documentation or configuration options for that application or perform testing to determine how it handles changes to the /etc/resolv.conf file.

There are two methods to modify the /etc/resolv.conf file of pods: using Kubernetes’ native support with spec.dnsConfig and spec.dnsPolicy, directly modifying the /etc/resolv.conf inside containers, and altering the resolv.conf based on Docker’s characteristics.

The first two methods are generally applicable, while the third method is specific to Docker (or containerd) as the container runtime. When modifying the /etc/resolv.conf file of a running container, it does not necessarily trigger applications to reread its contents. Different applications behave differently, with some reading the /etc/resolv.conf file only once at startup and not rereading it, while others may periodically reread it for updated DNS configurations.

how to graceful update the node /etc/resolv.conf on kubernetes cluster see this articles

Related Content

How to Avoid Automatically Activating Windows when Connected to the Internet

Koordinator Descheduler: LowNodeLoad Plugin Enhancing Node Balance and Application Stability

Best Practices for Cost Optimization in Kubernetes

Why Evicted Pods are not deleted and How to cleanup

A Deep Dive into HighNodeUtilization and LowNodeUtilization Plugins with Descheduler

Updated on 2022-12-02
 kubernetes, skill, dns, experience sharing
Back | Home