modify the contents of /etc/resolv.conf when the pod is running
Kubernetes provides a method to modify the configuration of the
/etc/resolv.conf file for pods using the
spec.dnsPolicy fields. You can find specific information on this in the Customizing DNS Service documentation. However, this approach leads to the recreation of pods.
In our specific business scenario, we need pods to use local DNS instead of the centralized CoreDNS, even for pods created before the change in cluster DNS configuration. We need to update the nameserver for these existing pods to point to the local DNS server. However, we cannot actively delete pods or restart containers. This practice is not considered ideal in container usage, but it aligns with our company culture, as the business application doesn’t support graceful termination.
The first method involves directly modifying the
/etc/resolv.conf file inside containers. Since containers run using Docker, all containers within a pod share the same mounted
/etc/resolv.conf file. Therefore, modifying the
/etc/resolv.conf in one container achieves the goal of changing the nameserver for the entire pod.
All containers within the pod use the same resolv.conf file.
Drawback: Requires a program inside the container that can modify files, like
echo (you cannot use
sed directly because they create a new file and overwrite the source file). Also, it requires the container to run as the root user (or it will result in “Permission denied” errors).
Advantage: Simple and straightforward.
sed inside a container run as root results in an error.
Modifying the file inside a non-root user container results in a “Permission denied” error.
1.2 Modify the
ResolvConfPath of the host mounted source file corresponding to /etc/resolv.conf in the Docker container
The second method involves modifying the source file mounted on the host corresponding to
/etc/resolv.conf inside Docker containers. Similar to the first method, you cannot use
sed directly for this, but you can use the
echo command to make changes.
The original idea for this method comes from dockershim, which modifies
/etc/resolv.conf inside the sandbox.
Drawback: Requires root permissions on the host machine and is only applicable when Docker is the container runtime.
Advantage: It’s a simple method.
Modifying the file using
vim on the host doesn’t take effect inside the container.
echo command to modify the mounted source file on the host allows the changes to be reflected inside the container. This method enables the container to detect and use the modified
/etc/resolv.conf file on the host.
Even though the content of the
/etc/resolv.conf file has been updated, does the application re-read the
Different applications exhibit different behaviors:
Java JVM: By default, it only reads the file at startup and does not re-read
/etc/resolv.conf once it’s running.
As a program (any process, JVM included) has its very first DNS request it reads and caches forever the entire contents of
/etc/resolv.confby default. It never refreshes that info later, even when it encounters a total DNS failure. The program would need to have some specific system calls programmed to behave in more user-friendly manner. This SO question explains the details.
Specified in java.security to indicate the caching policy for successful name lookups from the name service.. The value is specified as integer to indicate the number of seconds to cache the successful lookup.
A value of -1 indicates “cache forever”. The default behavior is to cache forever when a security manager is installed, and to cache for an implementation specific period of time, when a security manager is not installed.
Golang: When using the net/http library in Go 1.17 for testing, Go will re-read
The behavior of other applications may vary. Some applications periodically check the DNS configuration file for updates, while others read it once at startup and keep it unchanged. To understand the behavior of a specific application, you may need to consult the documentation or configuration options for that application or perform testing to determine how it handles changes to the
There are two methods to modify the
/etc/resolv.conf file of pods: using Kubernetes’ native support with
spec.dnsPolicy, directly modifying the
/etc/resolv.conf inside containers, and altering the
resolv.conf based on Docker’s characteristics.
The first two methods are generally applicable, while the third method is specific to Docker (or containerd) as the container runtime. When modifying the
/etc/resolv.conf file of a running container, it does not necessarily trigger applications to reread its contents. Different applications behave differently, with some reading the
/etc/resolv.conf file only once at startup and not rereading it, while others may periodically reread it for updated DNS configurations.
how to graceful update the node
/etc/resolv.conf on kubernetes cluster see this articles