Gracefully Changing the DNS Server IP for node on a Kubernetes Cluster Without Impacting Applications

DNS servers are typically stable components of infrastructure and are rarely changed. However, if the IP address of a DNS server needs to be updated, here’s how to change the DNS configuration of Kubernetes nodes.

1 Standard Procedure

To change the node’s DNS configuration, follow these steps:

1. Replace the DNS server IP address directly in the /etc/resolv.conf file on node.

2. The above step updates only the /etc/resolv.conf file on the node. It does not update the /etc/resolv.conf inside the Node Local DNS and CoreDNS pods.

3. To make the changes take effect, restart the following Kubernetes resources:

   1 2	kubectl rollout restart deployment -n kube-system coredns kubectl rollout restart daemonset -n kube-system node-local-dns

This article uses CoreDNS version 1.8.7 and Node Local DNS version 1.21.1.

2 DNS Resolution Downtime during Restart

The above method is straightforward, but there can be DNS resolution issues during restarts. When you deploy only a CoreDNS cluster and restart CoreDNS pods as described above, there is a chance that pods may fail to resolve domain names on Linux kernels below version 5.9 and when using kube-proxy in IPVS mode. For related issues, see this GitHub issue.

In deployments with both CoreDNS and Node Local DNS clusters, restarting Node Local DNS pods as described above can lead to DNS resolution problems for pods on nodes with a DNS mode of ClusterFirst (and a network mode other than hostnetwork).

Below, we describe a method to change the node’s DNS server IP address without interrupting services.

3 Gracefully Changing Node DNS Server IP

Follow these steps to gracefully change the node’s DNS server IP address:

1. Start by replacing the DNS server IP address in the /etc/resolv.conf file.
2. Then, replace the /etc/resolv.conf file in Node Local DNS and CoreDNS pods using the method described in Modifying the /etc/resolv.conf File on a Running Pod. Use this method to update the /etc/resolv.conf file in Node Local DNS and CoreDNS pods with the contents of the node’s /etc/resolv.conf file.

3.1 Replacing the /etc/resolv.conf File in node local dns

containerd runtime

1

cat /etc/resolv.conf >  $(crictl inspect -o go-template  --template='{{- range $mount := .info.runtimeSpec.mounts -}} {{- if eq $mount.destination "/etc/resolv.conf" -}} {{- $mount.source -}}   {{- end -}} {{- end -}} '  $(crictl ps |grep node-local-dns |awk '{print $1}'))

dockershim runtime

1

cat /etc/resolv.conf > $(docker inspect $(docker ps |grep k8s_node-cache_node-local-dns | awk '{print $1}') --format '{{ .ResolvConfPath }}')

3.2 Replacing the /etc/resolv.conf File in Coredns

run command at coredns pod running node

containerd runtime

1

crictl inspect -o go-template  --template='{{- range $mount := .info.runtimeSpec.mounts -}} {{- if eq $mount.destination "/etc/resolv.conf" -}} {{- $mount.source -}}   {{- end -}} {{- end -}} '  $(crictl ps |grep coredns- |awk '{print $1}') | xargs -i  bash -c "cat /etc/resolv.conf > {}"

dockershim runtime

1

docker inspect $(docker ps |grep k8s_coredns_coredns | awk '{print $1}') --format '{{ .ResolvConfPath }}' | xargs -i  bash -c "cat /etc/resolv.conf > {}"

3.3 Reloading /etc/resolv.conf

Simply replacing the file, as mentioned earlier, doesn’t guarantee that it will be reread; this depends on how the program is implemented.

Unfortunately, CoreDNS and Node Local DNS only read the /etc/resolv.conf file when they start or restart. The reload plugin only checks for changes in the Corefile, not in other files referenced by plugins, such as the zone file by the file plugin or resolv.conf by the forward plugin.

Yes - the forward plugin only reads the file (e.g. /etc/resolv.conf) when CoreDNS starts/restarts. The reload plugin only checks for changes in the Corefile, not other files referenced by plugins (e.g. the zone file by the file plugin, and resolv.conf by the forward plugin ).

Whether or not we periodically check/re-read plugin referenced files is currently left to the plugins themselves. For example, the file plugin periodically reloads its zone file, which makes sense, since a zone file is inherently dynamic. Currently, the forward plugin does not periodically check/reload its resolv.conf file - probably because its contents are generally static.

https://github.com/coredns/coredns/issues/3928#issuecomment-640586182

To make CoreDNS and Node Local DNS reload and reread /etc/resolv.conf, you need to add a line break in the ConfigMap content. This is the magical step. Just modifying the ConfigMap content triggers a reload.

When the configuration files change, the reload plugin triggers a reload. This means that simply modifying the configmaps of coredns and node-local-dns will trigger a reload. The magic here is that you only need to add a line break within the configmap content, and this line break must be within the same curly braces as the “reload” keyword.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

.:53 {
# Add a line break here
 errors
 cache 30
 reload
 loop
 bind 169.254.20.10 
 forward . __PILLAR__UPSTREAM__SERVERS__ {
 force_tcp
 }

After a while, you will see in the logs that the reload is complete:

1
2
3

[INFO] Reloading
....
[INFO] Reloading complete

4 Caveats

If a pod’s DNS mode is Default, or the network mode is HostNetwork and the DNS mode is ClusterFirst, you should also replace their /etc/resolv.conf file since they use the node’s /etc/resolv.conf. Whether the program rereads the /etc/resolv.conf file depends on its implementation.

5 Summary

Here are the steps to gracefully change the node’s DNS server IP:

1. Directly replace the DNS server IP in the /etc/resolv.conf file.
2. Use a method to modify the /etc/resolv.conf file inside CoreDNS and Node Local DNS pods, mounted on the node.
3. Add a line break in the ConfigMap content for CoreDNS and Node Local DNS to trigger a reload and reread of /etc/resolv.conf.
4. Use a method to modify the /etc/resolv.conf file in other pods (those with DNS mode as Default or network mode as HostNetwork and DNS mode).
5. Ensure that these pods reread the /etc/resolv.conf file.

5.1 Share ansible playbook

https://gist.github.com/wu0407/946354660972f81613417ce5a5904a4d