Hi, Forwarded zone cache was activated in this bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2006803 , I would like to know if it possible to activate the cache's serve_stale feature for the forwarded zone, serve_stale could help to improve application resilience in case of DNS resolver or DNS authoritative outage.

https://coredns.io/plugins/cache/

serve_stale, when serve_stale is set, cache always will serve an expired entry to a client if there is one available. When this happens, cache will attempt to refresh the cache entry after sending the expired cache entry to the client. The responses have a TTL of 0. DURATION is how far back to consider stale responses as fresh. The default duration is 1h.

Thanks.

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.1
• release-4.0

Contact the Test Platform or Automated Release teams for more information.

Plattform: OpenShift version 4.5.7
Hardware: 12 worker nodes, vmware vsphere, 8 CPU cores per node
Node uname (via oc debug node): Linux ocp-w-01 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Hi! We are experiencing 5 seconds timeouts on DNS and believe we are experiencing the NAT trouble described in bugzilla 1661928. The cluster-dns-operator is in use, with ClusterIP service and NAT to the rest of the network. No additional zones is defined, so the resulting CoreDNS configuration is:

.:5353 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        upstream
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        policy sequential
    }
    cache 30
    reload
}

Node /etc/resolv.conf contains two nameservers on local network, 10.150.13.2:53 and 10.150.13.3:53. The dns-pods has io-errors in the log output:

[ERROR] plugin/errors: 2 elasticsearch.openshift-logging.svc.cluster.local.ocp-prd.regsys.brreg.no. A: read udp 10.129.6.2:42578->10.150.13.2:53: i/o timeout

As I understand it, setting force_tcp in the forward-options could resolve the trouble, regardless of where the packet drop is (NAT iptables race condition or dropped packet on network).

You state in the readme:

Limited configuration of the CoreDNS Corefile or kubernetes plugin is supported.

Would you be open to allow more configuration supported? For example:

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  forwardOptions: "raw string of options here"

If the OCP cluster is distributed among many different vlans, it would be easier to run dns pods in just one part of the network. That would make network administrations a lot easier and operator more flexible.

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.16
• release-4.17

For more information, see the branching documentation.

dns pod daemonset currently cant be scheduled on tainted node for v4.2 and above. Looks fix in v4.1 does not being applied to v4.2 and above.

Test issue for the bot.

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.17
• release-4.18

For more information, see the branching documentation.

There is a problem with a long request to the dns server. It is shown in the example below. The request is sent from the pod and the problem manifests itself from the pod. I tried to solve the problem by doing forwarding, but it didn't help.

curl -w "dns_resolution: %{time_namelookup}, tcp_established: %{time_connect}, ssl_handshake_done: %{time_appconnect}, TTFB: %{time_starttransfer}, time_redirect: %{time_redirect}, time_total: %{time_total}s\n" -o /dev/null -s https://ya.ru
dns_resolution: 5.073743, tcp_established: 5.087487, ssl_handshake_done: 5.103775, TTFB: 5.166707, time_redirect: 0.000000, time_total: 5.193673s

I attach an example of forwarding.

  servers:  
   - name:
     zones:
       - *.ru
       - *.com
     forwardPlugin:
       upstreams:
         - dns1
         - dns2:5353

cluster version 4.5.0-0.okd-2020-10-15-235428

Since #56, this operator has been adding content to /etc/hosts. But that file is managed by the machine-config daemon, and when the MCD detects the altered content, it reboots the node. The node comes back up, and tries to restore the expected pods, and presumably the whole cycle repeats again. This makes it harder for external clients to connect to the Kubernetes and OpenShift API servers (openshift/origin#21612), and that shows up in failed aws-e2e CI runs as random, unrelated flakes. I'm not sure what the /etc/hosts additions are for, but can you find another approach that accomplishes the same goal? Or find some way to ask the MCD to append your content, instead of reaching around the MCD and touching the file directly?

CC @abhinavdahiya, @ironcladlou, @pravisankar

Bash processes run as pid 1 need to follow a certain set of patterns in order to avoid being unresponsive when the container runtime invokes them.

Right now, the sidecar sync container is ignoring SIGTERM and so pod deletion takes however long a single wait lasts, instead of happening near instantly. This is frustrating and delays upgrade and can lead to extended downtime on the node for DNS.

Specifically, you need to

1. Listen for SIGTERM explicitly: https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_node_group/files/sync.yaml#L59
2. Always sleep X & wait instead of just sleep X: https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_node_group/files/sync.yaml#L83
3. If you do run sub processes, be careful to wait on the right object (some examples exist, I don't have them at hand).

If CoreDNS becomes completely unavailable, as in during bootstrapping or after an outage, the operator needs to be able to bring up DNS. This means the operator itself can't rely on cluster DNS to talk to the API server.

Host networking, co-location with the apiserver, and communication with the apiserver over loopback could be the solution (see openshift/cluster-version-operator#25).

Priority classes docs:
https://docs.openshift.com/container-platform/3.11/admin_guide/scheduling/priority_preemption.html#admin-guide-priority-preemption-priority-class

Example: https://github.com/openshift/cluster-monitoring-operator/search?q=priority&unscoped_q=priority

Notes: The pre-configured system priority classes (system-node-critical and system-cluster-critical) can only be assigned to pods in kube-system or openshift-* namespaces. Most likely, core operators and their pods should be assigned system-cluster-critical. Please do not assign system-node-critical (the highest priority) unless you are really sure about it.

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.1
• release-4.0

Contact the Test Platform or Automated Release teams for more information.

How to enable logging in dns.operator/default ?
updating corefile in configmap/dns-default gets overwritten by the operator.

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.10
• release-4.9

Contact the Test Platform or Automated Release teams for more information.

This is one issue caused by 'Bug 1753059: Don't start DNS on NotReady nodes'

After the change(Bug 1753059), the DNS pods can not be started on the nodes which has taint point (for dedication deployment need).

In my case, we deployed some pods(with tolerations) we developed on the OCP environment, and taint some of the OCP nodes with taint(for example: oc adm taint node xx dedicated=myapp:NoSchedule).
The app works in OCP 4.2.0 version, but after upgrade the OCP environment to 4.2.21, we met problem to have dns pods support.
The problem is introduced by the Bug 1753059, which blocks the DNS pods deployed on the tained nodes.

And we didn't find way to make the DNS daemonset to toleration our tainted key, the added tolerations will be updated/deleted by dns-operator( cluster-dns-operator/pkg/operator/controller/controller_dns_daemonset.go)

Please help to support, either toleration all tainted key as previous(before Bug 1753059), or support customized tolerations.

-- thanks & best wishes

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.1
• release-4.0

Contact the Test Platform or Automated Release teams for more information.

Wildcard DNS queries are currently not supported by cluster dns operator. This is not implemented by upstream CoreDNS as this is not a mandatory requirement in the [dns spec] (https://github.com/kubernetes/dns/blob/master/docs/specification.md).
We may need to implement this functionality to support backward compatibility with OpenShift 3.x release.