Resolving down Swarm service from service with `dns: "127.0.0.11"` results in hundreds of errors per second in syslog about moby HOT 11 CLOSED

following manuals set dns: "127.0.0.11" to do not forward requests to the external DSN servers

I have no idea what manuals you're talking about but if that's one of our docs page we need to fix it.

The dns field in daemon.json, the docker run --dns flag, and the Compose dns field all have the same purpose: they set the DNS servers the Engine forward to when A / AAAA queries don't match any container. By putting 127.0.0.11 there, you are telling the Engine to forward to itself -- effectively creating an infinite loop of queries. That's why you see so much logs.

There are a couple of different ways to ensure DNS queries don't get forwarded to upstream resolvers:

• Using docker run --dns-option=ndots:0 ... (or equivalent Compose field) and using unqualified container names (eg. without the network name).
• Connecting your containers to internal networks only. That's probably not what you're looking for as you want your HAProxy to load balance traffic to your backend containers.

For the record, we're waiting for the ICANN report Proposed Top-Level Domain String for Private Use to decide whether we want to make the daemon an authoritative NS for the DNS zone dckr.internal. (or something similar). This would provide a better mechanism to ensure queries aren't forwarded.

I'll check if we can slightly improve our config validation to make sure we don't accept 127.0.0.11.

from moby.

Ah, right -- that's not available on Swarm. Well, in that case unfortunately you have no way to disable upstream forwarding.

from moby.

following manuals set dns: "127.0.0.11"
What is this supposed to mean? Is this set in daemon.json? Inside the container? Somewhere else?

It looks like part of a docker compose service definition, equivalent to --dns in a docker run command.

Asking because regularly seeing this same kind of log flood with no DNS set in daemon.json.

If you're seeing something similar, without configuring 127.0.0.11 as an external DNS resolver via docker compose/run/create or any other means ... please could you raise a new issue? Be sure to include examples of log lines and, ideally, a minimal way for us to reproduce the problem (or, at least, a description/examples of the configuration you're using and any ideas you have about what might trigger it).

from moby.

@bluikko I removed setting from the Swarm Stack file and I don't have log flood anymore when Swarm internal DNS can't resolve request.

You are right, the Swarm I am looking at had the same problem with DNSConfig set to 127.0.0.11. And whaddayaknow, it's also a HAProxy container!

There must be some bad advice being given in some documentation or "howto" somewhere: it's too much of a coincidence otherwise.

@elyulka I'll add my comment in the relevant HAProxy issue, hopefully this will get removed from the "howto". Oh how I despise howtos.

from moby.

It seems like @robmry has experience with this part of docker codebase

from moby.

@akerouanton Thank you so much for clarifications! I've asked Haproxy team to update blog post to don't mislead people like me.

from moby.

@akerouanton Unfortunately Swarm stack doesn't support dns_opt, i'm getting services.http Additional property dns_opt is not allowed error using next config:

  dns_opt:
    - "ndots:0"

from moby.

docker service create supports --dns-option though, right? dns_opt seems to be missing in plumbing for docker stack deploy. I guess it's worth creating a separate issue for this.

from moby.

following manuals set dns: "127.0.0.11"

What is this supposed to mean?
Is this set in daemon.json? Inside the container? Somewhere else?

Asking because regularly seeing this same kind of log flood with no DNS set in daemon.json.

from moby.

@elyulka Did you remove the upstream DNS server of 127.0.0.11 from your Docker DNS configuration and did it help anything? I suspect this may be a red herring.

please could you raise a new issue

TL;DR: I do not have any useful data except I note the two similarities to the OP: Swarm & tasks.[...] query.

I would love to open an issue but I have nothing tangible on this problem. Just the symptoms that seem to manifest randomly with average maybe 1 to 6 times a month. Probably I'll continue monitoring and try to figure out something/anything of use before opening an issue.

All I know is that in various Swarms once in a while machine(s) suddenly start to spam this, triggering journald default flood limits, but calculated to about 100 log lines per second.
This flood repeats until the machine (or perhaps just dockerd) is restarted.

The log lines are identical except of course things like port numbers, query etc.; and the A record query does have our search domain appended. No DNS settings whatsoever are defined in daemon.json.
It would be very helpful if the log event would include client IP address or any identifying information. Next time I'll packet capture on lo and try to find which container is the client.

Interesting that our "flooding query" also starts with tasks. same as in the OP.
It seems like this is some well-known label used by Docker and as such it is strange why it would ever be forwarded to external resolvers. I say "seems" because I could not find a reference to this DNS in docs.docker.com but I did see a closed issue 5854

This issue did give some great ideas but since Swarm we also cannot use dns_opt. Inside at least some containers libc is already configured with similar option options ndots:0.
But in any case demanding all DNS queries to be unqualified doesn't seem feasible unless one has a very tight control over content of all containers.

from moby.

@bluikko I removed setting from the Swarm Stack file and I don't have log flood anymore when Swarm internal DNS can't resolve request.

I also noticed that inside container there is options ndots:0 in /etc/resolv.conf present by default without explicit configuration.

from moby.