Cylc 8 not properly matching hostnames given in global.cylc about cylc-flow HOT 4 OPEN

Finding the FQDN on HPC systems is often awkward due to complex DNS setup. Finding an approach that works for most cases has taken us some time. Cylc is using the getfqdn interface which is what Python recommends for this purpose, e.g. see this quote from the socket.getaddrinfo method docs:

To find the fully qualified domain name, use the function getfqdn(). gethostbyaddr()

-- https://docs.python.org/3/library/socket.html#socket.getaddrinfo

The behaviour of getfqdn tends to match hostname -f, however, the behaviour of this, and other, socket interfaces can vary according to OS and even Python build. I'm not sure that SO comment is correct about the inner workings of the getfqdn interface. The behaviour that you're seeing, may be specific to your Python build + DNS setup. On my platform the FQDN does not appear in /etc/hosts but getfqdn still returns the right result, so it's not quite as simple as returning the first matching result.

To understand why the same interface can give different results, take a look at the source of the interface that getfqdn calls:

https://github.com/python/cpython/blob/0704166f9a9c6028314d50b493670e7862c968b3/Modules/socketmodule.c#L5954

The large number of #ifdef statements in that code show how radically one build can differ in implementation from another.

So unfortunately, this is a Python build / OS / DNS issue rather than a Cylc bug.

To help work around these issues, I did once try to refactor the configuration to allow sites to configure the method that Cylc uses to return the FQDN to something that works with the local setup. This turned out to be a bit involved so I didn't follow through with it at the time, but this is still a possibility if needed.

If you haven't tried it already, you might want to look at Cylc's address based approach which is generally more robust to DNS configuration:

https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc[scheduler][host%20self-identification]method

Let us know if address mode works for you. If not, we'll take a look into configuration options.

from cylc-flow.