default ISTIO_META_DNS_CAPTURE to true about istio HOT 15 OPEN

I think there are two different concepts here that we are debating
DNS_CAPTURE - Without this multi cluster stateful set functionality(specially TCP stateful sets like Zookeeper, HBase) doest not work. It is also needed for VM access to K8s Services. This is well tested by many companies I believe. I am +1 to enabling this by default
DNS_AUTO_ALLOCATE - I am not sure if this has been battle tested. We use it and some other companies also seem to use it (based on issues raised). We may need proper IPAM to be 100% reliable on this. So I can go either way on this

from istio.

. As we all know, the review process in Istio is pretty broken - we start with 'experiments' where the review bar is pretty low,
tell users they are safe to use in prod - and 1 year later we go back with 'experiments are used, we can no longer make changes, let's mark them as stable'. What is missing is the deep review that is needed before making a feature long-term supported. That may be ok for some things - but DNS is so critical to both reliability and security that we need to do better.

+1

from istio.

I helped a customer who was running a massive environment and they had issues with their DNS server being overloaded. This feature helped them significantly because Istio now performs the resolution.

from istio.

@howardjohn thoughts? I recall you had some objections but don't remember details.

cc @istio/wg-networking-maintainers

from istio.

something I can share, not sure anything updated

Since DNS Proxying has no cache capability, undeclared external domain names are still forwarded to upstream DNS servers each time they are requested. You may still encounter failures.

from istio.

I'm a -1 on this; the DNS subsystem in Istio is not as battle-hardened or necessarily preferable to customers' default DNS provider. It's a great feature to be able to turn on, but slurping up users' DNS traffic by default feels dangerous to me

from istio.

I'd argue this feature has been in battle for quite a long time now and from my perspective working with many of those large-scale battles, we always enable this by default. From a "battle tested" perspective I'd say this is battle tested. John I think raised some concerns in the past, but I wonder if those have been mitigated. Would be good to hear what he says.

from istio.

I am also -1 on this. @costinm has some more context on why that he can better elaborate on than I

from istio.

I am a bit more nuanced on this.

Strong -1 on the entire feature set getting promoted to stable or default.

However a subset of the feature - that is aligned with (real) DNS resolvers should be promoted and used in any environment lacking a secure DNS resolver in the platform - or in cases where platform resolver is slow or lacks ability to support 'split horizon'.

It should never be enabled in environments with a working, secure and standards compliant platform DNS resolver.

And any feature that is not aligned with DNS practices should not be used.

In particular auto-allocation of IPs in ServiceEntry or exposing ServiceEntry to DNS is a normal and expected feature of a DNS resolver - however the granularity and controls around it are off and I think we need to fix them first. The 'proof', like in k8s Gateway, would be having a second implementation backed by a real DNS resolver.

A 'real' DNS resolver is suposed to check dns-sec, use doh/dot/etc when talking to upstream, cache all records, have strict controls and audits, observability and much more. We are not in the business of implementing a competing, complete secure DNS resolver - I hope.

from istio.

I may have a slightly different view in one point. Because Istio Serviceentry has been GA, and we highly depend on local DNS to provide DNS resolve capability for SE's host , without this SE can not be said GA

from istio.

from istio.

@costinm Good input. istio DNS is not meant to provide complete capabilities of a DNS system, definitely some gaps with kube-dns.

from istio.

Not so much kube dns as all 'enterprise' or 'managed' resolvers - I don't know if kube dns is checking dns-sec signatures and last I checked the DOT support was not best or enabled - and kube dns is restricted to a cluster.

As long as we restrict the features to a point where non-istio integrations are feasible - like we did for CA and as we are doing for telemetry - not bad to have a minimal 'reference implementation'.

Tools like external dns allow use of real DNS for k8s and istio, and I think that is the ideal for a prod env.

from istio.

@ramaraochavali - I generally agree. We MUST have the base DNS capture for environments lacking a secure resolver, and we must return DNS for service for VMs or cases where VPC-wide DNS are not possible. I have no concern with those 2 features - as long as we validate that the functionality can be substituted with 'real DNS'.

We did the same for CA - we must have a minimal CA, but we also integrate with other CAs and usually make sure users are not forced to use our minimal CA by having unique features ( I know ambient is pushing this a bit ).

ServiceEntry is an entirely different story - not only the auto-allocation, but also the granularity and security model. There is a huge difference between Istio DNS acting as a secure cache/resolver - returning the same results as kube DNS or any other resolver - versus Istio injecting fake addresses with minimal security and depending on arcane 'exportTo/import' and visibility rules. Istio requires a secure DNS resolver - where fake IPs can't be injected, or our security doesn't work. We can't at the same time making it trivial for users with ServiceEntry permission to create fake DNS entries.

I am not saying we should not have a mechanism to create 'split horizon'/internal IPs for non-K8S names - but we need a very serious security review and much stricter controls. Using ServiceEntry for this purpose is also not very clear to be the ideal model.

As we all know, the review process in Istio is pretty broken - we start with 'experiments' where the review bar is pretty low,
tell users they are safe to use in prod - and 1 year later we go back with 'experiments are used, we can no longer make changes, let's mark them as stable'. What is missing is the deep review that is needed before making a feature long-term supported. That may be ok for some things - but DNS is so critical to both reliability and security that we need to do better.

from istio.

Also in K8S there's an optional feature NodeLocalDNS cache for users which have high QPS cacheable traffic (spins up a CoreDNS cache proxy on each node of the cluster).

Having ISTIO_META_DNS_CAPTURE=true by default would result that Istio users that need to also have NodeLocalDNS, they would have 2 dns cache servers in the DNS resolution chain.

from istio.