mspnp / aks-baseline-regulated Goto Github PK

Section 11 of "docs/deploy/11-gitops.md" requests Nginx and MI resource validation, which appears to fail due to the absence of the entire namespace. Unfortunately, after going over the steps, I can't see where it would have been deployed. Has anyone recently gone through these steps completely? 

kubectl describe AzureIdentity,AzureIdentityBinding -n ingress-nginx

Do you have plan to implement the infrastructure via Terraform in the near future? Terraform is also a popular IaC tool which should be under your radar.
Thank you in advance.

When I attempt to deploy Microsoft.ContainerService/managedClusters in cluster-stamp.json file, I keep getting below error:
{
"status": "Failed",
"error": {
"code": "ProvisioningControlPlaneError",
"message": "AKS encountered an internal error while attempting the requested Updating operation. AKS will continuously retry the requested operation until successful or a retry timeout is hit. Check back to see if the operation requires resubmission. Correlation ID: c6db0728-d434-46c4-b7eb-a7da3a34fdd6, Operation ID: 62887834-93d8-4efd-a2a9-0fb5f72445fe, Timestamp: 2021-10-14T18:34:04Z."
}
}
What does it mean?

Hello Team,

I just noticed, in clusterstamp.v2.json if I go to the saved searches section at line 888, in query "traefik_entrypoint_requests_total" filter is used.

Please check it's still relevant to this implementation? as we are using NGINX ingress in this.

Regards,

Cluster workload utility, Open Service Mesh (see: osm-system), the link leads to 404 and do not find OSM manifest in repository. Can we use Istio to replace OSM?

Hi, we are trying to implement regulated environment based on this guidelines and we are failing to get through our Pod.

We successfully deployed almost all resources (deploying on australiaeast for some reason we cant deploy agent pools) in azure and deployed cluster manifests (cluster-baseline-settings, ingress-nginx, kube-system). Then we changed from example in cluster manifests namespace creation yamls to our naming but all resources stayed the same. Same goes with workload yamls where, for now, we deployed single aspnetcore application hosted on kestrel and exposing http on port 8080.

Our application gateway backend health is showing 502.

And nginx ingress controller is filled with logs about ssl handshaking reset by peer. I am guessing from appgw probe.

The certificate used in ingress is full chain certificate and valid because we created VM inside network and made request on our ingress in browser and got valid certificate. On appgw we used root certificate from that chain as trusted root. Appgw is not accepting anything else.

If we port forward in this scenario from VM inside network to our pod service, we get application.

Also when we try to curl ingress from jumpbox we get this response.

curl -vvLk https://10.210.4.4/dashboard
* Trying 10.210.4.4...
* TCP_NODELAY set
* Connected to 10.210.4.4 (10.210.4.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.aks-ingress.*******.com
* start date: Nov 16 00:00:00 2021 GMT
* expire date: Feb 14 23:59:59 2022 GMT
* issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x558574169600)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /dashboard HTTP/2
> Host: 10.210.4.4
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 404
< date: Tue, 30 Nov 2021 19:42:00 GMT
< content-type: text/html
< content-length: 146
< strict-transport-security: max-age=15724800; includeSubDomains
<
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection #0 to host 10.210.4.4 left intact

Here you can see how aks-ingress certificate is valid when making request from VM inside network.

We already implemented baseline and it is working properly with traefik as ingress controller. From my standpoint the only difference that would cause this is OMS?

Here is ingress script and I can post other but its same as in workload example only with client naming.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend
  namespace: ****s
  labels:
    app.kubernetes.io/name: ****s
    app.kubernetes.io/component: frontent
    pci-scope: in-scope
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_name "frontend-sa.****s.cluster.local";
    nginx.ingress.kubernetes.io/proxy-ssl-secret: kube-system/osm-ca-bundle
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - ****s-00.aks-ingress.***up.com
  rules:
    - host: ****s-00.aks-ingress.***up.com
      http:
        paths:
          - path: /.*
            pathType: Prefix
            backend:
              service:
                name: frontend
                port:
                  number: 8080

Any idea what would be wrong with our setup?

When I attempt to deploy cluster-stamp.json, always get error on EnsureClusterIdentityHasRbacToSelfManagedResources deployment. I noticed there are actually 6 roleAssignments in this block. 4 of them regarding MI RBAC role assignments to subnet scope get the same error msg:
{
"status": "Failed",
"error": {
"code": "RoleAssignmentUpdateNotPermitted",
"message": "Tenant ID, application ID, principal ID, and scope are not allowed to be updated."
}
}

From my understanding, RBAC can only be applied on 1) Resource Groups 2) Resource (vnet is a resource and not subnets, subnets are the outcome of a resource) 3) Subscription
Subnets are not resources and you will not be able to restrict which subnets can and cannot be used, via RBAC/Custom roles feature.
Feel to correct me if I am wrong.

Dear @ckittel

Good Day!

I have tried the NGINX ingress controller but it's not working with policy readOnlyRootFilesystem = true.
I have checked
kubernetes/ingress-nginx#6256
it seems like an issue with NGINX and pods are not getting provisioned with we keep readOnlyRootFilesystem = true.

Regards,