razee-io / encryptedresource Goto Github PK

While testing the use of EncryptedResources in our CD workflow we encountered issues where the resource was not applied to the cluster. After investigating, we discovered that the encryptedresource-controller was in a CrashLoopBackOff with OOMKilled listed as last status.

None of these issues were encountered during local development on a kind cluster.

The underlying resource(s) being applied are mostly cluster image pull secrets in various namespaces. Currently, we have it structured that each Secret is wrapped by a single EncryptedResource. With that structure we noticed the crashes occurring if we tried to apply more than one. Even with one, we encountered the infrequent crash.

We then tried to include multiple resources in the resources: array field of the EncryptedResource CR. The same behavior was observed, i.e. a single secret seemed to be ok, but more than that and the controller crashed with an OOMKilled status.

We increased the memory requests/limits to 200Mi/500Mi, respectively, and have not observed any other crashes even with about a dozen or so separate EncryptedResources being managed.

Expected behavior:

• For the controller to run in a stable fashion when multiple EncryptedResources are being managed.

Example:

Containers:
  encryptedresource-controller:
    Container ID:   containerd://c6d34612f2fc2414405a5d73767589c6934845d2cb1468cb03b05d038a0f55f8
    Image:          us.icr.io/armada-master/encryptedresource:0.2.3_da66c79
    Image ID:       us.icr.io/armada-master/encryptedresource@sha256:be3096e8f17b0385b36014990f74417f2cefe46ed4d9d63a3d8382ffe8d32289
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 12 Aug 2021 11:02:44 -0400
      Finished:     Thu, 12 Aug 2021 11:03:09 -0400
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 12 Aug 2021 11:01:48 -0400
      Finished:     Thu, 12 Aug 2021 11:02:28 -0400
    Ready:          False
    Restart Count:  2
    Limits:
      cpu:     100m
      memory:  200Mi
    Requests:
      cpu:     40m
      memory:  75Mi

Disregard "Restart Count: 2" as this was after a restart of the controller deployment. We began this process witnessing ~167 restarts over 11h.

All seems well with the following (arbitrary) req/limits:

 resources:
          limits:
            cpu: 100m
            memory: 500Mi
          requests:
            cpu: 40m
            memory: 200Mi