[Question] Frakti with Rook.io , error about frakti HOT 14 CLOSED

@DarkBlaez @resouer Tried again with a new allinone cluster, rook.io works properly after changing the frakti's default memory to 256MB. The full step is here: https://gist.github.com/feiskyer/b023d703355753878053cedf1fa4b538.

from frakti.

I've resolved this by editing rook-operator.yaml and adding a securityContext of privileged to the container section. Am I correct that in doing so this then has Frakti run with dockerd instead of hyperd?

from frakti.

@DarkBlaez Yep, priviledged containers are running in docker. For Rook.io problem, will figure out what's wrong.

from frakti.

For Rook.io I was able to get the rook-operator to run by setting a securityContext to privileged: true. Now I'm just trying to solve getting the next step, rook-cluster to run

from frakti.

rook-operator failed because of OOM, after increasing memory, it could start normally:

# cat rook-operator.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-operator
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - serviceaccounts
  - secrets
  - pods
  - services
  - nodes
  - configmaps
  - events
  - persistentvolumes
  - persistentvolumeclaims
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - delete
- apiGroups:
  - extensions
  resources:
  - thirdpartyresources
  - deployments
  - daemonsets
  - replicasets
  verbs:
  - get
  - list
  - watch
  - create
  - delete
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterroles
  - clusterrolebindings
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - delete
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - get
  - list
  - watch
  - delete
- apiGroups:
  - rook.io
  resources:
  - "*"
  verbs:
  - "*"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-operator
  namespace: default
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rook-operator
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-operator
subjects:
- kind: ServiceAccount
  name: rook-operator
  namespace: default
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: rook-operator
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: rook-operator
    spec:
      serviceAccountName: rook-operator
      containers:
      - name: rook-operator
        image: quay.io/rook/rookd:master-latest
        resources:
          limits:
            memory: 512Mi
        args: ["operator"]
        env:
        - name: ROOKD_REPO_PREFIX
          value: quay.io/rook

But there are more problems, you can't change the memory for pods mon0, mon1, mon2 provisioned by rook operator. Kubernetes doesn't support this because of many reasons.

from frakti.

Added #152 which could help with the issue. After setting default memory to 512MB, rook.io works properly.

from frakti.

How would this be manifested once the change is in to allow the default memory and cpu to be configurable. This could vary based on POD or at the container level. Is this something that would be picked up from the yaml or would this have to be configured at the service level for configuring hyperd?

Along with this does hyperd/frakti report metrics back to insure kubernetes is aware of allocations. I think under docker since the pool of resources being utilized comes from the host this is known. If each hyperd spun up grabs memory is this conveyed to Kubernetes This comes in to play with autoscaling or placement of pods, etc. Just curious

Thanks
DB

from frakti.

@DarkBlaez frakti will always pick up information from Pod level cgroup, so the resource boundary of your pod VM is consistent with Kubernetes's resource model, nothing changed.

What #152 solves is: when Pod does not have resource limit set, then this Pod is classified as BestEffort (BE) in k8s. (some Burstable Pods without any limit set also drop into this case)

And in Hyper case, BE pod is not allowed (VM should always have resource limit), that's why we need to set a default resource for BE in frakti.

Please note BE pod will be evicted immediately when system resource is low and is not recommended to run your core business workloads.

btw, that's why we recommend user to run BE pod in Docker runtime by using mixed mode of frakti.

from frakti.

Thank you for explaining. That makes sense to me now. Thanks again

from frakti.

Is this issue something that can be resolved in frakti or should this be run as privileged?

from frakti.

It's already been fixed in frakti by #152 I think

from frakti.

Ok thanks. I will pull latest code and build so I can test further.

from frakti.

Even with the change in frakti defining defaults and allowing overrides with cpu/mem, rook-cluster fails (mon0, mon1, mon2, rook-api). rook osd runs fine. I can get this to run under kubernetes without frakti integration. I just wonder what the constraints are using frakti since in most cases one would just want to launch pods and let the backend deploy, manage, in a secure manner. Thoughts on what would be causing this not to run in a hyperd/frakti cluster?

from frakti.

The issue should be already fixed, closing it.

from frakti.