RFC: Long-term metrics storage. Move to only Statefulset? about helmchart HOT 26 CLOSED

from helmchart.

Using an init container sounds like a good idea to me, I just don't know about the config map. A shared volume could work very well though, as in this example.

The great thing about this idea is that it doesn't need changes to the netdata daemon.

from helmchart.

This field is specific to Red Hat hosts maybe it's mean like it exits but looks another. Because I have checked on Ubuntu 18.04, 16.04, Debian 9, Debian 10 and Centos 7 and I have gotten systemUID in my node manifest. I guess too. 1st try to get SystemUUID, after uid manifest.

from helmchart.

hrmm here's my hot take from my experience with k8s -- maybe others will disagree :)

imho this does not feel like a natural fit for a statefulset, nor have i seen this done with a statefulset in the past. Guaranteeing a pod running on each node while using a statefulset is really not ergonomic. You're trying to

a) use affinity rules to ensure that no two pods ever end up on the same node,

b) coerce the user into always scaling the statefulset so that it has the same number of replicas as there are nodes (a coercion that I think becomes increasingly painful as ops teams grow and the engineer on call who grew the node pool isn't up to speed with the monitoring setup -- you're bound to eventually miss a node), and

c) guarantee that, when a new node is added and the statefulset is scaled up, the pv is going to come up in a volume zone that is compatible with that of the new node. In my experience different pvc implementations will deal with this more or less gracefully. Some will always work out, others will happily bring up a pv in the wrong AZ and then leave the pod stuck in pending because it is unable to attach the volume to the node.

i'm not sure that those requirements make it actually worth it to use a statefulset for this. I think there's a few workarounds for this

1. use an init container coupled with some centralized mechanism to initialize and save the machine identifier. To give a very naive example, imagine an init container that uses an S3-compatible bucket to store a node hostname to guid mapping, fetches it on startup and places it under /var/lib/netdata/registry/netdata.public.unique.id so that the actual netdata container effectively always has the same id.

2. use an init container to deterministically generate the guid. This looks like similar to what this commit reverted, but i think the issues caused by the previous attempt had to do with it being implemented as a postStart hook instead of an init container. An init container would be able to set the permissions correctly and would run before netdata itself starts so there's no reason it should mess with the netdata process.

3. go with the statefulset, BUT implement some kind of slave-manager system that will actively listen for new hosts/pods and ensure that the replica count/distribution of pods is always acceptable. i guess under certain circumstances this slave manager would have to be willing to nuke a guid if the PV manager on the cluster screwed up.

4. bite the bullet with the hostPath. tbqh, i do not think this is that bad. netdata is being installed as a fairly low level, close to metal, monitoring solution. in my experience it is not uncommon for this sort of thing to come with special permissions. If I understand the current helm chart correctly, netdata looks to be running in god mode already as it mounts the docker socket and /proc into the pod, so what's another hostpath among friends?

personally i think i would bet for options 2 or 4 in terms of being the least intrusive to the user and least complex to implement

from helmchart.

This is awesome @josecv , I had never heard of init containers and it looks like the 2nd option would indeed be really good. The thing that bugs me about #4 is that I don't want netdata writing anything on the host. Not sure what type of permissions we have on that socket today tbh, but we definitely don't write anything and never should.

We do have to figure out how we can get a predictable UUID from the node's hostname, which is not what python's uuid.uuid3 was doing. I had missed that it gives you a random UUID. The discussion in netdata/netdata#7265 should settle that though, I'll get on it to see what people wrote there.

from helmchart.

One sec though, the 4th option could work without netdata having to write anything. If as an admin I have access to the node's filesystem, I can just put a netdata.public.unique.id with a UUID somewhere on the node, we mount that directory and allow netdata only read access to it. Right? I suppose that the init container could do the same, just look for a specific file that contains the UUID on a configurable place on the node and just put that UUID in the proper place for netdata to find it. The caveat with this approach is that it required an action from the user, but the other options have caveats as well...

from helmchart.

Hmm you're right that an administrator could easily write the uuid, then netdata just reads it. I guess that has a similar disadvantage as the statefulset though, in that it requires user intervention when a new node is added. On the other hand, I guess astute users could easily write things like cloud init scripts that generate the uuid automagically, but that will not be the same for every cluster, so it's kinda tricky to advise users on how to do this correctly.

I think on second thought though, I'd prefer the 2nd option since as you say it doesn't require writing to the node or manual user intervention. Not a crypto expert so not sure what you'd need to make the uuid deterministic but I guess if that can be surmounted the rest is easy.

from helmchart.

hmm i guess there's another option which is to leverage the existing uid on the node k8s object.

All kubernetes objects already have a UUID -- it's under metadata.uid. So if you had an init container with permissions to read from the kubernetes api, this init container could basically just kubectl get node $THE_NODE_IM_ON -o jsonpath='{.metadata.uid}' > netdata.public.unique.id. This would guarantee that it's always the same uid, as long as it's the same node.

I guess the disadvantage of this is that you'd need permissions on the cluster. Now, in terms of netdata itself you only really need read permissions. But the nature of helm is such that emitting clusterroles (which i think you'd need to do since k8s node objects are not namespaced) requires the tiller that installs this chart to be superuser on the cluster.

from helmchart.

Very interesting idea! I will check next week, I'm in a week long meetup.

from helmchart.

Finally got some time to look into this. We can definitely use .metadata.uid and our helmchart already includes the required access.

We'll need to either extend https://github.com/netdata/netdata/blob/master/daemon/get-kubernetes-labels.sh.in, or add a new script, just to retrieve the node UUID. The netdata daemon will also need to be extended.

from helmchart.

I wrote code on GOlang, I need a few time to review on best practice and after review, I will upload code to my repo a sent URL to my repo

from helmchart.

Not a specialist, just thinking out loud.

Could following work?:

• we have slave config-map
• init container gets local node uid
• and puts it into slave configmap

node-<NODE-NAME>-uid: value  # we can get it from slave using subPathExpr

• we mount that value to the slave /var/lib/netdata/registry/netdata.public.unique.id

We can make it dynamic using subPathExpr

0 ilyam:playground $  kubectl explain deployment.spec.template.spec.containers.volumeMounts.subPathExpr
KIND:     Deployment
VERSION:  apps/v1

FIELD:    subPathExpr <string>

DESCRIPTION:
     Expanded path within the volume from which the container's volume should be
     mounted. Behaves similarly to SubPath but environment variable references
     $(VAR_NAME) are expanded using the container's environment. Defaults to ""
     (volume's root). SubPathExpr and SubPath are mutually exclusive.

set MY_NODE_NAME env var

env:
  - name: MY_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

use it in subPathExpr

volumeMounts:
  - name: config
    mountPath: /var/lib/netdata/registry/netdata.public.unique.id
    subPathExpr: node-$(MY_NODE_NAME)-uid

from helmchart.

It seems you are right, i think init container that fetches local node uid is a way to go, it looks like a robust and easy way to achieve the goal. ConfigMap/shared volume is detail and both will do.

i think ill try to fix the issue 😄

from helmchart.

Using emptyDir volume is easier then configMap, busybox is enough for initContainer.

This works: a Pod with init container that fetches node uid

apiVersion: v1
kind: Pod
metadata:
  name: init-test
spec:
  serviceAccountName: node-viewwer
  initContainers:
    - name: install
      image: busybox
      env:
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
      command:
        - "/bin/sh"
      args:
        - "-c"
        - '
        TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token);
        URL="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/nodes/${MY_NODE_NAME}";
        HEADER="Authorization: Bearer ${TOKEN}";

        DATA=$(wget -q -T 5 --no-check-certificate --header "${HEADER}" -O - "${URL}");
        [ -z "${DATA}" ] && exit 1;

        UID=$(echo "${DATA}" | grep -m 1 uid | grep -o ":.*," | tr -d ": \",");
        [ -z "${UID}" ] && exit 1;

        echo "${UID}" > /work-dir/uid;
        '
      volumeMounts:
        - name: workdir
          mountPath: "/work-dir"
  containers:
    - name: busybox
      image: busybox
      command: [ "sleep" ]
      args: [ "infinity" ]
      volumeMounts:
        - name: workdir
          mountPath: /etc/node_uid
  volumes:
    - name: workdir
      emptyDir: {}

0 ilyam:playground $ kubectl exec -it init-test cat /etc/node_uid/uid
50d0f031-a82e-4f98-bb0e-f865b71390fa

from helmchart.

I dont really like having bash inside args (no syntax highlighting, checking), but it feels like it doesnt worth to create something (repo + image) for such simple task.

I could make it in such way but make it optional. What do you think @cakrit ?

from helmchart.

If this works, let's do a PR and we'll get additional feedback. I'm personally not worried about code in the configuration.

from helmchart.

Ok, it is done, i tested it on our test k8s cluster, it works 🎉

from helmchart.

Are you sure you choose the right UID? And didn't confuse UID? Because in manifest node we have 2 kinds of UID. Firs it's UID of Kubernetes, second UID of Node. Example manifest by node:

apiVersion: v1
kind: Node
metadata:
  annotations:
...
  labels:
    ...
  selfLink: /api/v1/nodes/dc-wks-ceph-c8349d69-01
  uid: cda0f4ee-896b-11e9-b816-fa163e99a8c7
spec:
  ...
status:
 ...
  images:
 ... 
  nodeInfo:
    architecture: amd64
    bootID: 4355d9dc-2849-424f-8b80-8d76376dd739
    containerRuntimeVersion: docker://18.9.5
    kernelVersion: 4.15.0-76-generic
    kubeProxyVersion: v1.14.1
    kubeletVersion: v1.14.1
    machineID: 626be7e4892b4a55a642050afff283df
    operatingSystem: linux
    osImage: Ubuntu 18.04.2 LTS
    systemUUID: 626BE7E4-892B-4A55-A642-050AFFF283DF

I think. the correct choose is systemUUID: 626BE7E4-892B-4A55-A642-050AFFF283DF

from helmchart.

The goal is to have something persistent and unique, we choose node.metadata.uid. It is generated by system and unique per object. It seems it satisfies our needs.

0 ilyam:~ $ kubectl explain node.metadata.uid
KIND:     Node
VERSION:  v1

FIELD:    uid <string>

DESCRIPTION:
     UID is the unique in time and space value for this object. It is typically
     generated by the server on successful creation of a resource and is not
     allowed to change on PUT operations. Populated by the system. Read-only.
     More info: http://kubernetes.io/docs/user-guide/identifiers#uid

Can we have problems with it?

from helmchart.

Yeap. Your script will cover 99.9% of cases. But if someone removes or reinstall node cluster. Your UID will be changed. Maybe my example unreal, it is only my opinion. Up to you )

kubectl explain node.status.nodeInfo.systemUUID
KIND:     Node
VERSION:  v1

FIELD:    systemUUID <string>

DESCRIPTION:
     SystemUUID reported by the node. For unique machine identification
     MachineID is preferred. This field is specific to Red Hat hosts
     https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/getting-system-uuid.html

from helmchart.

I am wrong, your script covers 100% of cases. Because after reinstall node your POD also redeploy and get new UID

from helmchart.

Hm, if an administrator removes the node from the cluster and then adds it back i guess node.metdata.uid will be changed. But probably i am wrong. We need to check it.

I restarted my minikube, uid and systemUUID wasnt changed.

uid: 50d0f031-a82e-4f98-bb0e-f865b71390fa
machineID: 42b23a4d42c54ec6922e64ddd2f0949f
systemUUID: ee6459e0-8cff-ff4d-ab0d-cd839a75d444

uid: 50d0f031-a82e-4f98-bb0e-f865b71390fa
machineID: 0189b558c36848c8be6f1f0b679be446
systemUUID: ee6459e0-8cff-ff4d-ab0d-cd839a75d444

from helmchart.

I see we have 2 cases:

• node reboot
• node drain => cordon => uncordon

We need uid that survives both cases if it is possible.

from helmchart.

i assume both uid and systemUUID will do 🤔

from helmchart.

I think minikube, can't cover all cases. Try kubectl delete node or kubeadm remove node and include node again to the cluster. For example with kubeadm

from helmchart.

I think minikube, can't cover all cases.

Yeah, ofc. I think if you completely remove node and then add it back uid will be changed. I think you are right systemUUID is more robust option.

I see there is This field is specific to Red Hat hosts, is there a chance there is no SytemUUID on a host? (link in the explain for the field is 404).

I guess safe solution is to try SystemUUID if it doesnt exist fallback to uid. What do you think?

from helmchart.