Git Product home page Git Product logo

helmchart's Introduction

Netdata Helm chart for Kubernetes deployments

Artifact HUB

Version: 3.7.89

AppVersion: v1.45.4

Based on the work of varyumin (https://github.com/varyumin/netdata).

Introduction

This chart bootstraps a Netdata deployment on a Kubernetes cluster using the Helm package manager.

By default, the chart installs:

  • A Netdata child pod on each node of a cluster, using a Daemonset
  • A Netdata k8s state monitoring pod on one node, using a Deployment. This virtual node is called netdata-k8s-state.
  • A Netdata parent pod on one node, using a Deployment. This virtual node is called netdata-parent.

Disabled by default:

  • A Netdata restarter CronJob. Its main purpose is to automatically update Netdata when using nightly releases.

The child pods and the state pod function as headless collectors that collect and forward all the metrics to the parent pod. The parent pod uses persistent volumes to store metrics and alarms, handle alarm notifications, and provide the Netdata UI to view metrics using an ingress controller.

Please validate that the settings are suitable for your cluster before using them in production.

Prerequisites

Installing the Helm chart

You can install the Helm chart via our Helm repository, or by cloning this repository.

Installing via our Helm repository (recommended)

To use Netdata's Helm repository, run the following commands:

helm repo add netdata https://netdata.github.io/helmchart/
helm install netdata netdata/netdata

See our install Netdata on Kubernetes documentation for detailed installation and configuration instructions. The remainder of this document assumes you installed the Helm chart by cloning this repository, and thus uses slightly different helm install/helm upgrade commands.

Install by cloning the repository

Clone the repository locally.

git clone https://github.com/netdata/helmchart.git netdata-helmchart

To install the chart with the release name netdata:

helm install netdata ./netdata-helmchart/charts/netdata

The command deploys ingress on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.

Tip: List all releases using helm list.

Once the Netdata deployment is up and running, read our guide, Monitor a Kubernetes (k8s) cluster with Netdata, for a breakdown of all the collectors, metrics, and charts available for health monitoring and performance troubleshooting.

Uninstalling the Chart

To uninstall/delete the my-release deployment:

 helm delete netdata

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The following table lists the configurable parameters of the netdata chart and their default values.

Parameter Description Default
kubeVersion Kubernetes version Autodetected
replicaCount Number of replicas for the parent netdata Deployment 1
imagePullSecrets An optional list of references to secrets in the same namespace to use for pulling any of the images []
image.repository Container image repo netdata/netdata
image.tag Container image tag Latest stable netdata release
image.pullPolicy Container image pull policy Always
service.type Parent service type ClusterIP
service.port Parent service port 19999
service.loadBalancerIP Static LoadBalancer IP, only to be used with service type=LoadBalancer ""
service.loadBalancerSourceRanges List of allowed IPs for LoadBalancer []
service.externalTrafficPolicy Denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints Cluster
service.healthCheckNodePort Specifies the health check node port Allocated a port from your cluster's NodePort range
service.clusterIP Specific cluster IP when service type is cluster IP. Use None for headless service Allocated an IP from your cluster's service IP range
service.annotations Additional annotations to add to the service {}
ingress.enabled Create Ingress to access the netdata web UI true
ingress.apiVersion apiVersion for the Ingress Depends on Kubernetes version
ingress.annotations Associate annotations to the Ingress kubernetes.io/ingress.class: nginx and kubernetes.io/tls-acme: "true"
ingress.path URL path for the ingress. If changed, a proxy server needs to be configured in front of netdata to translate path from a custom one to a / /
ingress.pathType pathType for your ingress contrller. Default value is correct for nginx. If you use yor own ingress controller, check the correct value Prefix
ingress.hosts URL hostnames for the ingress (they need to resolve to the external IP of the ingress controller) netdata.k8s.local
ingress.spec Spec section for ingress object. Everything there will be included into the object on deplyoment {}
ingress.spec.ingressClassName Ingress class declaration for Kubernetes version 1.19+. Annotation ingress.class should be removed if this type of declaration is used nginx
rbac.create if true, create & use RBAC resources true
rbac.pspEnabled Specifies whether a PodSecurityPolicy should be created. true
serviceAccount.create if true, create a service account true
serviceAccount.name The name of the service account to use. If not set and create is true, a name is generated using the fullname template. netdata
clusterrole.name Name of the cluster role linked with the service account netdata
APIKEY The key shared between the parent and the child netdata for streaming 11111111-2222-3333-4444-555555555555
restarter.enabled Install CronJob to update Netdata Pods false
restarter.schedule The schedule in Cron format 00 06 * * *
restarter.image.repository Container image repo bitnami/kubectl
restarter.image.tag Container image tag 1.25
restarter.image.pullPolicy Container image pull policy Always
restarter.image.restartPolicy Container restart policy Never
restarter.image.resources Container resources {}
restarter.concurrencyPolicy Specifies how to treat concurrent executions of a job Forbid
restarter.startingDeadlineSeconds Optional deadline in seconds for starting the job if it misses scheduled time for any reason 60
restarter.successfulJobsHistoryLimit The number of successful finished jobs to retain 3
restarter.failedJobsHistoryLimit The number of failed finished jobs to retain 3
parent.enabled Install parent Deployment to receive metrics from children nodes true
parent.port Parent's listen port 19999
parent.resources Resources for the parent deployment {}
parent.livenessProbe.initialDelaySeconds Number of seconds after the container has started before liveness probes are initiated 0
parent.livenessProbe.failureThreshold When a liveness probe fails, Kubernetes will try failureThreshold times before giving up. Giving up the liveness probe means restarting the container 3
parent.livenessProbe.periodSeconds How often (in seconds) to perform the liveness probe 30
parent.livenessProbe.successThreshold Minimum consecutive successes for the liveness probe to be considered successful after having failed 1
parent.livenessProbe.timeoutSeconds Number of seconds after which the liveness probe times out 1
parent.readinessProbe.initialDelaySeconds Number of seconds after the container has started before readiness probes are initiated 0
parent.readinessProbe.failureThreshold When a readiness probe fails, Kubernetes will try failureThreshold times before giving up. Giving up the readiness probe means marking the Pod Unready 3
parent.readinessProbe.periodSeconds How often (in seconds) to perform the readiness probe 30
parent.readinessProbe.successThreshold Minimum consecutive successes for the readiness probe to be considered successful after having failed 1
parent.readinessProbe.timeoutSeconds Number of seconds after which the readiness probe times out 1
parent.terminationGracePeriodSeconds Duration in seconds the pod needs to terminate gracefully 300
parent.nodeSelector Node selector for the parent deployment {}
parent.tolerations Tolerations settings for the parent deployment []
parent.affinity Affinity settings for the parent deployment {}
parent.priorityClassName Pod priority class name for the parent deployment ""
parent.database.persistence Whether the parent should use a persistent volume for the DB true
parent.database.storageclass The storage class for the persistent volume claim of the parent's database store, mounted to /var/cache/netdata the default storage class
parent.database.volumesize The storage space for the PVC of the parent database 2Gi
parent.alarms.persistence Whether the parent should use a persistent volume for the alarms log true
parent.alarms.storageclass The storage class for the persistent volume claim of the parent's alarm log, mounted to /var/lib/netdata the default storage class
parent.alarms.volumesize The storage space for the PVC of the parent alarm log 1Gi
parent.env Set environment parameters for the parent deployment {}
parent.envFrom Set environment parameters for the parent deployment from ConfigMap and/or Secrets []
parent.podLabels Additional labels to add to the parent pods {}
parent.podAnnotations Additional annotations to add to the parent pods {}
parent.dnsPolicy DNS policy for pod Default
parent.configs Manage custom parent's configs See Configuration files.
parent.claiming.enabled Enable parent claiming for netdata cloud false
parent.claiming.token Claim token ""
parent.claiming.room Comma separated list of claim rooms IDs ""
parent.extraVolumeMounts Additional volumeMounts to add to the parent pods []
parent.extraVolumes Additional volumes to add to the parent pods []
k8sState.enabled Install this Deployment to gather data fr K8s cluster yes
k8sState.port Listen port service.port (Same as parent's listen port)
k8sState.resources Compute resources required by this Deployment {}
k8sState.livenessProbe.initialDelaySeconds Number of seconds after the container has started before liveness probes are initiated 0
k8sState.livenessProbe.failureThreshold When a liveness probe fails, Kubernetes will try failureThreshold times before giving up. Giving up the liveness probe means restarting the container 3
k8sState.livenessProbe.periodSeconds How often (in seconds) to perform the liveness probe 30
k8sState.livenessProbe.successThreshold Minimum consecutive successes for the liveness probe to be considered successful after having failed 1
k8sState.livenessProbe.timeoutSeconds Number of seconds after which the liveness probe times out 1
k8sState.readinessProbe.initialDelaySeconds Number of seconds after the container has started before readiness probes are initiated 0
k8sState.readinessProbe.failureThreshold When a readiness probe fails, Kubernetes will try failureThreshold times before giving up. Giving up the readiness probe means marking the Pod Unready 3
k8sState.readinessProbe.periodSeconds How often (in seconds) to perform the readiness probe 30
k8sState.readinessProbe.successThreshold Minimum consecutive successes for the readiness probe to be considered successful after having failed 1
k8sState.readinessProbe.timeoutSeconds Number of seconds after which the readiness probe times out 1
k8sState.terminationGracePeriodSeconds Duration in seconds the pod needs to terminate gracefully 30
k8sState.terminationGracePeriodSeconds Duration in seconds the pod needs to terminate gracefully 300
k8sState.nodeSelector Node selector {}
k8sState.tolerations Tolerations settings []
k8sState.affinity Affinity settings {}
k8sState.priorityClassName Pod priority class name ""
k8sState.podLabels Additional labels {}
k8sState.podAnnotations Additional annotations {}
k8sState.podAnnotationAppArmor.enabled Whether or not to include the AppArmor security annotation true
k8sState.dnsPolicy DNS policy for pod ClusterFirstWithHostNet
k8sState.persistence.enabled Whether should use a persistent volume for /var/lib/netdata true
k8sState.persistence.storageclass The storage class for the persistent volume claim of /var/lib/netdata the default storage class
k8sState.persistence.volumesize The storage space for the PVC of /var/lib/netdata 1Gi
k8sState.env Set environment parameters {}
k8sState.envFrom Set environment parameters from ConfigMap and/or Secrets []
k8sState.configs Manage custom configs See Configuration files.
k8sState.claiming.enabled Enable claiming for netdata cloud false
k8sState.claiming.token Claim token ""
k8sState.claiming.room Comma separated list of claim rooms IDs ""
k8sState.extraVolumeMounts Additional volumeMounts to add to the k8sState pods []
k8sState.extraVolumes Additional volumes to add to the k8sState pods []
child.enabled Install child DaemonSet to gather data from nodes true
child.port Children's listen port service.port (Same as parent's listen port)
child.updateStrategy An update strategy to replace existing DaemonSet pods with new pods {}
child.resources Resources for the child DaemonSet {}
child.livenessProbe.initialDelaySeconds Number of seconds after the container has started before liveness probes are initiated 0
child.livenessProbe.failureThreshold When a liveness probe fails, Kubernetes will try failureThreshold times before giving up. Giving up the liveness probe means restarting the container 3
child.livenessProbe.periodSeconds How often (in seconds) to perform the liveness probe 30
child.livenessProbe.successThreshold Minimum consecutive successes for the liveness probe to be considered successful after having failed 1
child.livenessProbe.timeoutSeconds Number of seconds after which the liveness probe times out 1
child.readinessProbe.initialDelaySeconds Number of seconds after the container has started before readiness probes are initiated 0
child.readinessProbe.failureThreshold When a readiness probe fails, Kubernetes will try failureThreshold times before giving up. Giving up the readiness probe means marking the Pod Unready 3
child.readinessProbe.periodSeconds How often (in seconds) to perform the readiness probe 30
child.readinessProbe.successThreshold Minimum consecutive successes for the readiness probe to be considered successful after having failed 1
child.readinessProbe.timeoutSeconds Number of seconds after which the readiness probe times out 1
child.terminationGracePeriodSeconds Duration in seconds the pod needs to terminate gracefully 30
child.nodeSelector Node selector for the child daemonsets {}
child.tolerations Tolerations settings for the child daemonsets - operator: Exists with effect: NoSchedule
child.affinity Affinity settings for the child daemonsets {}
child.priorityClassName Pod priority class name for the child daemonsets ""
child.env Set environment parameters for the child daemonset {}
child.envFrom Set environment parameters for the child daemonset from ConfigMap and/or Secrets []
child.podLabels Additional labels to add to the child pods {}
child.podAnnotations Additional annotations to add to the child pods {}
child.hostNetwork Usage of host networking and ports true
child.dnsPolicy DNS policy for pod. Should be ClusterFirstWithHostNet if child.hostNetwork = true ClusterFirstWithHostNet
child.podAnnotationAppArmor.enabled Whether or not to include the AppArmor security annotation true
child.persistence.hostPath Host node directory for storing child instance data /var/lib/netdata-k8s-child
child.persistence.enabled Whether or not to persist /var/lib/netdata in the child.persistence.hostPath. true
child.podsMetadata.useKubelet Send requests to the Kubelet /pods endpoint instead of Kubernetes API server to get pod metadata false
child.podsMetadata.kubeletUrl Kubelet URL https://localhost:10250
child.configs Manage custom child's configs See Configuration files.
child.claiming.enabled Enable child claiming for netdata cloud false
child.claiming.token Claim token ""
child.claiming.room Comma separated list of claim rooms IDs ""
child.extraVolumeMounts Additional volumeMounts to add to the child pods []
child.extraVolumes Additional volumes to add to the child pods []
notifications.slack.webhook_url Slack webhook URL ""
notifications.slack.recipient Slack recipient list ""
initContainersImage.repository Init containers' image repository alpine
initContainersImage.tag Init containers' image tag latest
initContainersImage.pullPolicy Init containers' image pull policy Always
sysctlInitContainer.enabled Enable an init container to modify Kernel settings false
sysctlInitContainer.command sysctl init container command to execute []
sysctlInitContainer.resources sysctl Init container CPU/Memory resource requests/limits {}
sd.image.repository Service-discovery image repo netdata/agent-sd
sd.image.tag Service-discovery image tag Latest stable release (e.g. v0.2.2)
sd.image.pullPolicy Service-discovery image pull policy Always
sd.child.enabled Add service-discovery sidecar container to the netdata child pod definition true
sd.child.resources Child service-discovery container CPU/Memory resource requests/limits {resources: {limits: {cpu: 50m, memory: 150Mi}, requests: {cpu: 50m, memory: 100Mi}}}
sd.child.configmap.name Child service-discovery ConfigMap name netdata-child-sd-config-map
sd.child.configmap.key Child service-discovery ConfigMap key config.yml
sd.child.configmap.from.file File to use for child service-discovery configuration generation sdconfig/sd-child.yml
sd.child.configmap.from.value Value to use for child service-discovery configuration generation {}

Example to set the parameters from the command line:

$ helm install ./netdata --name my-release \
    --set notifications.slack.webhook_url=MySlackAPIURL \
    --set notifications.slack.recipient="@MyUser MyChannel"

Another example, to set a different ingress controller.

By default kubernetes.io/ingress.class set to use nginx as an ingress controller, but you can set Traefik as your ingress controller by setting ingress.annotations.

$ helm install ./netdata --name my-release \
    --set ingress.annotations=kubernetes.io/ingress.class: traefik

Alternatively to passing each variable in the command line, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install ./netdata --name my-release -f values.yaml

Tip: You can use the default values.yaml

Note:: To opt out of anonymous statistics, set the DO_NOT_TRACK environment variable to non-zero or non-empty value in parent.env / child.env configuration (e.g: DO_NOT_TRACK: 1) or uncomment the line in values.yml.

Configuration files

Parameter Description Default
parent.configs.netdata Contents of the parent's netdata.conf memory mode = dbengine
parent.configs.stream Contents of the parent's stream.conf Store child data, accept all connections, and issue alarms for child data.
parent.configs.health Contents of health_alarm_notify.conf Email disabled, a sample of the required settings for Slack notifications
parent.configs.exporting Contents of exporting.conf Disabled
k8sState.configs.netdata Contents of netdata.conf No persistent storage, no alarms
k8sState.configs.stream Contents of stream.conf Send metrics to the parent at netdata:{{ service.port }}
k8sState.configs.exporting Contents of exporting.conf Disabled
k8sState.configs.go.d Contents of go.d.conf Only k8s_state enabled
k8sState.configs.go.d-k8s_state Contents of go.d/k8s_state.conf k8s_state configuration
child.configs.netdata Contents of the child's netdata.conf No persistent storage, no alarms, no UI
child.configs.stream Contents of the child's stream.conf Send metrics to the parent at netdata:{{ service.port }}
child.configs.exporting Contents of the child's exporting.conf Disabled
child.configs.kubelet Contents of the child's go.d/k8s_kubelet.conf that drives the kubelet collector Update metrics every sec, do not retry to detect the endpoint, look for the kubelet metrics at http://127.0.0.1:10255/metrics
child.configs.kubeproxy Contents of the child's go.d/k8s_kubeproxy.conf that drives the kubeproxy collector Update metrics every sec, do not retry to detect the endpoint, look for the coredns metrics at http://127.0.0.1:10249/metrics

To deploy additional netdata user configuration files, you will need to add similar entries to either the parent.configs or the child.configs arrays. Regardless of whether you add config files that reside directly under /etc/netdata or in a subdirectory such as /etc/netdata/go.d, you can use the already provided configurations as reference. For reference, the parent.configs the array includes an example alarm that would get triggered if the python.d example module was enabled. Whenever you pass the sensitive data to your configuration like the database credential you can take an option to put it into the Kubernetes Secret by specifying storedType: secret in the selected configuration. Default all the configuration will be placed in the Kubernetes configmap.

Note that with the default configuration of this chart, the parent does the health checks and triggers alarms, but does not collect much data. As a result, the only other configuration files that might make sense to add to the parent are the alarm and alarm template definitions, under /etc/netdata/health.d.

Tip: Do pay attention to the indentation of the config file contents, as it matters for the parsing of the yaml file. Note that the first line under var: | must be indented with two more spaces relative to the preceding line:

  data: |-
    config line 1 #Need those two spaces
        config line 2 #No problem indenting more here

Persistent volumes

There are two different persistent volumes on parent node by design (not counting any Configmap/Secret mounts). Both can be used, but they don't have to be. Keep in mind that whenever persistent volumes for parent are not used, all the data for specific PV is lost in case of pod removal.

  1. database (/var/cache/netdata) - all metrics data is stored here. Performance of this volume affects query timings.
  2. alarms (/var/lib/netdata) - alarm log, if not persistent pod recreation will result in parent appearing as a new node in netdata.cloud (due to ./registry/ and ./cloud.d/ being removed).

In case of child instance it is a bit simpler. By default, hostPath: /var/lib/netdata-k8s-child is mounted on child in: /var/lib/netdata. You can disable it but this option is pretty much required in a real life scenario, as without it each pod deletion will result in new replication node for a parent.

Service discovery and supported services

Netdata's service discovery, which is installed as part of the Helm chart installation, finds what services are running on a cluster's pods, converts that into configuration files, and exports them, so they can be monitored.

Applications

Service discovery currently supports the following applications via their associated collector:

Prometheus endpoints

Service discovery supports Prometheus endpoints via the Prometheus collector.

Annotations on pods allow a fine control of the scraping process:

  • prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation excludes the pod from the scraping process.
  • prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
  • prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports.

Configure service discovery

If your cluster runs services on non-default ports or uses non-default names, you may need to configure service discovery to start collecting metrics from your services. You have to edit the default ConfigMap that is shipped with the Helmchart and deploy that to your cluster.

First, copy netdata-helmchart/sdconfig/child.yml to a new location outside the netdata-helmchart directory. The destination can be anywhere you like, but the following examples assume it resides next to the netdata-helmchart directory.

cp netdata-helmchart/sdconfig/child.yml .

Edit the new child.yml file according to your needs. See the Helm chart configuration and the file itself for details. You can then run helm install/helm upgrade with the --set-file argument to use your configured child.yml file instead of the default, changing the path if you copied it elsewhere.

helm install --set-file sd.child.configmap.from.value=./child.yml netdata ./netdata-helmchart/charts/netdata
helm upgrade --set-file sd.child.configmap.from.value=./child.yml netdata ./netdata-helmchart/charts/netdata

Now that you pushed an edited ConfigMap to your cluster, service discovery should find and set up metrics collection from your non-default service.

Custom pod labels and annotations

Occasionally, you will want to add specific labels and annotations to the parent and/or child pods. You might want to do this to tell other applications on the cluster how to treat your pods, or simply to categorize applications on your cluster. You can label and annotate the parent and child pods by using the podLabels and podAnnotations dictionaries under the parent and child objects, respectively.

For example, suppose you're installing Netdata on all your database nodes, and you'd like the child pods to be labeled with workload: database so that you're able to recognize this.

At the same time, say you've configured chaoskube to kill all pods annotated with chaoskube.io/enabled: true, and you'd like chaoskube to be enabled for the parent pod but not the childs.

You would do this by installing as:

$ helm install \
  --set child.podLabels.workload=database \
  --set 'child.podAnnotations.chaoskube\.io/enabled=false' \
  --set 'parent.podAnnotations.chaoskube\.io/enabled=true' \
  netdata ./netdata-helmchart/charts/netdata

Contributing

If you want to contribute, we are humbled!

helmchart's People

Contributors

ancairon avatar cakrit avatar charlieegan3 avatar cosmix avatar dependabot[bot] avatar dex4er avatar emiran-orange avatar etienne-napoleone avatar ferroin avatar github-actions[bot] avatar huzlak avatar ilyam8 avatar joelhans avatar kam1kaze avatar karuppiah7890 avatar knatsakis avatar ktsakalozos avatar m4itee avatar mansm avatar ncmans avatar netdatabot avatar odyslam avatar prologic avatar rbjorklin avatar rickard-von-essen avatar shimonohayon avatar srueg avatar tkatsoulas avatar varyumin avatar witalisoft avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

helmchart's Issues

kubernetes 1.16.0 compatibility

trying to deploy the helm chart with kubernetes 1.16.0, I get the following error
Error: validation failed: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2"

Changing the api from apps/v1beta2 to apps/v1 in statefulset.yaml fixes it.
Let me know if you want me to make a PR.

"alarms" persistence flag affects netdata cloud

This Github issue is synchronized with Zendesk:

Ticket ID: #20
Group: Support
Requester: [email protected]
Assignee: Zack
Issue escalated by: Christopher Akritidis

Original ticket description:

Hi,
I have trouble signing in to my netdata cloud.
I am using the master-slave functionality of netdata. But I cannot see the nodes properly and slave nodes are getting disconnected.

Would you please kindly help me how can I solve this issue?

Regards,
Pegah

Allow configuring deployment strategy

Use Cases

  • When making updates to a cluster with many netdata children, the update will take a very long time if they're updated one by one as is the default.
$ kubectl explain deployment.spec.strategy
KIND:     Deployment
VERSION:  extensions/v1beta1

RESOURCE: strategy <Object>

DESCRIPTION:
     The deployment strategy to use to replace existing pods with new ones.

     DeploymentStrategy describes how to replace existing pods with new ones.

FIELDS:
   rollingUpdate	<Object>
     Rolling update config params. Present only if DeploymentStrategyType =
     RollingUpdate.

   type	<string>
     Type of deployment. Can be "Recreate" or "RollingUpdate". Default is
     RollingUpdate.

ingress.path not working

helm install
--name=netdata
--set ingress.path=/netdata
--set parent.database.persistence=false
--set parent.alarms.persistence=false
--set service.type=LoadBalancer
--namespace netdata
./netdata-helmchart

k8s 1.16.10

logs from ingress:
"GET /netdata HTTP/1.1" 404 101 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" 520 0.001 [netdata-netdata-http] [] 10.233.65.20:19999 101 0.000 404 8484098d1dab671264b2601699df380d

Not able to install on Kubernetes Cluster

Followed the exact procedure like you mentioned. The master and slave pods are up. But There are no logs within the pods and its not creating an ingress elb as expected.

RFC: Long-term metrics storage. Move to only Statefulset?

I'd love to find a way to persist at least /var/lib/netdata/registry/netdata.public.unique.id, in order to allow long-term retention of the metrics collected from each node, regardless of pod restarts. That file contains the MACHINE_GUID and previous efforts to keep it the same have failed.

If I read kubernetes/kubernetes#78902 correctly, with daemonset we're stuck at the time with hostpath volume mounts, which requires root access and generally looks like just a hack.

From what I've read, it looks like the only proper way to do this is by converting the slave nodes from a daemonset to a statefulset. I understand there are caveats though, and it would be a significant change, so I'd appreciate feedback before moving ahead with this.

@varyumin @kam1kaze @ktsakalozos @karuppiah7890 @joshw123 @charlieegan3 @josecv can any of you provide insights?

can't create /tmp/as.log: Permission denied

master node goes into CrashLoopBackOff
Here are the logs

2019-09-08 02:06:05: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'FATAL' 'netdata:MAIN' '0023@registry/r:registry_init  /13'                    │
│ /usr/libexec/netdata/plugins.d/anonymous-statistics.sh: line 28: can't create /tmp/as.log: Permission denied                                                                      │
│ 2019-09-08 02:06:06: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...                                                                                         │
│ 2019-09-08 02:06:06: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'EXIT' 'ERROR' '-'                                                             │
│ /usr/libexec/netdata/plugins.d/anonymous-statistics.sh: line 28: can't create /tmp/as.log: Permission denied                                                                      │
│ 2019-09-08 02:06:06: netdata INFO  : MAIN : EXIT: cleaning up the database...                                                                                                     │
│ 2019-09-08 02:06:06: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...                                                                                                  │
│ 2019-09-08 02:06:06: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...  

I'm using two volumes with access mode ReadWriteOnce with rook and ceph fs
Should I have used ReadWriteMany?

One and only one netdata instance should always run on each pod

I saw a case with no netdata running on a slave (crash related to preStart apparently not creating the GUID on time) and another case with two netdata running (no idea how that beauty happened). I hope that the slave liveness/readiness probes in PR #19 will resolve these issues, but I need to run some tests to make sure it doesn't happen again.

Deployment error on Kubernetes

Error is the following:
Netdata entrypoint script starting

2020-08-04 22:49:16: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
2020-08-04 22:49:16: netdata ERROR : MAIN : Ignoring host prefix '/host': path '/host' failed to stat() (errno 2, No such file or directory)
2020-08-04 22:49:16: netdata INFO  : MAIN : SIGNAL: Enabling reaper
2020-08-04 22:49:16: netdata INFO  : MAIN : process tracking enabled.
2020-08-04 22:49:16: netdata ERROR : MAIN : LISTENER: Invalid listen port 0 given. Defaulting to 19999. (errno 22, Invalid argument)
2020-08-04 22:49:16: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
2020-08-04 22:49:16: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2020-08-04 22:49:16: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
2020-08-04 22:49:16: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
2020-08-04 22:49:16: netdata INFO  : MAIN : netdata started on pid 1.
2020-08-04 22:49:16: netdata FATAL : MAIN : Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument

2020-08-04 22:49:16: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
2020-08-04 22:49:16: netdata INFO  : MAIN : EXIT: cleaning up the database...
2020-08-04 22:49:16: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
2020-08-04 22:49:16: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...

Kubernetes version is v1.18.5,
Storageclass: Local,
here attached my storageclass and pv yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: netdata-database  #StorageClass name
provisioner: kubernetes.io/no-provisioner
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: netdata-alarms  #StorageClass name
provisioner: kubernetes.io/no-provisioner
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-database-netdata-parent
  labels:
    type: local
spec:
  storageClassName: netdata-database
  persistentVolumeReclaimPolicy: Recycle
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-alarms-netdata-parent
  labels:
    type: local
spec:
  storageClassName: netdata-alarms
  persistentVolumeReclaimPolicy: Recycle
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

logs are show quite a lot of errors

the logs show quite a lot of errors. I'm wondering if this is normal or if I setup something wrong

Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:37:52: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'apps.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:37:52 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:37:52: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'users.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 f357.fuchsia.servdiscount-customer.com k8s_netdata_netdata-slave-bkx99_default_7f4c018d-d799-4d05-ba8f-b40f22221981_0 2019-09-13 23:42:42: netdata ERROR : PLUGINSD[apps] : RRDSET: chart name 'groups.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'users.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.cpu' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.cpu_user' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.cpu_system' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.threads' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.processes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime_min' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime_avg' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.uptime_max' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.mem' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.vmem' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.swap' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.minor_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.major_faults' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.lreads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.lwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.preads' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.pwrites' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.files' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.sockets' on host 'f357.fuchsia.servdiscount-customer.com' already exists.
Sept 14 08:42:42 netdata-master-0 k8s_netdata_netdata-master-0_default_7fead036-8616-494c-b229-27d6480e582a_0 2019-09-13 23:42:42: netdata ERROR : STREAM_RECEIVER[f357.fuchsia.servdiscount-customer.com,[10.244.0.1]:55262] : RRDSET: chart name 'groups.pipes' on host 'f357.fuchsia.servdiscount-customer.com' already exists. (errno 22, Invalid argument)

I'm using the following values for the chart

replicaCount: 1

image:
  repository: netdata/netdata
  tag: v1.17.0
  pullPolicy: Always

sysctlImage:
  enabled: false
  repository: alpine
  tag: latest
  pullPolicy: Always
  command: []

service:
  type: ClusterIP
  port: 19999

ingress:
  enabled: false
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - netdata.k8s.local
#  tls:
#    - secretName: netdata-tls
#      hosts:
#        - netdata.k8s.local

rbac:
  create: true

serviceAccount:
  create: true
  name: netdata


master:
  resources: {}
    # limits:
    #  cpu: 4
    #  memory: 4096Mi
    # requests:
    #  cpu: 4
    #  memory: 4096Mi

  nodeSelector: {}

  tolerations: []

  affinity: {}

  env: {}

  podLabels: {}

  podAnnotations: {}

  database:
    persistence: true
    storageclass: rook-ceph-block
    volumesize: 2Gi

  alarms:
    persistence: true
    storageclass: rook-ceph-block
    volumesize: 100Mi

  configs:
    stream:
      enabled: true
      path: /etc/netdata/stream.conf
      data: |
        [11111111-2222-3333-4444-555555555555]
          enabled = yes
          history = 3600
          default memory mode = save
          health enabled by default = auto
          allow from = *
    netdata:
      enabled: true
      path: /etc/netdata/netdata.conf
      data: |
        [global]
          memory mode = save
          bind to = 0.0.0.0:19999
        [plugins]
          cgroups = no
          tc = no
          enable running new plugins = no
          check for new plugins every = 72000
          python.d = no
          charts.d = no
          go.d = no
          node.d = no
          apps = no
          proc = no
          idlejitter = no
          diskspace = no
    health:
      enabled: true
      path: /etc/netdata/health_alarm_notify.conf
      data: |
        SEND_EMAIL="NO"
        SEND_SLACK="YES"
        SLACK_WEBHOOK_URL=""
        DEFAULT_RECIPIENT_SLACK=""
        role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[domainadmin]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[dba]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[webmaster]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[proxyadmin]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[sitemgr]="${DEFAULT_RECIPIENT_SLACK}"
    example:
      enabled: false
      path: /etc/netdata/health.d/example.conf
      data: |
        alarm: example_alarm1
          on: example.random
        every: 2s
        warn: $random1 > (($status >= $WARNING)  ? (70) : (80))
        crit: $random1 > (($status == $CRITICAL) ? (80) : (90))
        info: random
          to: sysadmin

slave:
  resources: {}
    # limits:
    #  cpu: 4
    #  memory: 4096Mi
    # requests:
    #  cpu: 4
    #  memory: 4096Mi

  nodeSelector: {}

  tolerations:
    - operator: Exists
      effect: NoSchedule

  affinity: {}

  podLabels: {}

  podAnnotationAppArmor:
    enabled: true

  podAnnotations: {}

  configs:
    netdata:
      enabled: true
      path: /etc/netdata/netdata.conf
      data: |
        [global]
          memory mode = none
        [health]
          enabled = no
    stream:
      enabled: true
      path: /etc/netdata/stream.conf
      data: |
        [stream]
          enabled = yes
          destination = netdata:19999
          api key = 11111111-2222-3333-4444-555555555555
          timeout seconds = 60
          buffer size bytes = 1048576
          reconnect delay seconds = 5
          initial clock resync iterations = 60
    coredns:
      enabled: true
      path: /etc/netdata/go.d/coredns.conf
      data: |
        update_every: 1
        autodetection_retry: 0
        jobs:
          - url: http://127.0.0.1:9153/metrics
    kubelet:
      enabled: true
      path: /etc/netdata/go.d/k8s_kubelet.conf
      data: |
        update_every: 1
        autodetection_retry: 0
        jobs:
          - url: http://127.0.0.1:10255/metrics
    kubeproxy:
      enabled: true
      path: /etc/netdata/go.d/k8s_kubeproxy.conf
      data: |
        update_every: 1
        autodetection_retry: 0
        jobs:
          - url: http://127.0.0.1:10249/metrics

  env: {}

I'm using the latest version of the chart.

Finally, let me know if that's not the right place to post this issue.

netdata/wget is not built for ARM

I'm testing this on my k3s cluster on RPi - my slaves aren't starting as it relies on node/wget which is only built for amd64 (netdata though is built for ARM in of itself)

Is there any possibility of getting a wget image built for arm?

We're hiring!

We're looking for a couple of DevOps/SREs with k8s expertise.
Check out the job post and apply!

Apparmor annotation still not taking affect

container.apparmor.security.beta.kubernetes.io/netdata-slave: unconfined

This annotation did not take affect in my case. First off, it seems that the second annotations clause for checksum/config on line 28 below seems to stomp over the apparmor annotation.

After merging the two annotations clauses, I then got errors from K8S because the annotation reference a container name not found in the pod. (The pod is named netdata-slave but the container within that pod is named netdata.) I found it was necessary to change /netdata-slave to /netdata, after which everything worked.

In summary, the following change got things working for me:

--- a/templates/daemonset.yaml
+++ b/templates/daemonset.yaml
@@ -17,7 +17,8 @@ spec:
   template:
     metadata:
       annotations:
-        container.apparmor.security.beta.kubernetes.io/netdata-slave: unconfined
+        container.apparmor.security.beta.kubernetes.io/netdata: unconfined
+        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
       labels:
         app: {{ template "netdata.name" . }}
         release: {{ .Release.Name }}
@@ -25,8 +26,6 @@ spec:
 {{- with .Values.slave.podLabels }}
 {{ toYaml . | trim | indent 8 }}
 {{- end }}
-      annotations:
-        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
 {{- with .Values.slave.podAnnotations }}
 {{ toYaml . | trim | indent 8 }}
 {{- end }}

netdata with efs provisioner

I know that others had this issue:

│ netdata 2019-12-08 07:03:36: netdata ERROR : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json (errno 2, No such file or directory)
│ netdata 2019-12-08 07:03:36: netdata FATAL : MAIN :Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument

but I can't solve it, trying to use efs-provisioner, PVCs are created fine so I'm not sure the storageclass is the issue.

my helm release file:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: netdata
  namespace: netdata
  annotations:
    fluxcd.io/ignore: "false"
    fluxcd.io/automated: "false"
spec:
  releaseName: netdata
  chart:
    git: https://github.com/netdata/helmchart.git
    path: "."
    ref: master
  values:
    [snip]
    master:
      database:
        persistence: true
        storageclass: efs
        volumesize: 2Gi
      alarms:
        persistence: true
        storageclass: efs
        volumesize: 2Gi

netdata master: survive node failure

Netdata master runs as Statefulset.

https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/#statefulset-considerations

StatefulSet ensures that, at any time, there is at most one Pod with a given identity running in a cluster.

It means that if the node with Netdata master is not available (network problems, power problems, etc) statefulset woudlnt be rescheduled on the other node of the cluster. Which bascially means no monitoring.

Related issues:

Suggested way to handle such kind of situation is manually delete the node from the cluster to get rescheduling statefulsets possible.


So, what is the best way to handle node with netdata master failure?

  • Do not schedule netdata master on the worker node?
  • Install netdata master outside of the cluster?

@knatsakis @ncmans

Init container cannot connect to Kubernetes API server

Chart version: version: 1.1.14.
Kubernetes version 1.15(AWS EKS)

netdata-slave-* containers cannot start due to an error in init-nodeuid container.

>> kubectl logs netdata-slave-xxxxx init-nodeuid
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer

It's related to this command:

wget -q -T 5 --no-check-certificate --header "${HEADER}" -O - "${URL}"

I've checked old busybox container(1.28) with the same result.

Mb, it'd better switch to alpine based image with GNU wget?

e.g.

...
        - name: init-nodeuid
          image: alpine:3.11
          volumeMounts:
            - name: nodeuid
              mountPath: "/nodeuid"
          env:
            - name: MY_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          command:
            - "/bin/sh"
          args:
            - "-c"
            - '
            apk add --no-cache wget;
            TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token);
...

No way to add custom annotations and labels to pods

It would be cool to be able to annotate/label the netdata pods with custom annotations and labels. One thing that comes to mind, for example, is using annotations to make a prometheus server scrape netdata.

One way this could be done is to have master.podLabels, master.podAnnotations, slave.podLabels and slave.podAnnotations available in values.yaml. This would be consistent with how I've seen some other charts (e.g. the postgresql chart) do it.

If this is of interest, I'd be happy to submit a PR implementing it 🙂

Improved NOTES.txt

The instructions we give after installation can probably be smarter. I didn't know almost anything about k8s when I made changes on top of what @varyumin had written and I'm still not aware of the various set-ups people may have out there. If anyone has a better example to provide, or help improve this, I'd appreciate it

Stateful set requires serviceName to work

Trying to deploy I am getting the following error:

$ kubectl apply -f statefulset.yaml

error: error validating "statefulset.yaml": error validating data: ValidationError(StatefulSet.spec): missing required field "serviceName" in io.k8s.api.apps.v1beta2.StatefulSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

According to the spec this field is missing from the helm chart template.

cnnot open the file /var/lib/netdata/health.silencers.json

With netdata 1.17 I get a new issue on master

2019-09-09 19:09:44: netdata ERROR : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json (errno 2, No such file or directory)                                      │
│ 2019-09-09 19:09:44: netdata FATAL : MAIN :Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument                                                              │
│                                                                                                                                                                                   │
│ 2019-09-09 19:09:44: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'FATAL' 'netdata:MAIN' '0023@registry/r:registry_init  /13'                    │
│ 2019-09-09 19:09:45: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...                                                                                         │
│ 2019-09-09 19:09:45: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'EXIT' 'ERROR' '-'                                                             │
│ 2019-09-09 19:09:45: netdata INFO  : MAIN : EXIT: cleaning up the database...     

Here is my values.yml (I have just updated the image tag, and the storage class for the volumes)

replicaCount: 1

image:
  repository: netdata/netdata
  tag: v1.17.0
  pullPolicy: Always

sysctlImage:
  enabled: false
  repository: alpine
  tag: latest
  pullPolicy: Always
  command: []

service:
  type: ClusterIP
  port: 19999

ingress:
  enabled: false
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - netdata.k8s.local
#  tls:
#    - secretName: netdata-tls
#      hosts:
#        - netdata.k8s.local

rbac:
  create: true

serviceAccount:
  create: true
  name: netdata


master:
  resources: {}
    # limits:
    #  cpu: 4
    #  memory: 4096Mi
    # requests:
    #  cpu: 4
    #  memory: 4096Mi

  nodeSelector: {}

  tolerations: []

  affinity: {}

  env: {}

  podLabels: {}

  podAnnotations: {}

  database:
    persistence: true
    storageclass: "netdata-database"
    volumesize: 2Gi

  alarms:
    persistence: true
    storageclass: "netdata-alarms"
    volumesize: 100Mi

  configs:
    stream:
      enabled: true
      path: /etc/netdata/stream.conf
      data: |
        [11111111-2222-3333-4444-555555555555]
          enabled = yes
          history = 3600
          default memory mode = save
          health enabled by default = auto
          allow from = *
    netdata:
      enabled: true
      path: /etc/netdata/netdata.conf
      data: |
        [global]
          memory mode = save
          bind to = 0.0.0.0:19999
        [plugins]
          cgroups = no
          tc = no
          enable running new plugins = no
          check for new plugins every = 72000
          python.d = no
          charts.d = no
          go.d = no
          node.d = no
          apps = no
          proc = no
          idlejitter = no
          diskspace = no
    health:
      enabled: true
      path: /etc/netdata/health_alarm_notify.conf
      data: |
        SEND_EMAIL="NO"
        SEND_SLACK="YES"
        SLACK_WEBHOOK_URL=""
        DEFAULT_RECIPIENT_SLACK=""
        role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[domainadmin]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[dba]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[webmaster]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[proxyadmin]="${DEFAULT_RECIPIENT_SLACK}"
        role_recipients_slack[sitemgr]="${DEFAULT_RECIPIENT_SLACK}"
    example:
      enabled: false
      path: /etc/netdata/health.d/example.conf
      data: |
        alarm: example_alarm1
          on: example.random
        every: 2s
        warn: $random1 > (($status >= $WARNING)  ? (70) : (80))
        crit: $random1 > (($status == $CRITICAL) ? (80) : (90))
        info: random
          to: sysadmin

slave:
  resources: {}
    # limits:
    #  cpu: 4
    #  memory: 4096Mi
    # requests:
    #  cpu: 4
    #  memory: 4096Mi

  nodeSelector: {}

  tolerations:
    - operator: Exists
      effect: NoSchedule

  affinity: {}

  podLabels: {}

  podAnnotationAppArmor:
    enabled: true

  podAnnotations: {}

  configs:
    netdata:
      enabled: true
      path: /etc/netdata/netdata.conf
      data: |
        [global]
          memory mode = none
        [health]
          enabled = no
    stream:
      enabled: true
      path: /etc/netdata/stream.conf
      data: |
        [stream]
          enabled = yes
          destination = netdata:19999
          api key = 11111111-2222-3333-4444-555555555555
          timeout seconds = 60
          buffer size bytes = 1048576
          reconnect delay seconds = 5
          initial clock resync iterations = 60
    coredns:
      enabled: true
      path: /etc/netdata/go.d/coredns.conf
      data: |
        update_every: 1
        autodetection_retry: 0
        jobs:
          - url: http://127.0.0.1:9153/metrics
    kubelet:
      enabled: true
      path: /etc/netdata/go.d/k8s_kubelet.conf
      data: |
        update_every: 1
        autodetection_retry: 0
        jobs:
          - url: http://127.0.0.1:10255/metrics
    kubeproxy:
      enabled: true
      path: /etc/netdata/go.d/k8s_kubeproxy.conf
      data: |
        update_every: 1
        autodetection_retry: 0
        jobs:
          - url: http://127.0.0.1:10249/metrics

  env: {}

I'm using rook and cephfs for the volumes, let me know if you want the details.

Clarify requirement for "alarms" PVC

The "alarms" directory has a lot more than alarms now (claiming information, machine guid and we'll be adding more). It's becoming really necessary to have this directory survive pod restarts.

We basically have three options here:

  • Provide more options for persistence, that distinguish between the various files stored under /var/lib/netdata. A claim.d directory is getting added there as well. I'm not fond of this option, because it means we'll need to always add new options and require new volumes every time we add a new file or directory there.
  • Change the docs and the config files so that it's clear it's not just alarms that supposedly use the persistent storage for /var/lib/netdata but also key things like the machine guid, the claiming information and the health management API key. We strongly recommend that users persist this volume and only disable it if they don't want to use netdata cloud.
  • Just demand at least one persistent volume, to be used for /var/lib/netdata, removing completely the option to disable it.

Would like to hear your views.

netdata-master CrashLoopBackOff

Hi, we are tring to deploy this helm chart, but we cant make that the netdata-master be up.

helm install --namespace netdata netdata ./netdata  \
    --set APIKEY=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX \
    --set master.database.storageclass=nfs-client \
    --set master.database.volumesize=5Gi \
    --set master.alarms.storageclass=nfs-client \
    --set master.alarms.volumesize=1Gi

And we edited, in values.yaml, the "ingress.hosts" and "service.type" this, to NodePort. because we was having errors if we set this on the command.

And created .helmignore with:

.git
.github

Looking the container logs we get that:

Netdata entrypoint script starting
2020-04-24 12:27:25: netdata ERROR : MAIN : Ignoring host prefix '/host': path '/host' failed to stat() (errno 2, No such file or directory)
2020-04-24 12:27:25: netdata INFO  : MAIN : SIGNAL: Enabling reaper
2020-04-24 12:27:25: netdata INFO  : MAIN : process tracking enabled.
2020-04-24 12:27:25: netdata ERROR : MAIN : LISTENER: Invalid listen port 0 given. Defaulting to 19999. (errno 22, Invalid argument)
2020-04-24 12:27:25: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
2020-04-24 12:27:25: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2020-04-24 12:27:25: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
2020-04-24 12:27:25: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
2020-04-24 12:27:25: netdata INFO  : MAIN : netdata started on pid 1.
2020-04-24 12:27:25: netdata INFO  : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_NAME=Alpine Linux
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID=alpine
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=3.9.5
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_DETECTION=/etc/os-release
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_HOST_OS_NAME=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_HOST_OS_ID=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_HOST_OS_ID_LIKE=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION_ID=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_HOST_OS_DETECTION=unknown
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-1127.el7.x86_64
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=hypervisor
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=/proc/cpuinfo
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=docker
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=4
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_FREQ=0
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_DETECTION=lscpu
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_RAM=8181825536
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=123715584
2020-04-24 12:27:26: netdata INFO  : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
2020-04-24 12:27:26: netdata INFO  : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json, so Netdata will work with the default health configuration.
2020-04-24 12:27:26: netdata FATAL : MAIN :Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument

2020-04-24 12:27:26: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'FATAL' 'netdata:MAIN' '0023@registry/r:registry_init  /13'
2020-04-24 12:27:28: netdata ERROR : MAIN : child pid 76 exited with code 7.
2020-04-24 12:27:28: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
2020-04-24 12:27:28: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'EXIT' 'ERROR' '-'
2020-04-24 12:27:31: netdata ERROR : MAIN : child pid 87 exited with code 7.
2020-04-24 12:27:31: netdata INFO  : MAIN : EXIT: cleaning up the database...
2020-04-24 12:27:31: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
2020-04-24 12:27:31: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...

Looks like the problem is creating /var/lib/netdata/registry directory but we don't know if this is because any that we modified.

Thanks.

Configmap size

Hello,

When I am trying to install netdata with helm, I get this message

helm install netdata ./netdata Error: create: failed to create: Secret "sh.helm.release.v1.netdata.v1" is invalid: data: Too long: must have at most 1048576 characters

It looks configmap is bigger than the 1mb configmap allowed size

helm version version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

++

Cannot create directory '/var/lib/netdata/registry' - Maybe a permission problem?

When using this chart, the netdata-parent-69f84c876b-9pdst pod fails.
P.S. Using minikube.

Temporary (dirty) solution

Look at the PersistentVolume that this pod (deployment) uses. Then do sudo chmod -R 777 /path/to/that/volume. Then everything works. However, obviously we should never chmod in production env!

Logs

Netdata entrypoint script starting
2020-08-05 08:18:14: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
2020-08-05 08:18:14: netdata ERROR : MAIN : Ignoring host prefix '/host': path '/host' failed to stat() (errno 2, No such file or directory)
2020-08-05 08:18:14: netdata INFO  : MAIN : SIGNAL: Enabling reaper
2020-08-05 08:18:14: netdata INFO  : MAIN : process tracking enabled.
2020-08-05 08:18:14: netdata ERROR : MAIN : LISTENER: Invalid listen port 0 given. Defaulting to 19999. (errno 22, Invalid argument)
2020-08-05 08:18:14: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
2020-08-05 08:18:14: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2020-08-05 08:18:14: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
2020-08-05 08:18:14: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
2020-08-05 08:18:14: netdata INFO  : MAIN : netdata started on pid 1.
2020-08-05 08:18:14: netdata FATAL : MAIN : Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument
2020-08-05 08:18:14: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
2020-08-05 08:18:14: netdata INFO  : MAIN : EXIT: cleaning up the database...
2020-08-05 08:18:14: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
2020-08-05 08:18:14: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...

kubectl describe

Name:         netdata-parent-69f84c876b-9pdst
Namespace:    default
Priority:     0
Node:         minikube/172.17.0.2
Start Time:   Wed, 05 Aug 2020 16:10:10 +0800
Labels:       app=netdata
              pod-template-hash=69f84c876b
              release=netdata
              role=parent
Annotations:  checksum/config: 094d5c7af68d3e6e34ac86b75f3769089726a09e3bc30cb40755a88f4ebcaa22
Status:       Running
IP:           172.18.0.25
IPs:
  IP:           172.18.0.25
Controlled By:  ReplicaSet/netdata-parent-69f84c876b
Containers:
  netdata:
    Container ID:   docker://8d64835fe907ba9be34ff07eefe74190dfb94c91940c62f278d14e1eacdb37a6
    Image:          netdata/netdata:v1.23.2
    Image ID:       docker-pullable://netdata/netdata@sha256:fc9fb61fb58f5428d8b9dee6ad5639bdfbb7639aec83e7c47b45d1b03dc36786
    Port:           19999/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 05 Aug 2020 16:18:14 +0800
      Finished:     Wed, 05 Aug 2020 16:18:14 +0800
    Ready:          False
    Restart Count:  5
    Liveness:       http-get http://:http/api/v1/info delay=0s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:http/api/v1/info delay=0s timeout=1s period=30s #success=1 #failure=3
    Environment:
      MY_POD_NAME:       netdata-parent-69f84c876b-9pdst (v1:metadata.name)
      MY_POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /etc/netdata/health_alarm_notify.conf from config (rw,path="health")
      /etc/netdata/netdata.conf from config (rw,path="netdata")
      /etc/netdata/stream.conf from config (rw,path="stream")
      /var/cache/netdata from database (rw)
      /var/lib/netdata from alarms (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from netdata-token-wjdbg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      netdata-conf-parent
    Optional:  false
  database:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  netdata-parent-database
    ReadOnly:   false
  alarms:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  netdata-parent-alarms
    ReadOnly:   false
  netdata-token-wjdbg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  netdata-token-wjdbg
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  <unknown>               default-scheduler  running "VolumeBinding" filter plugin for pod "netdata-parent-69f84c876b-9pdst": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>               default-scheduler  running "VolumeBinding" filter plugin for pod "netdata-parent-69f84c876b-9pdst": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         <unknown>               default-scheduler  Successfully assigned default/netdata-parent-69f84c876b-9pdst to minikube
  Normal   Pulling           3m17s (x4 over 8m39s)   kubelet, minikube  Pulling image "netdata/netdata:v1.23.2"
  Normal   Pulled            3m12s (x4 over 4m29s)   kubelet, minikube  Successfully pulled image "netdata/netdata:v1.23.2"
  Normal   Created           3m12s (x4 over 4m28s)   kubelet, minikube  Created container netdata
  Normal   Started           3m11s (x4 over 4m28s)   kubelet, minikube  Started container netdata
  Warning  BackOff           2m34s (x10 over 4m12s)  kubelet, minikube  Back-off restarting failed container

Streaming issue, cannot resolve host netdata

We have re-installed k8s, and installed netdata with helm chart (v21) again. I have also set the persistent flags for alarm and database to true. But we still cannot access the slaves immediately. There are only 2 nodes: "netdata-master-0 & master" So we still have streaming problem for slaves. And this is the log I receive from all workers:

netdata ERROR : STREAM_SENDER[worker-2] : Cannot resolve host 'netdata', port '19999': Try again (errno 22, Invalid argument)

This discussion started here: netdata/netdata#8847

Child port exposure doesn't work as intended

According to current HELM chart, Netdata childs are exposed over the {{ .Values.service.port }} value:

From https://github.com/netdata/helmchart/blob/master/templates/daemonset.yaml#L118

ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP

However, that service port is intended to be used by the parent only (according to https://github.com/netdata/helmchart#configuration). If this value is changed, the container exported port will be different than the port netdata is actually bind to, as the port binding as affected by the ${NETDATA_PORT} environment variable, configured by the run.sh inside the container image:

https://github.com/netdata/netdata/blob/master/packaging/docker/Dockerfile#L109

ENTRYPOINT ["/usr/sbin/run.sh"]

https://github.com/netdata/netdata/blob/master/packaging/docker/run.sh#L22

exec /usr/sbin/netdata -u "${DOCKER_USR}" -D -s /host -p "${NETDATA_PORT}" -W set web "web files group" root -W set web "web files owner" root "$@"

As this environment variable is not passed to the child container as part of the HELM chart, so netdata will try to bind to the hardcoded 19999 port all the time.

Suggestions:

  • Create a separate value in the values.yaml for child port exposure (as the current value is intended to be used by the parent only)
  • Include the NETDATA_PORT variable as part of the HELM chart

resources requests defaults

I see both netdata master and slave have no resources requests defaults. I think it is wrong and we need to provide default values.

Requests needed by scheduler and it is good practice to not have empty resources.

Btw GKE has webhook pod-ready.config.common-webhooks.networking.gke.io and it (silently for you) sets resources.requests.cpu to 100m if it is not set.


Do you have something in defense of no resources defaults? @varyumin @knatsakis @ncmans

Make slaves run in memory mode NONE

Now that netdata/netdata#5819 has been merged, we can have the liveness probe working even with the agents in memory mode None. We'll need to wait until we have an official release that contains the change in that PR (probably netdata v1.14), but creating this to keep in the TODOs.

TODOS

  • Update the default repo and release as soon as v1.13 is out (#5)
  • Starting the master takes a while and upgrading brings it down. Check if and how we can upgrade more gracefully.
  • Package and share properly, e.g. as shown here
  • See if and what needs to change so we can monitor applications on other pods (e.g. mysql).
  • Update documentation with instructions on how to modify the configmap from the helm command line. (#8)
  • Clean up master plugins, leave the bare minimum ones (#10)

DNS errors with HostNetwork and ClusterFirstWithHostNet settings

Hi All @cakrit @ilyam8 @zack-shoylev @thiagoftsm @vlvkobal

Here is what we found to debug connectivity and streaming problem:
The Helm chart installs netdata with dnsPolicy: ClusterFirstWithHostNet, which causes DNS errors, making slave - master streaming haphazard (even without encrypting streaming, or influxdb backend). Setting it to ClusterFirst solves the problem, but we are afraid that we lose some of the metrics.

Originally posted by @Peggy4444 in netdata/netdata#8847 (comment)

Default storage mode should be dbengine

We still have a couple of bugs to fix with the dbengine, but it should become the default memory mode very soon. We now use save in k8s, dbengine everywhere else.

Slave pods should use persistent identifiers per node

I just noticed in /var/cache/netdata that we lose the history of each slave node when we do a helm upgrade. This certainly has to do with the way the slaves identify themselves to the master and we need to fix it.

upstream_response_time chart in netdata helm chart

Hi,
I want to enable upstream_response_time in netdata using helm chart (k8s). I have enabled go.d plugin in values.yaml.
Then using ./edit-config go.d.conf on the netdata_master_0 pod, I have explicitly enabled weblog module web_log: yes. How can I restart netdata now to see the new charts?

I have also installed Nginx with helm chart.
Is there any other configuration I need to do in order to get the chart in the UI?

Doubt Regarding the Deployment.

Hi all,
I have an issue trying to do this Netdata Deployment. Please if anyone can help me, I really appreciated.

I run helm install --name my-release ./netdata. However, the netdata-master pod stays with pending status.

In the describe pods command, I can see this message.
"pod has unbound immediate PersistentVolumeClaims"

I access the Netdata dashboard from the slaves, but the dashboard from the master I can't access.

Any idea what I have to do?

Thanks. André

RFC: Should we move this to https://github.com/helm/charts?

From what I know...

Pros:

  • Get feedback from the helm/charts maintainers to make improvements
  • More exposure & possibly more contributors

Cons:

  • Lose control of the process. We'll be depending on the helm/charts maintainers for merges
  • Any issues on this helm chart will be "lost" amongst irrelevant issues other charts there and on helm itself.

What do you think?

Deployment error on Kubernetes

Error is as follows

Netdata entrypoint script starting
2019-10-31 09:32:00: netdata ERROR : MAIN : Ignoring host prefix '/host': path '/host' failed to stat() (errno 2, No such file or directory)
2019-10-31 09:32:00: netdata INFO : MAIN : SIGNAL: Enabling reaper
2019-10-31 09:32:00: netdata INFO : MAIN : process tracking enabled.
2019-10-31 09:32:00: netdata ERROR : MAIN : LISTENER: Invalid listen port 0 given. Defaulting to 19999. (errno 22, Invalid argument)
2019-10-31 09:32:00: netdata INFO : MAIN : resources control: allowed file descriptors: soft = 65536, max = 65536
2019-10-31 09:32:00: netdata INFO : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2019-10-31 09:32:00: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
2019-10-31 09:32:00: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
2019-10-31 09:32:00: netdata INFO : MAIN : netdata started on pid 1.
2019-10-31 09:32:00: netdata INFO : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_OS_NAME="Alpine Linux"
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_OS_ID=alpine
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_OS_ID_LIKE=unknown
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_OS_VERSION=unknown
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_OS_VERSION_ID=3.9.4
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_OS_DETECTION=/etc/os-release
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-957.el7.x86_64
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=hypervisor
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=/proc/cpuinfo
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER=docker
2019-10-31 09:32:00: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=dockerenv
2019-10-31 09:32:00: netdata ERROR : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json (errno 2, No such file or directory)
2019-10-31 09:32:00: netdata FATAL : MAIN :Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument

2019-10-31 09:32:00: netdata INFO : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'FATAL' 'netdata:MAIN' '0023@registry/r:registry_init /13'
2019-10-31 09:32:00: netdata INFO : MAIN : EXIT: netdata prepares to exit with code 1...
2019-10-31 09:32:00: netdata INFO : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'EXIT' 'ERROR' '-'
2019-10-31 09:32:01: netdata INFO : MAIN : EXIT: cleaning up the database...
2019-10-31 09:32:01: netdata INFO : MAIN : Cleaning up database [0 hosts(s)]...
2019-10-31 09:32:01: netdata INFO : MAIN : EXIT: all done - netdata is now exiting - bye bye...

Kubernetes version is v1.14.3
Storage USES ceph

CoreDNS URL

I believe the CoreDNS URL should be set to

- url: http://kube-dns.kube-system.svc.cluster.local:9153/metrics

It's currently set to localhost in the default configmap but I think the service URL should be used instead. At least in my kubeadm setup, port 9153 isn't exposed locally but rather via the service URL.

Only possible downside is that every netdata slave will be querying the same metrics from CoreDNS. I could see concern about large clusters with lots of nodes causing issues with this kind of setup.

Saying that, maybe it's something better handled by the master?

env field at wrong position in DaemonSet and StatefulSet

Using the latest version v2.14 of helm, which has manifest validation feature, installation will lead to this error:

$ helm install --name test . --dry-run
Error: error validating "": error validating data: ValidationError(DaemonSet.spec.template.spec): unknown field "env" in io.k8s.api.core.v1.PodSpec

on fixing daemonset, it tells about statefulset error too

$ helm install --name test . --dry-run
Error: error validating "": error validating data: ValidationError(StatefulSet.spec.template.spec): unknown field "env" in io.k8s.api.core.v1.PodSpec

I will push a fix for this when I raise a PR for #17 😄

Also, extra info - there's a bug in 2.14 which doesn't allow upgrading releases which have bad manifests (like this one, with env in wrong position) even though the latest chart has the correct manifests (which passes validations). If people are using netdata chart and are facing any issues after upgrading to helm v2.14 , then we need to ask them to use latest version of the chart (after the PR fix) and also use canary version helm and tiller, or ask them to wait for v2.14.1 and just use latest version of chart, and everything will still work, and releases will also get upgraded with proper manifests

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.