logzio / logzio-k8s Goto Github PK

License: Apache License 2.0

Ruby 15.70% Dockerfile 84.30%

logzio-k8s's Introduction

logzio-k8s

This implementation uses a Fluentd DaemonSet to collect Kubernetes logs. For Kubernetes, a DaemonSet ensures that some or all nodes run a copy of a pod. Fluentd is flexible enough and has the proper plugins to distribute logs to different third parties, such as Logz.io.

The logzio-k8s image comes pre-configured for Fluentd to gather all logs from the Kubernetes node environment and append the proper metadata to the logs.

You have two options for deployment:

Default configuration (recommended)
Custom configuration

Important notes:

K8S 1.19.3+ - If you’re running on K8S 1.19.3+ or later, be sure to use the DaemonSet that supports a containerd at runtime. It can be downloaded and customized fromlogzio-daemonset-containerd.yaml.
K8S 1.16 or earlier - If you’re running K8S 1.16 or earlier, you may need to manually change the API version in your DaemonSet to apiVersion: rbac.authorization.k8s.io/v1beta1. The API versions of ClusterRole and ClusterRoleBinding are found in logzio-daemonset-rbac.yaml and logzio-daemonset-containerd.yaml. If you are running K8S 1.17 or later, the DaemonSet is set to use apiVersion: rbac.authorization.k8s.io/v1 by default. No change is needed.
The latest version pulls the image from logzio/logzio-fluentd. Previous versions pulled the image from logzio/logzio-k8s.
ARM architecture is supported as of logzio/logzio-fluentd:1.0.2.

Deploy logzio-k8s with default configuration

For most environments, we recommend using the default configuration. However, you can deploy a custom configuration if your environment needs it.

To deploy logzio-k8s

1. Create a monitoring namespace

Your DaemonSet will be deployed under the namespace monitoring.

kubectl create namespace monitoring

2. Store your Logz.io credentials

Save your Logz.io shipping credentials as a Kubernetes secret.

kubectl create secret generic logzio-logs-secret \
--from-literal=logzio-log-shipping-token='<<LOG-SHIPPING-TOKEN>>' \
--from-literal=logzio-log-listener='https://<<LISTENER-HOST>>:8071' \
-n monitoring

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.
Replace <<LISTENER-HOST>> with your region's listener host (for example, listener.logz.io). For more information on finding your account's region, see Account region.

3. Deploy the DaemonSet

For an RBAC cluster:

kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/configmap.yaml

For a non-RBAC cluster:

kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset.yaml -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/configmap.yaml

For container runtime Containerd:

kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-containerd.yaml -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/configmap.yaml

For monitoring fluentd with runtime Containerd:

kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-containerd-monitoring.yaml -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/configmap-monitoring.yaml

For monitoring fluentd with runtime Docker:

kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac-monitoring.yaml -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/configmap-monitoring.yaml

4. Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don't see your logs, see log shipping troubleshooting.

Deploy logzio-k8s with custom configuration

You can customize the configuration of the Fluentd container. This is done using a ConfigMap that overwrites the default DaemonSet.

To deploy logzio-k8s

1. Create a monitoring namespace

This is the namespace where the Daemonset will be deployed.

kubectl create namespace monitoring

2. Store your Logz.io credentials

Save your Logz.io shipping credentials as a Kubernetes secret.

kubectl create secret generic logzio-logs-secret \
--from-literal=logzio-log-shipping-token='<<SHIPPING-TOKEN>>' \
--from-literal=logzio-log-listener='https://<<LISTENER-HOST>>:8071' \
-n monitoring

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.
Replace <<LISTENER-HOST>> with your region's listener host (for example, listener.logz.io). For more information on finding your account's region, see Account region.

3. Configure Fluentd

There are 3 DaemonSet options: RBAC DaemonSet, non-RBAC DaemonSet, Containerd. Download the relevant DaemonSet and open it in your text editor to edit it.

If you wish to make advanced changes in your Fluentd configuration, you can download and edit the configmap yaml file.

Environment variables

The following environment variables can be edited directly from the DaemonSet without editing the Configmap.

Parameter	Description
output_include_time	Default: `true` To append a timestamp to your logs when they're processed, `true`. Otherwise, `false`.
LOGZIO_BUFFER_TYPE	Default: `file` Specifies which plugin to use as the backend.
LOGZIO_BUFFER_PATH	Default: `/var/log/Fluentd-buffers/stackdriver.buffer` Path of the buffer.
LOGZIO_OVERFLOW_ACTION	Default: `block` Controls the behavior when the queue becomes full.
LOGZIO_CHUNK_LIMIT_SIZE	Default: `2M` Maximum size of a chunk allowed
LOGZIO_QUEUE_LIMIT_LENGTH	Default: `6` Maximum length of the output queue.
LOGZIO_FLUSH_INTERVAL	Default: `5s` Interval, in seconds, to wait before invoking the next buffer flush.
LOGZIO_RETRY_MAX_INTERVAL	Default: `30s` Maximum interval, in seconds, to wait between retries.
LOGZIO_FLUSH_THREAD_COUNT	Default: `2` Number of threads to flush the buffer.
LOGZIO_LOG_LEVEL	Default: `info` The log level for this container.
INCLUDE_NAMESPACE	Default: `""`(All namespaces) Use if you wish to send logs from specific k8s namespaces, space delimited. Should be in the following format: `kubernetes.var.log.containers._<<NAMESPACE-TO-INCLUDE>>_ kubernetes.var.log.containers._<<ANOTHER-NAMESPACE>>_`.
KUBERNETES_VERIFY_SSL	Default: `true` Enable to validate SSL certificates.
FLUENT_FILTER_KUBERNETES_URL	Default: `nil` (doesn't appear in the pre-made Daemonset) URL to the API server. Set this to retrieve further kubernetes metadata for logs from kubernetes API server. If not specified, environment variables `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` will be used if both are present which is typically true when running fluentd in a pod. Please note that this parameter does NOT appear in the pre-made environment variable list in the Daemonset. If you wish to use & set this variable, you'll have to add it to the Daemonset's environment variables.
AUDIT_LOG_FORMAT	Default: `audit` The format of your audit logs. If your audit logs are in json format, set to `audit-json`.
CRI	The CRI of the cluster. In `logzio-daemonset` & `logzio-daemonset-rbac` is set to `docker`, and in `logzio-daemonset-containerd` is set to `containerd`. The configmap uses this var to determin which includes it needs to make for the fluent.conf file, when configuration needs to be adjusted by the CRI.

If you wish to make any further changes in Fluentd's configuration, download the configmap file, open the file in your text editor and make the changes that you need.

4. Deploy the DaemonSet

For the RBAC DaemonSet:

kubectl apply -f /path/to/logzio-daemonset-rbac.yaml -f /path/to/configmap.yaml

For the non-RBAC DaemonSet:

kubectl apply -f /path/to/logzio-daemonset.yaml -f /path/to/configmap.yaml

For container runtime Containerd:

kubectl apply -f /path/to/logzio-daemonset-containerd.yaml -f /path/to/configmap.yaml

5. Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don't see your logs, see log shipping troubleshooting.

Sending logs from nodes with taints

If you want to ship logs from any of the nodes that have a taint, make sure that the taint key values are listed in your in your daemonset configuration as follows:

tolerations:
- key: 
  operator: 
  value: 
  effect:

To determine if a node uses taints as well as to display the taint keys, run:

kubectl get nodes -o json | jq ".items[]|{name:.metadata.name, taints:.spec.taints}"

Troubleshooting

See the troubleshooting document if you encounter error while running this integration.

Disabling systemd input

To suppress Fluentd system messages, set the FLUENTD_SYSTEMD_CONF environment variable to disable in your Kubernetes environment.

Enable prometheus monitoring

In order to monitor fluentd and collect input & output metrics. You can deploy logzio-daemonset-containerd-monitoring or logzio-daemonset-rbac-monitoring, and configmap-monitoring.yaml. These templates collects and exposes fluentd metrics on port 24231, /metrics endpoint. The templates contains annotations to easly ship when using promehteus shipper.

Changelog

logzio/logzio-fluentd:

1.5.3:
- Upgrade amd image to v1.16.5-debian-logzio-amd64-1.0.
- Upgrade arm image to v1.16.5-debian-logzio-arm.
1.5.2:
- Upgrade amd image to v1.16.3-debian-logzio-amd64-2.1.
1.5.1:
- Upgrade fluentd to 1.16.3.
1.5.0:
- Upgrade fluentd to 1.16.
- Upgrade gem fluent-plugin-logzio to 0.2.2:
  - Do not retry on 400 and 401. For 400 - try to fix log and resend.
  - Generate a metric (logzio_status_codes) for response codes from Logz.io.

Expand to check old versions

v1.4.0:
- Upgrade gem fluent-plugin-logzio to 0.1.0:
  - Use fluentd's retry instead of retry in code (raise exception on non-2xx response).
v1.3.1:
- Added fluent-plugin-prometheus.
- Added logzio-daemonset-containerd-monitoring, logzio-daemonset-rbac-monitoring and configmap-monitoring.yaml which exposes fluentd metrics on the pods port 24231, /metrics endpoint
v1.3.0:
- Added plugin fluent-plugin-dedot_filter.
- Updated image in daemonset templates, increased memory and cpu requirements.
- Added configuration for dedot plugin in configmp template.
v1.2.0:
- Upgrade fluentd to 1.15.
- Upgrade plugin fluent-plugin-kubernetes_metadata_filter to 3.1.2(#80).
v1.1.1:
- Upgrade plugin fluent-plugin-kubernetes_metadata_filter to 2.11.1.
- upgrade plugin fluent-plugin-logzio to 0.0.22.
v1.1.0:
- Upgrade base image to v1.14.
- Upgrade fluent-plugin-kubernetes_metadata_filter to 2.10.
v1.0.2:
- The docker image is now available also for ARM architecture.
v1.0.1:
- Upgrade base image to 'fluent/fluentd-kubernetes-daemonset:v1.13-debian-logzio-amd64-1'.
v1.0.0:
- Fluentd configuration will be pulled from configmap.yaml.
- Allow changing audit logs format via env var AUDIT_LOG_FORMAT.
- Update API version for RBAC Daemonsets.

logzio/logzio-k8s:

This docker image is deprecated. Please use the logzio/logzio-fluentd image instead.

v1.1.6
- Allow changing of SSL configurations.
v1.1.5
- Bumped Fluentd version to v.1.11.5 (thanks @jeroenzeegers).
- Fixed docker image: changed workdir & removed wrong gem path (thanks @pete911).
- Configured Fluentd to exclude its own logs.
- Allow sending logs from specific k8s namespaces.
v1.1.4
- Add fluent-plugin-kubernetes_metadata_filter.
v1.1.3
- Support containerd.
v1.1.2
- Fix token display issue.
v1.1.1
- Upgrade fluentd base image to v.1.10.4
v1.1.0
- Update deprecated conifg
v1.0.9
- Update base image
- Update libjemalloc package
v1.0.8
- Update deprecated APIs
v1.0.7
- Update dependencies
v1.0.6
- Use Kubernets secrets for Shipping Token and Listener URL.
- Fix log level

logzio-k8s's People

Contributors

Stargazers

Watchers

logzio-k8s's Issues

Buffer Queue Limit Size

Cluster:
AWS EKS

Description:
Fluentd pods unable to ship to logzio because of BufferQueueLimitError
related kubernetes/kubernetes#19405

[warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers*

Workaround:
Because we've deployed fluentd as a daemonset, we restart the pod experiencing this issue.

JSON in message field is no parsed

Kibana just shows the message fields as a JSON blob instead of pulling out each of the fields. Think this is an issue related to this.

See fluent/fluentd#2021

daemon.extensions wont create

The DaemonSet "fluentd-logzio" is invalid

I get the following error when trying to run the following command for my eks cluster:

$ kubectl create secret generic logzio-logs-secret --from-literal=logzio-log-shipping-token='' --from-literal=logzio-log-listener='listener-eu.logz.io' -n kube-system

$ kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml

The DaemonSet "fluentd-logzio" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"k8s-app":"fluentd-logzio", "version":"v1"}: selector does not match template labels

Systemd::JournalError: No such file or directory retrying in 1s

Using the RBAC DaemonSet (https://app.logz.io/#/dashboard/data-sources/Kubernetes) for Kubernetes (1.12.3), systemd constantly spam with :

[in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s
[in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s

This seems to be related to : fluent/fluentd-kubernetes-daemonset#203
Is there a way to fix this?

ARM support

Please consider supporting ARM servers.
Me, and many others are using ARM for easy-to-use Kubernetes clusters

From running the daemonset on ARM cluster:
standard_init_linux.go:211: exec user process caused "exec format error"

rewrite_tag_filter sets message key to null with this installation

Hi guys, we are trying to use the rewrite_tag_filter with this docker image and after using it, it is setting the message key to null.

This is the bit of extra config that we are doing:

 # Splitting users logs
   <match kubernetes.var.log.containers.users**>
     @type rewrite_tag_filter
     <rule>
       key message
       pattern /access_log/
       tag access.${tag}
     </rule>
     <rule>
       key message
       pattern /.+/
       tag app.${tag}
     </rule>
   </match>
  
   # These ones are to see the output before sending the message to logzio
   <match app.kubernetes.var.log.containers.users**>
     @type stdout
   </match>

   <match access.kubernetes.var.log.containers.users**>
     @type stdout
   </match>

This produces the following output:

2021-03-11 11:20:47.710023717 +0000 access.kubernetes.var.log.containers.users-9d589d58-mnrqf_kube-namespace_users-452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a.log: {"stream":"stdout","type":"k8s","message":null,"docker":{"container_id":"452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a"},"kubernetes":{"container_name":"users","namespace_name":"kube-namespace","pod_name":"users-5785b46454-mnrqf","pod_id":"2aa0e7ce-b45c-482a-9d58-1c4cf906ebf1","host":"aks-secondary-a228a228-vmss000002","labels":{"app":"users","pod-template-hash":"5785b46454"},"master_url":"https://10.0.0.1:443/api","namespace_id":"9d9cd9bf-869a-4546-a228-71bb773523b2"}}
2021-03-11 11:20:49.359273955 +0000 access.kubernetes.var.log.containers.users-9d589d58-mnrqf_kube-namespace_users-452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a.log: {"stream":"stdout","type":"k8s","message":null,"docker":{"container_id":"452162d94859a803dcd8e6034dd516092302bfac76329123e9ecd130d031ff7a"},"kubernetes":{"container_name":"users","namespace_name":"kube-namespace","pod_name":"users-5785b46454-mnrqf","pod_id":"2aa0e7ce-b45c-482a-9d58-1c4cf906ebf1","host":"aks-secondary-a228a228-vmss000002","labels":{"app":"users","pod-template-hash":"5785b46454"},"master_url":"https://10.0.0.1:443/api","namespace_id":"9d9cd9bf-869a-4546-a228-71bb773523b2"}}

As you can see it rewrites the tag without issues, but the message key is then set to null.

If we isolate this config, it works OK and doesn't set to null the message key, the issue is with this image.

token is still visible in the output

Even after this commit - f335872
token is still visible in the output. This is most likely because record_modifier filter (replace log key with message key) is done after. The best would be to move the config from conf/system.conf to conf/fluent.conf.

daemon.extensions wont create

serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created
The DaemonSet "fluentd-logzio" is invalid: spec.template.spec.containers[0].env[1].valueFrom.secretKeyRef.key: Invalid value: "https://listener.logz.io:8071": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name', or 'KEY_NAME', or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')

I create the YAML file using kubectl and this happened. did i missed something?

json logs are not parsed

Hi there, I am trying to hook my k8s cluster to ship logs to logz using this repo. Currently, I am getting issues where the application's json logs are not parsed.

In Kibana table view: message: {"level":"info","msg":"log message","time":"2020-01-12T15:43:28Z"}

I was expecting it to be parsed recursively like the k8s ones. Can you please help me fix it?

Possibly related to something like this: fluent/fluentd-kubernetes-daemonset#328

include dedot plugin

Would you consider adding the dedot plugin to the docker image? Even if it's not used in the default configs it would be nice to have to avoid having to make a derived container.

Use case being k8s logs often have labels whose keys have dots in them which can cause problems with elasticsearch mappings. For example the app label is fairly common for internal apps, eg: app: nginx which parses as kubernetes.labels.app and logz.io will create a string mapping. But it is also very common for pods managed by kube itself (or a provider like gke/eks/aks) or pods deployed by helm to have a label like app.kubernetes.io: nginx. This will be parsed as kubernetes.labels.app.kubernetes.io and cause a mapping error because suddenly kubernetes.labels.app is an object when ES was expecting a string.

AKS issue on namespace watch

Hello,

I try to deploy your example of deployment with RBAC on my AKS cluster.
When I deploy the DaemonSet I get the following error:

2019-04-05 13:35:38 +0000 [info]: starting fluentd-1.3.3 pid=6 ruby="2.3.3"
2019-04-05 13:35:38 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-concat' version '2.3.0'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.1.6'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.19'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.1'
2019-04-05 13:35:39 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2019-04-05 13:35:39 +0000 [info]: gem 'fluentd' version '1.3.3'
2019-04-05 13:35:39 +0000 [info]: adding match pattern="fluent.**" type="null"
2019-04-05 13:35:39 +0000 [info]: adding match pattern="logzio.**" type="detect_exceptions"
2019-04-05 13:35:39 +0000 [info]: adding filter pattern="**" type="record_modifier"
2019-04-05 13:35:39 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
Unexpected error start_namespace_watch: Exception encountered setting up namespace watch from Kubernetes API v1 endpoint https://stagingaks-77b9173a.hcp.westeurope.azmk8s.io:443/api: Failed to open TCP connection to stagingaks-77b9173a.hcp.westeurope.azmk8s.io:443 (getaddrinfo: Temporary failure in name resolution)
  /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-kubernetes_metadata_filter-2.1.6/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:34:in `rescue in start_namespace_watch'
  /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-kubernetes_metadata_filter-2.1.6/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:27:in `start_namespace_watch'
  /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-kubernetes_metadata_filter-2.1.6/lib/fluent/plugin/filter_kubernetes_metadata.rb:262:in `block in configure'

However I use the ServiceAccount and the ClusterRole:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---

Do you know why it doesn't works ?

Thanks

FR: exclude/only include logs from certain k8s namespaces

Hi again, is it possible to add support to customise exclusion or inclusion of logs from certain k8s namespaces? Currently there are a lot of irrelevant logs from namespaces like kube-system and from the fluentd container with messages like:

2020-02-16 17:34:52 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead

Fluentbit support

Hi, does anyone considered using fluentbit instead of fluentd? As far as I know it shouldn't hard to migrate configs to fluent bit and lower memory footprint would be worth it.
Any thoughts or suggestions? :)

Logzio Daemonset on GKE is deleted on kube-system namespace after a few seconds

If you remove the label kubernetes.io/cluster-service: "true" solves the problem.

Related to: kubernetes/kubernetes#51376

deprecated apis in k8s config

User left this on the logz-docs repo, but looks like it belongs here.

Original issue: logzio/logz-docs#332

Describe the issue

Issue is described best here: https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

Recommended fix

Ran into this when trying to ship Kubernetes logs on the newest version. I had to make a couple small changes the logzio-daemonset.yml file to successfully ship.

Change apiVersion: extensions/v1beta1 to apiVersion: apps/v1

and add the following object to spec:

selector:
  matchLabels:
    k8s-app: fluentd-logzio
    version: v1

Container crash (CrashLoopBackOff)

I am using this example:
https://github.com/logzio/logzio-k8s/blob/master/logzio-daemonset.yaml

I have replaced my credentials in the example (token, url).

However the container crashes immediately and the pod status (for all logzio pods) is CrashLoopBackOff.

These are the events for a crashing pod:

$ kubectl describe pod fluentd-logzio-abcd123 -n kube-system 

Events:
  Type     Reason     Age                    From                  Message
  ----     ------     ----                   ----                  -------
  Normal   Scheduled  4m14s                  default-scheduler     Successfully assigned kube-system/fluentd-logzio-tnm6x to pool-1-x7q3
  Normal   Pulled     3m21s (x4 over 4m11s)  kubelet, pool-1-x7q3  Successfully pulled image "logzio/logzio-k8s:latest"
  Normal   Created    3m21s (x4 over 4m11s)  kubelet, pool-1-x7q3  Created container fluentd
  Normal   Started    3m21s (x4 over 4m11s)  kubelet, pool-1-x7q3  Started container fluentd
  Warning  BackOff    2m44s (x8 over 4m7s)   kubelet, pool-1-x7q3  Back-off restarting failed container
  Normal   Pulling    2m33s (x5 over 4m13s)  kubelet, pool-1-x7q3  Pulling image "logzio/logzio-k8s:latest"

Also there aren't any interesting logs to debug the crash:

 $ kubectl logs -l k8s-app=fluentd-logzio  -n kube-system 

2019-06-10 14:31:13 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"

This seems a bug with your image.

Testing on Kubernetes v1.14.1

Separate logging for each namespace

Hello,

I have a Kubernetes cluster with multiple applications (i.e. different websites), each one in its own namespace.

Each application should send the logs (only for that specific app / namespace) to a different Logz.io account.

Is it possible to achieve that?

Basically I would like logging limited to a specific namespace, and not for the whole cluster.

fluentd DaemonSet degrades and stops sending logs

We found that after some time the fluentd DaemonSet degrades and stops sending logs to Logz.io. This degradation may take somewhere between a couple of hours to 2 or 3 days to happen. We haven't been able to run a working DaemonSet for a whole week.

Research

We've confirmed on our monitoring systems that the problem happens as soon as the error (fluentd.output.status.num.errors.gauge) and retry (fluentd.output.status.retry.count.gauge) counters are higher than 0. The DaemonSet remains totally healthy in terms of memory and CPU (no throttling).

Environment

We've found this behavior with release 1.3.0. We've tried release 1.4.0 and the problem arises faster.

Workaround

When that happens, logs stop being sent to Logz.io and the only known workaround is to restart the affected DaemonSet.

Issue running on EKS

Installed K8s secret and RBAC deployment per instructions. Receiving the following:

2019-11-11 22:49:27 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'                                                                                                                                                                    │
│ 2019-11-11 22:49:27 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'                                                                                                                                                                               │
│ 2019-11-11 22:49:27 +0000 [info]: gem 'fluentd' version '1.7.3'                                                                                                                                                                                             │
│ 2019-11-11 22:49:27 +0000 [info]: gem 'fluentd' version '1.3.3'                                                                                                                                                                                             │
│ 2019-11-11 22:49:27 +0000 [info]: adding match pattern="fluent.**" type="null"                                                                                                                                                                              │
│ 2019-11-11 22:49:27 +0000 [info]: adding match pattern="logzio.**" type="detect_exceptions"                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding filter pattern="**" type="record_modifier"                                                                                                                                                                         │
│ 2019-11-11 22:49:27 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"                                                                                                                                                          │
│ 2019-11-11 22:49:27 +0000 [info]: adding match pattern="**" type="logzio_buffered"                                                                                                                                                                          │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: adding source type="tail"                                                                                                                                                                                                 │
│ 2019-11-11 22:49:27 +0000 [info]: #0 starting fluentd worker pid=13 ppid=8 worker=0                                                                                                                                                                         │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-proxy-7hhgl_kube-system_kube-proxy-513f19f85cc63748b835cae5184d3cf139b5d5b5603e310e86227e92fc6a2060.log                                            │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/aws-node-pkxl5_kube-system_aws-node-cacfd05dbaac792723d71894b6ea6870a6e57e4df91e2b98a605c87c301c4778.log                                                │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/calico-node-xjrph_kube-system_calico-node-e25013c69c5b50e15764b2527fa3c9acc94cca1bc439ecb6c3485b1db9c2e2f8.log                                          │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/istio-init-crd-10-b2vt4_istio-system_istio-init-crd-10-e1e4c74148e7bf7273c5c27ef0d1324b96d89ace92ce251490dacff5b71c3ca6.log                             │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/fluentd-fluentd-cloudwatch-8dcll_fluentd_copy-fluentd-config-eb685ed3077decab1df0212bc8acfb6d7aacb9c3027871e291e57adbaef0c611.log                       │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/fluentd-fluentd-cloudwatch-8dcll_fluentd_fluentd-fluentd-cloudwatch-c1ae285523705715c57f5c17d82748e9756a7cc707ebfda75c939f0be16e76ad.log                │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/istio-tracing-595796cf54-rfbzh_istio-system_jaeger-5398fca0a8818dfa9c5e1c9653b51ff262657bc592b98306b26588c656163441.log                                 │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/istio-citadel-657c84d86f-lsxb8_istio-system_citadel-da6715368cc11cb3bc9c7663baf6dd5bbb1ceeca64910b4703a1656737b644b9.log                                │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kiali-55fcfc86cc-4vpw6_istio-system_kiali-755ea22a5e7423d92df15477abb6f7226c77f8dca37cc451b0eff829b8b55f4c.log                                          │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/istio-pilot-6b65d765b5-4vnhp_istio-system_discovery-abb8669facfe3d5515b0d3dc644906fb4956181c2d180408058ed342e2149f2b.log                                │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/istio-pilot-6b65d765b5-4vnhp_istio-system_istio-proxy-d91dcc6d05a71892a0ec3d5d5d729b23e97dcd4b384236e7c024f5526c356ca4.log                              │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/istio-sidecar-injector-78949dd945-fljbb_istio-system_sidecar-injector-webhook-493756c257f6f0a8862deedb5b24d299c5161a201b15eb4ab1615294cb339de2.log      │
│ 2019-11-11 22:49:27 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/fluentd-logzio-5xp4m_kube-system_fluentd-d36556814527a9f699f476356189e190f4605b15ba63d92c820dbd01f353cbff.log                                           │
│ 2019-11-11 22:49:27 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::KubernetesMetadataFilter] uses `#filter_stream` method.                                                                                                     │
│ 2019-11-11 22:49:27 +0000 [info]: #0 fluentd worker is now running worker=0                                                                                                                                                                                 │
│ 2019-11-11 22:49:28 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::KubernetesMetadataFilter] uses `#filter_stream` method.                                                                                                     │
│ 2019-11-11 22:49:28 +0000 [warn]: #0 [out_logzio] got unrecoverable error in primary and no secondary error_class=ArgumentError error="no HTTP request path given"                                                                                          │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http/generic_request.rb:27:in `initialize'                                                                                                                                             │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http/request.rb:15:in `initialize'                                                                                                                                                     │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-logzio-0.0.20/lib/fluent/plugin/out_logzio_buffered.rb:127:in `new'                                                                                             │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-logzio-0.0.20/lib/fluent/plugin/out_logzio_buffered.rb:127:in `send_bulk'                                                                                       │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-logzio-0.0.20/lib/fluent/plugin/out_logzio_buffered.rb:81:in `block in write'                                                                                   │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-logzio-0.0.20/lib/fluent/plugin/out_logzio_buffered.rb:119:in `encode_chunk'                                                                                    │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-logzio-0.0.20/lib/fluent/plugin/out_logzio_buffered.rb:80:in `write'                                                                                            │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.7.3/lib/fluent/plugin/output.rb:1125:in `try_flush'                                                                                                                 │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.7.3/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'                                                                                                          │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.7.3/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'                                                                                                  │
│   2019-11-11 22:49:28 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.7.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'                                                                                               │
│ 2019-11-11 22:49:28 +0000 [warn]: #0 [out_logzio] bad chunk is moved to /tmp/fluent/backup/worker0/out_logzio/59719f0ba7393a82437f214edfacbb22.log

[EKS 1.21/1.22][BoundServiceAccountTokenVolume] Refresh AWS service account tokens automatically

Describe the bug
I am using logzio/logzio-fluentd : 1.0.2 on AWS EKS 1.21 and received the following email from AWS:

`Description
We have identified applications running in one or more of your Amazon EKS clusters that are not refreshing service account tokens. Applications making requests to Kubernetes API server with expired tokens will fail. You can resolve the issue by updating your application and its dependencies to use newer versions of Kubernetes client SDK that automatically refreshes the tokens.

What is the problem?

Kubernetes version 1.21 graduated BoundServiceAccountTokenVolume feature [1] to beta and enabled it by default. This feature improves security of service account tokens by requiring a one hour expiry time, over the previous default of no expiration. This means that applications that do not refetch service account tokens periodically will receive an HTTP 401 unauthorized error response on requests to Kubernetes API server with expired tokens. You can learn more about the BoundServiceAccountToken feature in EKS Kubernetes 1.21 release notes [2].

To enable a smooth migration of applications to the newer time-bound service account tokens, EKS v1.21+ extends the lifetime of service account tokens to 90 days. Applications on EKS v1.21+ clusters that make API server requests with tokens that are older than 90 days will receive an HTTP 401 unauthorized error response.

We recommend that you update your applications and its dependencies that are using stale service accounts tokens to use one of the newer Kubernetes Client SDKs that refetches tokens.

If the service account token used is close to expiry (<90 days) and you do not have sufficient time to update your client SDK versions before expiry, then you can terminate existing pods and create new ones. This results in refetching of the service account token, giving you additional time (90 days) to update your client SDKs.
`

Similar issue is also reported in fluet-bit fluent/fluent-bit#5445. But i didnt see anything in fluentd github

To Reproduce

Deploy an logzio/logzio-fluentd : 1.0.2 pod on AWS EKS 1.21
Expected behavior
I think the fluentd pod should automatically refresh its IRSA credentials.

Your Environment

Version used:
AWS EKS 1.21
logzio/logzio-fluentd : 1.0.2

ooom killed ruby2.3 process on instances where fluentd is running.

we have been facing the below errors when we run the fluentd on our cluster.. apparently getting killed with oom

on researching it looks like may have to include jmalloc as part of the ruby base image.. can you please confirm if you have any other proposed solutions instead ?

fluent/fluent-bit-kubernetes-logging#14

391524.805414] Memory cgroup out of memory: Kill process 3685518 (ruby2.3) score 1792 or sacrifice child
[468138.916915] Memory cgroup out of memory: Kill process 859933 (ruby2.3) score 1792 or sacrifice child
[647901.248286] Memory cgroup out of memory: Kill process 2442358 (ruby2.3) score 1792 or sacrifice child
[675732.117926] IPv6: ADDRCONF(NETDEV_UP): vethb0d99b50: link is not ready
[729039.765037] [371792] 0 371792 164959 39926 266 4 0 994 ruby2.3
[729039.774334] Memory cgroup out of memory: Kill process 371792 (ruby2.3) score 1775 or sacrifice child
[729039.783915] Killed process 371792 (ruby2.3) total-vm:659836kB, anon-rss:149212kB, file-rss:10492kB, shmem-rss:0kB
[729039.821620] oom_reaper: reaped process 371792 (ruby2.3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[853128.254617] Memory cgroup out of memory: Kill process 2316873 (ruby2.3) score 1792 or sacrifice child
[953790.700466] Memory cgroup out of memory: Kill process 3246659 (ruby2.3) score 1792 or sacrifice child`

Docker image?

Are you supposed to build and use your own docker image for this?

Notice that the sample yaml templates reference image logzio/logzio-k8s:1.0.0 which doesn't exist in docker hub, at least.

Or are these published somewhere else?

Provide a convenient way to install (and use Kubernetes secrets)

It would be useful to install logz.io and update its config with a simple command:

kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/logzio-daemonset.yaml

The above command is very frequent and common - e.g. it is used by the Kubernetes dashboard and by other logging services.

Currently however we cannot use that simple command directly because you store the env variables inside the daemonset. This prevents the usage of the above command and it is also unsecure.

I suggest that you provide a file that can be used directly.

I also suggest that in the new file you use these env variables:

env:
- name: LOGZIO_TOKEN
  valueFrom:
    secretKeyRef:
      name: logzio-secrets
      key: logzio_token
- name: LOGZIO_URL
  valueFrom:
    secretKeyRef:
      name: logzio-secrets
      key: logzio_url

Dockerhub environment description seems to be outdated

Guys, description on https://hub.docker.com/r/logzio/logzio-k8s/ has wrong environment variables in the description.

Is that something you could fix?

logzio / logzio-k8s Goto Github PK

logzio-k8s's Introduction

logzio-k8s

Deploy logzio-k8s with default configuration

To deploy logzio-k8s

1. Create a monitoring namespace

2. Store your Logz.io credentials

3. Deploy the DaemonSet

4. Check Logz.io for your logs

Deploy logzio-k8s with custom configuration

To deploy logzio-k8s

1. Create a monitoring namespace

2. Store your Logz.io credentials

3. Configure Fluentd

4. Deploy the DaemonSet

5. Check Logz.io for your logs

Sending logs from nodes with taints

Troubleshooting

Disabling systemd input

Enable prometheus monitoring

Changelog

logzio-k8s's People

Contributors

Stargazers

Watchers

Forkers

logzio-k8s's Issues

Describe the issue

Recommended fix

Research

Environment

Workaround

Recommend Projects

Recommend Topics

Recommend Org