Fluentd is a log shipper that needs to be deployed for shipping logs to elasticsearch.

A helm chart will be needed for easy installation of logging-operator.

Hi,

Thanks for the operator! I was wondering if it is possible to install a single node ES for testing?
I know this is possible for generic installation if you change the discovery.type: single-node in the elasticsearch.yml but I couldnt find this option in the operator.

Thanks!

Openshift 4.11+ does not allow anyuid.

create Pod elastic-master-0 in StatefulSet elastic-master failed error: pods "elastic-master-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, spec.initContainers[0].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000760000, 1000769999], spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000760000, 1000769999], provider restricted: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "velero-privileged": Forbidden: not usable by user or serviceaccount]

1.688129737572256e+09	INFO	controller_logging_operator	Statefulset get action failed	{"Namespace": "openshift-operators", "Name": "elasticsearch-master", "Resource Type": "StatefulSet"}
1.688129737599003e+09	INFO	KubeAPIWarningLogger	would violate PodSecurity "restricted:latest": privileged (container "sysctl-init" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "sysctl-init", "elastic" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "sysctl-init", "elastic" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "sysctl-init", "elastic" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "sysctl-init" must not set runAsUser=0), seccompProfile (pod or containers "sysctl-init", "elastic" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
1.6881297376007473e+09	INFO	controller_logging_operator	Statefulset successfully created	{"Namespace": "openshift-operators", "Name": "elasticsearch-master", "Resource Type": "StatefulSet"}

Fluentd is only parsing /////////////////////////////////////////////////////// to Elasticsearch. When checking the Fluentd logs, we observe that only /////////////////////////////////////////////////////// is being logged, and the same data is reflected in Elasticsearch.

error seen when starting ES pods in statefulSet:

{"type": "server", "timestamp": "2021-12-21T20:34:18,952Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "production", "node.name": "prod-cluster-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [prod-cluster-master-1, prod-cluster-master-2, prod-cluster-master-3] to bootstrap a cluster: have discovered [{prod-cluster-master-0}{xDH5es11QgKEX0v1nLnVYA}{c391A-elStqo6dXv-6rUgg}{192.168.157.8}{192.168.157.8:9300}{lmr}{ml.machine_memory=7569141760, xpack.installed=true, transform.node=false, ml.max_open_jobs=20}, {prod-cluster-master-1}{75WHjYWFTs6zsefUuSazYQ}{C5SGMQdnQOiSuNy9c5sh5A}{192.168.91.238}{192.168.91.238:9300}{lmr}{ml.machine_memory=7569141760, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}]; discovery will continue using [] from hosts providers and [{prod-cluster-master-0}{xDH5es11QgKEX0v1nLnVYA}{c391A-elStqo6dXv-6rUgg}{192.168.157.8}{192.168.157.8:9300}{lmr}{ml.machine_memory=7569141760, xpack.installed=true, transform.node=false, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2021-12-21T20:34:19,743Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "production", "node.name": "prod-cluster-master-0", "message": "failed to resolve host [prod-cluster-master-headless]", 
"stacktrace": ["java.net.UnknownHostException: prod-cluster-master-headless",

attempting to resolve headless service from a debug pod in the same NS:

root@sh:/# nslookup prod-cluster-master-headless
Server:		10.100.0.10
Address:	10.100.0.10#53

** server can't find prod-cluster-master-headless: NXDOMAIN

after changing service to add publishNotReadyAddresses: True

root@sh:/# nslookup prod-cluster-master-headless
Server:		10.100.0.10
Address:	10.100.0.10#53

Name:	prod-cluster-master-headless.runtime.svc.cluster.local
Address: 192.168.91.238
Name:	prod-cluster-master-headless.runtime.svc.cluster.local
Address: 192.168.157.8

deprecation of old annotation

apparently the annotation service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true' has been deprecated or isn't used by EKS coreDNS.

https://kubernetes.io/docs/reference/using-api/deprecation-guide/
since v1.22

/mnt/c/Users/Administrator/yaml/olm/logging-operator/logging-operator # helm upgrade logging-operator ./helm-charts/logging-operator/  -f ./helm-charts/logging-operator/
values.yaml --namespace logging-operator --install
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /mnt/c/Users/Administrator/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /mnt/c/Users/Administrator/.kube/config
Release "logging-operator" does not exist. Installing it now.
Error: failed to install CRD crds/elasticsearch.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

I am using documentation to install kibana but its failing. it creates pod but kibana does not starts. I logged in to pod and kibana.yaml under

`$ kubectl exec -it kibana-8469b466b9-72lk6 bash -n ot-operators
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-4.4$ cat config/kibana.yml

** THIS IS AN AUTO-GENERATED FILE **

Default Kibana configuration for docker target

server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true`

where as elasticsearch server should elasticsearch-master plus from log I can see kibana cant connect to elasticsearch server

Hi
Is it possible to put http proxy configuration?
current parameters looks could not configure HTTP proxy.

For ease of deployment of elasticsearch a separate helm chart is needed.

after applying operator and sample config files following errors occure:
error validating "logging.opstreelabs.in_elasticsearches.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec): unknown field "additionalPrinterColumns" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "subresources" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "validation" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "version" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "logging.opstreelabs.in_fluentds.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec): unknown field "additionalPrinterColumns" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "subresources" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "validation" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "version" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "logging.opstreelabs.in_indexlifecycles.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec): unknown field "additionalPrinterColumns" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "subresources" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "validation" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "version" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "logging.opstreelabs.in_indextemplates.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec): unknown field "additionalPrinterColumns" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "subresources" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "validation" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "version" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "logging.opstreelabs.in_kibanas.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec): unknown field "additionalPrinterColumns" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "subresources" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "validation" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec, ValidationError(CustomResourceDefinition.spec): unknown field "version" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionSpec]; if you choose to ignore these errors, turn validation off with --validate=false

Kibana deployment support is needed by using helm as a deployment tool.

Keystore support is not added inside elasticsearch yet and we have to add it.