elastic / helm-charts Goto Github PK
View Code? Open in Web Editor NEWYou know, for Kubernetes
License: Apache License 2.0
You know, for Kubernetes
License: Apache License 2.0
% helm ls elasticsearch-data
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
elasticsearch-data 3 Sat Dec 15 21:30:24 2018 DEPLOYED elasticsearch-6.5.3-alpha1 6.5.3 default
Running helm test:
% helm test elasticsearch-data
RUNNING: elasticsearch-data-rwrbf-test
FAILED: elasticsearch-data-rwrbf-test, run `kubectl logs elasticsearch-data-rwrbf-test --namespace default` for more info
Error: 1 test(s) failed
Helm test logs:
% kubectl logs elasticsearch-data-rwrbf-test --namespace default
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (52) Empty reply from server
The curl command doesn't use https (nor configured w/ credentials and certificates)
% kubectl get pod elasticsearch-data-rwrbf-test -o yaml | grep curl
curl -XGET --fail 'redacted:9200/_cluster/health?wait_for_status=green&timeout=1s'
https enabled in the pods
% kubectl get pod redacted-0 -o yaml | grep -A1 security.http.ssl.enabled
- name: xpack.security.http.ssl.enabled
value: "true"
I have tried to use the Helm chart 6.6.0-alpha1 with the Docker image 7.0.0-SNAPSHOT, but it did not find a minimum masters number to start (2), I know it probably does not work but I report it only in case.
{"type": "server", "timestamp": "2019-03-07T15:52:32,007+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elasticsearch-master-2}{3augAIonSmya_3Nh8ejlVQ}{9dnV86RNT_OElXMEifJqGg}{10.8.8.97}{10.8.8.97:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{vOIg6cnCQSiAefXM_d0jpA}{DLZ_IGVRQW6tgE2EfidHfg}{10.8.6.34}{10.8.6.34:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [10.8.8.97:9300, 10.8.6.34:9300] from hosts providers and [{elasticsearch-master-0}{EhVP4Rl-Qv-ZnTRpyZ7lHQ}{TF2RPZB6Rw6_EFKuJfAU4g}{10.8.5.126}{10.8.5.126:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-03-07T15:52:32,314+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])" }
{"type": "server", "timestamp": "2019-03-07T15:52:32,314+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}" ,
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:549) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
This is the configuration used
clusterName: elasticsearch
nodeGroup: "master"
masterService: "elasticsearch-master"
roles:
master: "true"
ingest: "true"
data: "true"
replicas: 3
#minimumMasterNodes: 2
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.0.0-SNAPSHOT"
imagePullPolicy: "Always"
esJavaOpts: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "100m"
memory: "2Gi"
limits:
cpu: "1000m"
memory: "2Gi"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "standard"
resources:
requests:
storage: 60Gi
protocol: http
httpPort: 9200
transportPort: 9300
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: elastic-basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
path: /
hosts:
- elasticsearch.example.com
tls:
- hosts:
- elasticsearch.example.com
secretName: ingress-tls
As pointed out in #26 (comment) most of the examples are pulling secrets from our vault service. Apart from the license we use for testing everything else can be auto generated for testing purposes. For the license it is also possible to use the trial API for testing. As part of automated testing we will continue to use the real license because some rules are relaxed when running a trial (such as no enforced node to node encryption).
The end goal of this issue is to make sure that all users and contributors can run any of the examples without needing to first figure out how to create and generate their own secrets.
Hi, the elasticsearch chart in the community repo spins up client nodes as deployment. Looking at elastic's ES chart, if all 3 roles are set to false and cross cluster search is disabled through esConfig, the corresponding statefulset will compose of coordinating nodes(?)
Can you please explain why a statefulset is preferred over deployment for coordinating nodes and why does it need a persistent volume claim? Or is this chart not supposed to be used for client-only node? I looked through the examples and readme but didn't see any mention of coordinating only nodes. TIA
Hi,
your doc recommend to enable memory locking at bootstrap - this is completely missing on the helm chart.
It's a bit tricky on kubernetes but it's possible:
enable the required capabilities
securityContext:
capabilities:
add:
- IPC_LOCK
- SYS_RESOURCE
set ulimit before starting elasticsearch
command:
- /bin/sh
- -cxe
- |
ulimit -l unlimited
/usr/local/bin/docker-entrypoint.sh eswrapper
enable via environment
env:
- name: bootstrap.memory_lock
value: "true"
This colud be configurable via helm variable.
when I change the value of cluster name to "test", the headless service name is changed correctly, but the value of env "discovery.zen.ping.unicast.hosts" kept as "elasticsearch-master-headless", and i have to change it manually. here is my value.yaml:
clusterName: test
nodeGroup: master
replicas: 2
minimumMasterNodes: 2
antiAffinity: soft
esMajorVersion: 7
esJavaOpts: "-Xmx8g -Xms8g"
resources:
requests:
cpu: 1
memory: 16Gi
limits:
cpu: 1
memory: 16Gi
volumeClaimTemplate:
accessModes: ["ReadWriteOnce"]
storageClassName: es-hdd-storage
resources:
requests:
storage: "5Ti"
updateStrategy: RollingUpdate
maxUnavailable: 1
Are there plans to implement Operator https://coreos.com/operators/ for deploying and managing ES clusters?
initContainers doesn't seem to be exposed. Is there a way to install plugins with this chart?
First of all - thank you guys for the chart!
I was playing around with the multi-node example and experienced some odd behavior. Here's how I'm reproducing the issue.
After the multi example is deployed, open up the multi-data service to your local in one terminal:
$ kubectl port-forward service/multi-data 9200
Watch the call to /_cat/master
in another terminal:
$ watch -n1 time curl -s http://localhost:9200/_cat/master?v
In a third terminal, whoever the elected master is, delete them:
$ kubectl delete pod multi-master-0
The API call in the second terminal will now hang. After 30 seconds, the request will timeout and we might see the following error for a split second:
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
Soon after, the cluster recovers and the API call from the second window starts responding again. Here are the logs off another master node before and after the re-election:
[2019-02-17T01:49:03,736][INFO ][o.e.d.z.ZenDiscovery ] [multi-master-1] master_left [{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [shut_down]
[2019-02-17T01:49:03,736][WARN ][o.e.d.z.ZenDiscovery ] [multi-master-1] master left (reason = shut_down), current nodes: nodes:
{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, master
{multi-data-0}{Ndz2WGGiSz6Y1XO1tWyWRw}{nPoZagPdRr2Tq45BPAkh_g}{10.40.2.13}{10.40.2.13:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
{multi-data-2}{MmkHmP6XRTibriDLazv_1A}{iS1mfA0JQ-yURqZ4-ng2zQ}{10.40.1.8}{10.40.1.8:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
{multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
{multi-master-1}{uetiuhetRbasFRNLqI6ixg}{AwsDmF2aTZS1JesCE0HZ0A}{10.40.2.15}{10.40.2.15:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, local
{multi-data-1}{Uh9aucMmS6uLRBeG5559_w}{tlxqIDoZRjCLMfMWxU0IyQ}{10.40.0.11}{10.40.0.11:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
[2019-02-17T01:49:03,830][WARN ][o.e.t.TcpTransport ] [multi-master-1] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.40.1.14:36958}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2019-02-17T01:49:07,107][INFO ][o.e.c.s.ClusterApplierService] [multi-master-1] detected_master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, reason: apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [95]])
[2019-02-17T01:49:34,832][WARN ][o.e.c.NodeConnectionsService] [multi-master-1] failed to connect to node {multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [multi-master-0][10.40.1.14:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1576) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-6.6.0.jar:6.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-02-17T01:49:37,200][WARN ][o.e.c.s.ClusterApplierService] [multi-master-1] cluster state applier task [apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [95]])] took [30s] above the warn threshold of 30s
[2019-02-17T01:49:40,315][INFO ][o.e.c.s.ClusterApplierService] [multi-master-1] removed {{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [96]])
[2019-02-17T01:49:55,945][WARN ][o.e.t.TransportService ] [multi-master-1] Received response for a request that has timed out, sent [47893ms] ago, timed out [17870ms] ago, action [internal:discovery/zen/fd/master_ping], node [{multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], id [579]
[2019-02-17T01:50:03,784][INFO ][o.e.c.s.ClusterApplierService] [multi-master-1] added {{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{LqkyAGeTSUW2mtNmt49HIA}{10.40.1.15}{10.40.1.15:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [98]])
I figured Kubernetes might be killing the pods too abruptly, so I followed the instructions at https://www.elastic.co/guide/en/elasticsearch/reference/6.6/stopping-elasticsearch.html to stop Elasticsearch. Sure enough, if we kill the process from the elected master pod directly, the re-election will be quick!
Assuming mutli-master-2
is the new master:
$ kubectl exec multi-master-2 -- kill -SIGTERM 1
Notice how the API call from the second terminal only hangs for around 3 seconds this time!
Reading through the docs for the termination of pods (https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods), Kubernetes does in fact send a SIGTERM to the container, so I'm guessing deleting a pod does something more than just send a SIGTERM that Elasticsearch doesn't like.
Hi Guys,
I really want to say that it's very nice that we have yet another version of helm charts for Elastic and Kibana,
But the question is why you don't participate in creating charts from helm/charts repo ?
They are more mature and well tested. And for sure they have many more options.
WDYT guys ?
Regards,
Maciek
Hello,
When customising the basePath, Kibana expect according request, which make the readiness failing, and Kibana service never up.
It's here:
When using this config:
kibana.yml: |
server.basePath: "/kibana"
server.rewriteBasePath: true
We would need to have the readiness probe hitting http '/kibana/app/kibana'
.
Thank you.
It does not look like imagePullSecrets tag in values.yaml is being processed. We need this because we pull from an internal Docker store, rather than the public ones.
Steps to reproduce:
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "6.6.0"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: "somesecret"
helm install elasticsearch --debug --dry-run --name elasticsearch
Output:
image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 6.6.1
I tried several permutations (double quotes, no quotes, newline with the ' - somesecret ' notation, etc) to no avail.
The Kibana chart also suffers from the same problem. If someone fixes the Elasticsearch chart, I'll fix the Kibana chart.
Chart version: 7.0.1-alpha1
Kubernetes version: v1.13.1
Kubernetes provider: Barer metal
Helm Version: v2.11.0
Run the helm chart.
helm install elastic/elasticsearch --namespace efk --name efk-elasticsearch --version 7.0.1-alpha1 --set imageTag=7.0.1 --set replicas=1 --set resources.requests.memory=1Gi --set volumeClaimTemplate.storageClassName=nfs --set volumeClaimTemplate.resources.requests.storage=600Gi --set "nodeSelector.kubernetes\.io/hostname"=node3
Persistent Volume is created, and gets the status "bound".
elasticsearch-master-elasticsearch-master-0 Bound pvc-a23fd9a2-7347-11e9-bdfc-0050563dbb49 600Gi ReadWriteOnce nfs
But the elasticsearch pod does not starts.
pod has unbound immediate PersistentVolumeClaims Unable to mount volumes for pod "elasticsearch-master-0_efk(a240e2f2-7347-11e9-bdfc-0050563dbb49)": timeout expired waiting for volumes to attach or mount for pod "efk"/"elasticsearch-master-0". list of unmounted volumes=[elasticsearch-master]. list of unattached volumes=[elasticsearch-master default-token-q8bks]
Thanks for the help.
I see that this Chart is tested against GKE. I work at Nike and we are using EKS.
According to the Helm Chart documentation, I believe the only thing I need to change in the values.yaml
file is the storageClassName: "standard"
variable to storageClassName: "gp2"
I have verified that the cluster does, in fact, have a storage class set up. However, when I go to deploy the Chart and I describe my master elasticsearch nodes, I am getting this error message:
pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
I believe that I am still missing some configuration setting having to do with Storage. Please let me know if you need any additional information. Any help is appreciated.
As discovered in #22 (review) the values used to connect to Elasticsearch differ between Kibana and Elasticsearch. In the future when charts are added for Logstash/Beats/others this could become even more fragmented and annoying to configure.
Because the Elasticsearch chart needs to be able to access parts of the Elasticsearch URL separately I think it makes most sense to split the URL up into separate variables.
Proposal:
esHost: elasticsearch-master
esProtocol: http
esPort: 9200
Request came from: #111 (comment)
Hi all,
I read how this chart works and I don't understand why is there masterService
in values.yaml.
Because service is defined with {{ template "uname" . }}-headless
which is exactly the concatenation of clusterName and nodeGroup {{ .Values.clusterName }}-{{ .Values.nodeGroup }}
.
In helpers :
{{- define "uname" -}}
{{ .Values.clusterName }}-{{ .Values.nodeGroup }}
{{- end -}}
In service.yaml :
kind: Service
apiVersion: v1
metadata:
name: {{ template "uname" . }}-headless
But in statefulset.yaml :
- name: discovery.zen.ping.unicast.hosts
value: "{{ .Values.masterService }}-headless"
The discovery.zen.ping.unicast.hosts
avoid Elasticsearch to start if you use the 8.0.0 Docker image
{"type": "server", "timestamp": "2019-04-17T13:51:31,328+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [discovery.zen.ping.unicast.hosts] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"Caused by: java.lang.IllegalArgumentException: unknown setting [discovery.zen.ping.unicast.hosts] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:148) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.node.Node.<init>(Node.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.node.Node.<init>(Node.java:253) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"... 6 more"] }
Describe the feature:
The ability to import saved objects like dashboards, index pattern, ... would be most useful.
Logs from foo-master-0 show it timed out searching for other masters:
[2018-12-18T06:49:37,298][INFO ][o.e.p.PluginsService ] [foo-master-0] loaded plugin [ingest-user-agent]
[2018-12-18T06:49:49,072][INFO ][o.e.x.s.a.s.FileRolesStore] [foo-master-0] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2018-12-18T06:49:50,286][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [foo-master-0] [controller/205] [Main.cc@109] controller (64 bit): Version 6.5.3 (Build f418a701d70c6e) Copyright (c) 2018 Elasticsearch BV
[2018-12-18T06:49:51,588][INFO ][o.e.d.DiscoveryModule ] [foo-master-0] using discovery type [zen] and host providers [settings]
[2018-12-18T06:49:54,000][INFO ][o.e.n.Node ] [foo-master-0] initialized
[2018-12-18T06:49:54,000][INFO ][o.e.n.Node ] [foo-master-0] starting ...
[2018-12-18T06:49:54,594][INFO ][o.e.t.TransportService ] [foo-master-0] publish_address {10.0.5.23:9300}, bound_addresses {[::]:9300}
[2018-12-18T06:49:54,608][INFO ][o.e.b.BootstrapChecks ] [foo-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-12-18T06:50:24,687][WARN ][o.e.n.Node ] [foo-master-0] timed out while waiting for initial discovery state - timeout: 30s
[2018-12-18T06:50:24,696][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [foo-master-0] publish_address {10.0.5.23:9200}, bound_addresses {[::]:9200}
[2018-12-18T06:50:24,697][INFO ][o.e.n.Node ] [foo-master-0] started
[2018-12-18T06:50:29,077][WARN ][r.suppressed ] [foo-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
<the above message repeats every 10 seconds>
[root@foo-master-0 elasticsearch]# ping foo-master-headless
PING foo-master-headless.default.svc.cluster.local (10.0.5.23) 56(84) bytes of data.
64 bytes from foo-master-0.foo-master-headless.default.svc.cluster.local (10.0.5.23): icmp_seq=1 ttl=64 time=0.013 ms
...
[root@foo-master-0 elasticsearch]# dig +short foo-master-headless.default.svc.cluster.local
10.0.4.7
10.0.5.23
10.0.3.39
It seems somehow that dns has cached that 'foo-master-headless` resolves to self. Symptoms of this problem are noticed when a node fails to join the cluster:
% kubectl get pods
NAME READY STATUS RESTARTS AGE
foo-data-0 1/1 Running 0 32m
foo-data-1 1/1 Running 0 32m
foo-data-2 0/1 Running 0 20m
foo-master-0 0/1 Running 0 20m
foo-master-1 1/1 Running 0 32m
foo-master-2 1/1 Running 0 32m
Sometimes the cluster forms fine. Other times, certain nodes get stuck and never seem to join the cluster.
heml install line still has 6.6.2-alpha1
In the migration example I can see a couple of issues:
Also, even there doesn't seem to be a way to disable PVC creation for a Client node. PVC always gets created.
Hi
I'm unable to find a way to get the free / open source x-pack license
can you please guide how to?
The readinessProbe only appears to target HTTP, not HTTPS.
Imho it would be cool if the charts would be a bit promoted. Sp please add your charts to https://hub.helm.sh by creating a pr on: https://github.com/helm/hub/blob/master/config/repo-values.yaml
It would be great to have official helm charts for running APM Server on kubernetes.
#17, according to GitHub, is in v6.6.0-alpha1:
However, the actual release seems to be missing this PR?
% grep -A2 kibana requirements.yaml
- name: kibana
repository: https://helm.elastic.co
version: 6.6.0-alpha1
% helm dep update
...
Downloading kibana from repo https://helm.elastic.co
...
% grep -A2 kibana requirements.lock
- name: kibana
repository: https://helm.elastic.co
version: 6.6.0-alpha1
% tar -zxf charts/kibana-6.6.0-alpha1.tgz -O kibana/templates/deployment.yaml
...
readinessProbe:
{{ toYaml .Values.readinessProbe | indent 10 }}
...
http "/app/kibana"
This doesn't match what was merged in the #17.
Thoughts?
The Kibana chart should provide the podAnnotations
value to add annotations to the created pod.
This feature is essential to using the new "autodiscovery hints" feature in Filebeat and Metricbeat.
When using nested charts (parent/child/etc), it's often useful to add additional entries for values to pass into in child charts. However, extraEnvs
is a list, and helm cannot merge lists, but it can merge maps! (Citation: https://github.com/jordansissel/experiments/tree/master/helm/subchart-values-from-parent)
By way of example, I might have one child chart which configures Elasticsearch with my favorite default features (enable security, enable tls, etc). And I could create a new chart where I want this, plus creating a SAML realm. Because extraEnvs
is a list, I can't just inject these SAML-specific settings. Instead, I have to copy+paste the child chart's extraEnvs
(enable security, enable tls, etc) and append my SAML ones.
If extraEnvs
were a map, I believe this would be solved:
Example appearance in the child described in above:
extraEnvs:
xpack.security.enabled: { value: true }
...
ELASTIC_PASSWORD:
valueFrom: { secretKeyRef: { name: elastic-credentials, key: password } }
And the parent chart (to add SAML, for example)
extraEnvs:
xpack.security.authc.realms.saml1.type: { value: true }
...
Thoughts? I know this would deviate from Kubernetes "environment variables are defined as an list", so I'm open to discussion.
Stepping back, one alternative could be to have a general "settings" which could be used to generate a configmap instead of using environment variables for settings.
Hello,
we are running kubernetes on-premise where local storage is used. Unfortunately, auto-provisioning for local storage volumes is not yet available on kubernetes. It would be very useful if the chart supports "hostPath" as option alternatively to volumeClaimTemplate.
I added a similar feature to redis-ha a while ago: helm/charts#11815
I you agree to this change, I'd create a PR for elastic.
Best regards,
Michael.
helm repo add elastic https://helm.elastic.co
helm repo update
helm install --name filebeat elastic/filebeat --version 7.0.0-alpha1
Error: failed to download "elastic/filebeat"
I'm trying to follow Installing, yet getting following error:
toor@suey:~$ helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
toor@suey:~$ helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
elastic https://helm.elastic.co
toor@suey:~$ helm install --name elasticsearch elastic/elasticsearch --version 6.5.3-alpha1
Error: could not find tiller
toor@suey:~$
toor@suey:~$ tiller -version
v2.12.0
toor@suey:~$
Please advise.
Hello there,
I was looking for the best way to deploy elasticsearch / kibana on openshift and it looks like I could use some help. : )
I tried adapting these charts and kept some notes with the problems that came up.
I would welcome suggestions (even for a completely different method of deploying into openshift - e.g. openshift templates / operator) and am happy to help with any questions you might have so we can figure something out (I have an openshift cluster standing by).
Thanks!
Hello,
When trying to set up a multi-role based cluster I encounter the following issue after setting up the master nodes :
{"type": "server", "timestamp": "2019-05-02T01:16:08,994+0000", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-data-0", "message": "failed to resolve host [master-headless]" ,
"stacktrace": ["java.net.UnknownHostException: master-headless",...
The configs of the masters :
clusterName: "elasticsearch"
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.0.0"
imagePullPolicy: "IfNotPresent"
esMajorVersion: 7
protocol: "http"
nodeSelector:
app: elasticsearch
replicas: 3
nodeGroup: "master"
roles:
master: "true"
ingest: "false"
data: "false"
ressources:
requests.cpu: 500m
requests.memory: 2Gi
limits.cpu: 1500m
limits.memory: 2Gi
esJavaOpts: "-Xmx1g -Xms1g"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
storageClassName: do-block-storage
The configs of the data nodes :
clusterName: "elasticsearch"
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.0.0"
imagePullPolicy: "IfNotPresent"
esMajorVersion: 7
protocol: "http"
nodeSelector:
app: elasticsearch
masterService: "master"
replicas: 3
nodeGroup: "data"
roles:
master: "false"
ingest: "false"
data: "true"
ressources:
requests.cpu: 500m
requests.memory: 2Gi
limits.cpu: 1500m
limits.memory: 2Gi
esJavaOpts: "-Xmx1g -Xms1g"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 250Gi
storageClassName: do-block-storage
EDIT :
Its look like there is an issue wirth the name generation (?) because the concerned service should be elasticsearch-master-headless
but only master-headless
is mentioned in the log of the data
nodes
We can see as well that the DNS receive the actual request :
2019-05-02T03:07:07.093Z [INFO] xx.xx.xx.133:60301 - 35027 "A IN master-headless.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000196715s
2019-05-02T03:07:07.093Z [INFO] xx.xx.xx.133:60301 - 4324 "AAAA IN master-headless.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000291045s
2019-05-02T03:07:07.094Z [INFO] xx.xx.xx.133:59951 - 47699 "A IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000130911s
2019-05-02T03:07:07.094Z [INFO] xx.xx.xx.133:59951 - 26206 "AAAA IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000127551s
2019-05-02T03:07:07.095Z [INFO] xx.xx.xx.133:43284 - 19072 "A IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000137498s
2019-05-02T03:07:07.096Z [INFO] xx.xx.xx.133:43284 - 62598 "AAAA IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000205652s
2019-05-02T03:07:09.616Z [INFO] xx.xx.xx.157:57068 - 64127 "AAAA IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000086844s
2019-05-02T03:07:09.616Z [INFO] xx.xx.xx.157:57068 - 1916 "A IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000160398s
2019-05-02T03:07:10.021Z [INFO] xx.xx.xx.176:33201 - 15919 "A IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,rd 140 0.000085146s
2019-05-02T03:07:10.022Z [INFO] xx.xx.xx.176:33201 - 60982 "AAAA IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,rd 140 0.000082036s
I have set esJavaOpts
to -Xmx8g -Xms8g"
which correctly sets the env var in the container: ES_JAVA_OPTS=-Xmx8g -Xms8g
There also exists a jvm.options
configuration which sets -Xms1g
and -Xmx1g
. This seems to cause the following settings when starting Elasticsearch:
1000290+ 1 0 99 07:36 ? 04:12:54 /opt/jdk-11.0.1/bin/java -Xms1g -Xmx1g
[...]
-Xmx8g -Xms8g
[...]
Hopefully most clusters have something along the lines of restricted for the default service account in a namespace. By default, with no serviceAccount specified, the default
serviceAccount is used.
Two values are required, one for the initContainer, which is already requiring privileged permissions and root access, and one for the es container, which doesn't require the same level.
Chart version: 6.6.0-alpha1
Istio version: 1.1.0-snapshot.6
When deploying the chart on a Istio cluster, I get the error "not enough master nodes discovered during pinging"
After some debugging looks like since Elasticsearch is using elasticsearch-master-headless
service to discover the master nodes, this service only has defined the port 9200
in its service definition. Adding the transport
port of 9300
fixed this error.
Seems the Docker imageTag is set to "6.6.0" in the '6.6.0-alpha1' helm chart.
Is the '6.6.0-alpha1' chart a stable '6.6.0' application release?
Seems like it is not possible to find any node in the cluster, even do all are up and running checking for the other nodes.
{"type": "server", "timestamp": "2019-04-17T14:12:01,189+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304] from hosts providers and [{elasticsearch-master-0}{-6TRq956TxeZVWcFddKTKQ}{kLzc0NhoQj6v2LvAiCBd7A}{10.8.0.12}{10.8.0.12:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
Changing the name to the whole headless name does not work eather elasticsearch-master-{0..3}.elasticsearch-master-headless.default.svc.cluster.local
{"type": "server", "timestamp": "2019-04-17T14:20:53,452+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0.elasticsearch-master-headless.default.svc.cluster.local, elasticsearch-master-1.elasticsearch-master-headless.default.svc.cluster.local, elasticsearch-master-2.elasticsearch-master-headless.default.svc.cluster.local] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304] from hosts providers and [{elasticsearch-master-0}{-6TRq956TxeZVWcFddKTKQ}{y5MQasEnQ2W-geYeRuaHyA}{10.8.0.13}{10.8.0.13:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
similar to #72
@Crazybus can you please bump the chart version to 7.0.0?
Hey,
we already deployed elasticsearch 6.6.0 and would like to use helm for the next update. Is it possible to use the "old" pvc containting our data without having copy it somewhere?
Right know, after reading the docs, I thought of:
Is there any easier way?
I'm trying to mount the elasticsearch keystore per the documentation but I keep getting the error below. I've verified the the elasticsearch keystore file is valid (by adding it to a test continer and testing the values).
Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
Command to create the secret:
kubectl create secret generic elasticsearch-keystore --from-file=./elasticsearch.keystore
SecretMounts in my yaml configuration:
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
- name: elastic-license
secretName: elastic-license
path: /usr/share/elasticsearch/config/license
- name: elasticsearch-keystore
secretName: elasticsearch-keystore
path: /usr/share/elasticsearch/config/elasticsearch.keystore
subPath: elasticsearch.keystore
Is there something I am missing?
With elasticsearch , in some cases , we need some extra volumes with your own properties to store snapshots or backups (for example). Current version , only supports one VolumeClaimTemplate. Could be very useful for us, support for multi volumes on elasticsearch group nodes.
Thanks
After deploying the elasticsearch chart to one namespace, attempting to deploy it to a second namespace appears to fail with the following error:
Release "foo" does not exist. Installing it now.
Error: release draal failed: poddisruptionbudgets.policy "foo-data-pdb" already exists
There seems to be some conflict on the name. Should this resource be named to include the namespace where the chart is installed?
@Crazybus any chance to publish the new chart if it's ready ?
I saw you committed all the changes for ES7 and I am hoping to use the new chart asap without cloning the repo.
Thanks in advance.
Are you working on Logstash Helm chart by chance? I found this one, but it's not working right: https://github.com/helm/charts/tree/master/stable/logstash - it's got some issues....
For Elasticsearch, the statefulset name seems to follow {clusterName}-{nodeGroup}
Kibana's deployment, however, seems to be kibana-kibana
. If I set nameOverride
for Kibana, the deployment becomes kibana-{nameOverride}
.
I kinda want to see things grouped by cluster name, and while Kibana has no such concept, I would wish for the kibana deployment to be called {nameOverride}-kibana
.
Given this is a naming proposal, I'm almost willing to close it immediately myself because naming is hard and maybe nobody will like any one proposal ;)
Thoughts?
Including podManagementPolicy
in values.yaml makes it seem like it may be, but I am not sure it would work, with the way readiness probes are used. Could you please clarify this? (Maybe also in the docs ...)
In the elasticsearch statefulset readinessProbe command, the cluster health status has a hardcoded value of green
. It's not always the case that, as with coordinator nodes, green is the minimum acceptable state for cluster health.
Hello,
I'm trying to follow Installing, yet running into following scenario:
mbp:~ alexus$ helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
mbp:~ alexus$ helm install --name elasticsearch elastic/elasticsearch --version 6.5.3-alpha1
NAME: elasticsearch
LAST DEPLOYED: Thu Dec 20 21:32:34 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/StatefulSet
NAME DESIRED CURRENT AGE
elasticsearch-master 3 3 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 0/1 Pending 0 0s
elasticsearch-master-1 0/1 Pending 0 0s
elasticsearch-master-2 0/1 Pending 0 0s
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
elasticsearch-master-pdb N/A 1 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-master ClusterIP 10.110.160.47 <none> 9200/TCP,9300/TCP 0s
elasticsearch-master-headless ClusterIP None <none> 9200/TCP 0s
NOTES:
1. Watch all cluster members come up.
$ kubectl get pods --namespace=default -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
$ helm test elasticsearch
mbp:~ alexus$
mbp:~ alexus$ kubectl get pods --namespace=default -l app=elasticsearch-master -w
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 0/1 Pending 0 25s
elasticsearch-master-1 0/1 Pending 0 25s
elasticsearch-master-2 0/1 Pending 0 25s
mbp:~ alexus$ helm test elasticsearch
RUNNING: elasticsearch-rxsef-test
FAILED: elasticsearch-rxsef-test, run `kubectl logs elasticsearch-rxsef-test --namespace default` for more info
Error: 1 test(s) failed
mbp:~ alexus$ kubectl logs elasticsearch-rxsef-test --namespace default
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0curl: (7) Failed connect to elasticsearch-master:9200; Connection refused
mbp:~ alexus$
mbp:~ alexus$ helm version
Client: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
mbp:~ alexus$ docker --version
Docker version 18.09.0, build 4d60db4
mbp:~ alexus$ uname -a
Darwin mbp.lan 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
mbp:~ alexus$
I'm using latest Desktop Docker (macOS).
Please advise.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.