Comments (19)
Well that was a fun journey. I found it though!
In elastic/elasticsearch#41701 the internal format for the keystore was bumped from version 3 to version 4. On startup Elasticsearch notices this and attempts to upgrade the format of the keystore. All mounted secrets and configmaps in Kubernetes are always readonly. So when it tries to make the change you get the error as seen above.
So the fix is to either:
- Make sure you are creating the keystore with the right version of Elasticsearch. So if you are using 7.1.1 in the cluster you should use the same version when creating the keystore
- If you already have an existing keystore you should run
elasticsearch-keystore upgrade
with the right Elasticsearch version to get it upgraded to the right format.
from helm-charts.
@JanKowalik Thank you for providing the extra details. I think that the issue you are running into is because you have the $ELASTIC_PASSWORD
set but aren't adding it to your keystore as the bootstrap password. The docker image startup script tries to add this to the keystore when it is set.
You want to add it with:
elasticsearch-keystore add -x bootstrap.password
Once #154 is finished off there won't be any need to manually create and update the keystore anymore.
from helm-charts.
It looks like you did everything right to me.
Can you give me the output of the following commands:
You can attach into one of the containers by running kubectl exec -ti elasticsearch-master-0 bash
head -n1 /usr/share/elasticsearch/config/elasticsearch.keystore
ls -lhatr /usr/share/elasticsearch/config/
df -h
elasticsearch-keystore list
Could you also give me the following:
- The rest of your configuration
- Output of
kubectl get events
after attempting to deploy - Kubernetes provider and version (e.g. Google Kubernetes Engine 1.12)
from helm-charts.
I'm also running into this same issue after following the instructions for using the keystore in the readme - created the same way as in this issue.
This is running on GKE 1.11.8-gke-6 and I'm not seeing anything of note in kubectl get events
, just appears to create the container/pull container etc, and then "back off restarting failed container".
Full error log from the container is:
Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
at java.base/java.nio.file.Files.move(Files.java:1424)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:500)
at org.elasticsearch.common.settings.AddStringKeyStoreCommand.execute(AddStringKeyStoreCommand.java:97)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)
from helm-charts.
@ofaz Thanks for the report! Could you give me the following information to try to reproduce it:
- Output of
helm get elasticsearch
(or whatever your release name is) - Output of
kubectl get pod -o yaml elasticsearch-master-0
- Output of
kubectl get events
- The output of these commands run from within one of the containers:
head -n1 /usr/share/elasticsearch/config/elasticsearch.keystore
ls -lhatr /usr/share/elasticsearch/config/
df -h
elasticsearch-keystore list
from helm-charts.
I just bumped one of our internal clusters to use 7.1.0 and got the same error. So this seems to be an issue with how 7.1.0 is accessing the keystore differently to previous releases. No need to send anymore debug information now that I can reproduce it.
from helm-charts.
Hi,
I have the same issue with version 6.8.1. Keystore was created using the same version, so the fix above does not do the trick for me.
from helm-charts.
@JanKowalik if it really was made with the same version than this sounds like a different issue. Can you give me the exact commands you used to create the keystore and the output of helm get elasticsearch
(replace elasticsearch with your release name).
from helm-charts.
I will try it again to make sure and if it does not work I will provide the information you are sking for.
Thank you.
from helm-charts.
It did not work this time too.
I generated the keystore within docker using elastic 6.8.1 and the created a secrete out of it.
The command I ran within docker image is:
elasticsearch-keystore create
I moved the created file out of docker container and used this command to create the secret:
kubectl create secret generic elasticsearch-keystore --from-file=./elasticsearch-keystore/elasticsearch.keystore -o yaml --dry-run > manifests/elasticsearch-keystore-config-secret.yaml
I used helm chart to generate manifests only. I can attach the manifest files and values I used if that helps?
elasticsearch-master-nodes.txt
values-data-nodes.txt
values-master-nodes.txt
elasticsearch-data-nodes.txt
I did not include secrets here and image
needs to be replaced with:
docker.elastic.co/elasticsearch/elasticsearch:6.8.1
I use kustomize for that.
The error message I get:
Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
at java.base/java.nio.file.Files.move(Files.java:1424)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:500)
at org.elasticsearch.common.settings.AddStringKeyStoreCommand.execute(AddStringKeyStoreCommand.java:97)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)
from helm-charts.
Yeah, I did not add the bootstrap.password. I did not think it was necessary if I am using the default password.
I will give it a go and report back.
Thank you for your help!
from helm-charts.
Nearly works. But I think this is a different problem now.
How does setting bootstarp.password
influence elastic
user password? I have 2 out of 3 nodes in either master-nodes or data-nodes groups connected fine but thirds are complaining about authentication.
[2019-07-18T16:20:48,202][INFO ][o.e.x.s.a.AuthenticationService] [elasticsearch-data-2] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
from helm-charts.
Nearly works. But I think this is a different problem now.
How does setting bootstarp.password influence elastic user password? I have 2 out of 3 nodes in either master-nodes or data-nodes groups connected fine but thirds are complaining about authentication.
Which version of Elasticsearch are your running? In the manifest output I see the image is image: "elasticsearch:pulse"
. The reason that I ask is that in Elasticsearch versions before security was included in basic (before 6.8 release) you needed to activate the license for the cluster to fully form. What would happen is that the first 2 nodes would start bootstrapping the cluster, and the third one would fail to join until the license was enabled. The output from curl -u elastic:$ELASTIC_PASSWORD localhost:9200/
will show the version.
from helm-charts.
It uses version 6.8.1
from helm-charts.
How does setting bootstarp.password influence elastic user password?
To be clear, the bootstrap.password
should be set to what you have for $ELASTIC_PASSWORD
. Have all nodes in the cluster been restarted with the new keystore btw?
from helm-charts.
It is all working fine now. I deleted everything and recreated a cluster from scratch and it worked.
I tried scaling everything down to 0 and then back up, but it did not help. Not sure why. After that all nodes had authentication errors. Maybe adding bootstrap.password to an existing cluster is not advisable.
Thank you for your time and help.
from helm-charts.
I have an issue with kibana-keystore if mounted as k8s secret now.
onfiguring logger failed: { Error: EISDIR: illegal operation on a directory, read
at Object.readSync (fs.js:494:3)
at tryReadSync (fs.js:333:20)
at readFileSync (fs.js:370:19)
at Keystore.load (/usr/share/kibana/src/server/keystore/keystore.js:97:45)
at new Keystore (/usr/share/kibana/src/server/keystore/keystore.js:46:10)
at readKeystore (/usr/share/kibana/src/cli/serve/read_keystore.js:40:20)
at applyConfigOverrides (/usr/share/kibana/src/cli/serve/serve.js:186:41)
at applyConfigOverrides (/usr/share/kibana/src/cli/serve/serve.js:50:42)
at config_1.RawConfigService.rawConfig (/usr/share/kibana/src/core/server/bootstrap.js:33:134)
at MapSubscriber.RawConfigService.config$.rawConfigFromFile$.pipe.operators_1.map.rawConfig [as project] (/usr/share/kibana/src/core/server/config/raw_config_service.js:41:24) errno: -21, syscall: 'read', code: 'EISDIR' }
FATAL Error: EISDIR: illegal operation on a directory, read
@Crazybus: Shall I open another ticket for that?
from helm-charts.
@Crazybus: Shall I open another ticket for that?
Yes please! If I'm honest I have never actually used the keystore for Kibana with the helm-charts. My bet is that its going to be failing for the same issue as Elasticsearch (docker image trying to automatically add the ELASTICSEARCH_PASSWORD
to the keystore on startup.
from helm-charts.
The changes in #90 will also be ported to the other charts which will make this a lot easier to manage.
from helm-charts.
Related Issues (20)
- [Kibana] 8.5.1"message":"secrets \"kibana-kibana-es-token\" already exists" HOT 4
- Support file realm in the Elasticsearch spec
- readiness probe also always fails when security is disabled HOT 1
- Readiness probe is failing because of bad SSL HOT 1
- Kibana chart helm install failing with Error: INSTALLATION FAILED: failed pre-install: job failed: BackoffLimitExceeded HOT 5
- Logstash does not respect logstashJavaOpts
- Out of memory error both on
- Kibana chart fails when security disabled HOT 1
- Enabling security breaks ES endpoints.
- Kibana post delete can fail, leaving artifact behind. HOT 2
- Cannot enable Metricbeat modules HOT 1
- Elasticsearch 7.17.9 HOT 1
- [kibana] helm deployment with no values fails, kubeadm kubernetes deployment 1.25.3 HOT 1
- elasticsearch chart optimizes the discovery.seed_hosts Settings HOT 1
- How to add license on the helmchart
- Randomly receive list of indices with kubectl exec on single node installation.
- Kibana 8.5.1 Upgrade failed from 7.17.3- Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
- UNABLE_TO_VERIFY_LEAF_SIGNATURE HOT 5
- [logstash] Statefulset annotations
- kibana cannot be installed, output self signed certificate in certificate chain HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from helm-charts.