Comments (13)
This would be useful even for local development where you'd like to emulate a production cluster environment on localhost.
from puppet-elasticsearch.
Hi @afx114 how would you see the instance creation part?
I was thinking of the following:
elasticsearch::instance { 'instance-a':
config => { }
}
elasticsearch::instance { 'instance-b':
config => { }
}
These will create their own directory for the config and init files.
from puppet-elasticsearch.
Really don't see why add more than one ES to one VM because in production even in cluster you will operate (1:1) one hardware (VM) = one ES. And with Vagrant already you can set a multi-VM, multi-ES cluster locally, just using small OS install footprints. For example https://github.com/kenny13/elasticsearch/blob/master/Vagrantfile with "vagrant up /localInstance[1-3]/" you will end up with ES cluster of three VM's .
from puppet-elasticsearch.
I do know that in some situations people in production have very powerful machines ( 32 cores, 128GB mem or more )
in some of those cases people could run more then 1 instance on 1 hardware machine.
from puppet-elasticsearch.
Plus, it's much easier to just provision using @electrical 's proposal above than having to deal with multiple VMs.
from puppet-elasticsearch.
On a support call with Elasticsearch yesterday, it was recommended that we double the RAM on our data nodes (to 128GB) and run two instances per-node. We were thinking of a custom multi-instance initscript (overwriting the RPM-provided one) and separate configs (to provide different data and logs directories, as well as config to ensure the primary and replica dataset aren't both stored on the same host, etc.), but I don't have a good idea of the wider concerns. Perhaps managing the instances as separate services is preferable, but potentially more confusing.
What I'm most concerned with at the moment is trying to work within the RPM-provided skeleton (/etc/sysconfig/elasticsearch
, /etc/init.d/elasticsearch
) while also adding a second instance. So that would require either being able to update the initscript to support multiple instances with different configs, or configuring one instance the "normal" way and being able to tack on an additional instance with a new initscript + defaults.
I know that systemd can handle multi-instance support moderately easily, but I'm stuck in SysV world for better or worse.
from puppet-elasticsearch.
Im currently in the process of making a design document for the multi-instance support.
There is only 1 issue with the current plan: it will be a non backwards compatible release.
The idea is to have multi instance support in this form:
class { ‘elasticsearch’:
manage_repo => true,
repo_version => ‘1.2’,
config => { cluster.name’ => ‘clustername’ }
}
elasticsearch::instance { ‘instance_a’:
config => { ‘node.name’ => ‘instance_a’ }
}
elasticsearch::instance{ ‘instance_b’:
config => { ‘node.name’ => ‘instance_b’ }
}
The downside of this is that when only a single instance is required, you still need to define an instance since we can't detect automatically if an instance is defined or not by the user.
The only way would be to add an option multi_instance => true/false
to the class to create the default instance or not.
What are your opinions about this?
from puppet-elasticsearch.
As part of my personal preference, I very much don't like making the "simple use cases" more complex. So I'm a fan of the notion of having the multi_instance flag, and having it default to "false". And when it's false, it rolls out the default instance for you.
And then I would say if you set the multi_instance flag to "true", you would then need to define all of your instances at that point.
I mean... I guess in theory with that model you could leave multi_instance set to "false", and then define other elasticsearch::instance objects anyway? It would create a potentially confusing scenario where you SAY you're not doing multi_instance, and then really you are... but I don't think that's a deal-breaker, personally.
And if you want to have multiple instances, it seems to me that having to know to add multi_instance => true; and then define all of your instances by hand, really isn't that unexpected of a thing.
Unless I'm missing something, having the multi_instance flag and defaulting it to "false" seems most elegant way to go here. Keeps the simple use cases simple, and the more complex ones... still pretty intuitive.
from puppet-elasticsearch.
Agreed with @abaddononion -- since multiple instances seems like such an edge-case, the configs for it should probably be edge-case as well. Best to keep the non-edge-case stuff simple.
from puppet-elasticsearch.
I'm in favor of of requiring one elasticsearch::instance
resource per ES instance because:
Objectively (hopefully, anyway)
- it permits use with hiera more easily
- it permits creation of any number of ES nodes using create_resource()
Given the above, I am in favor of this being the minimal declaration:
class { "elasticsearch": ... ; }
elasticsearch::instance { "node-name-here": ; }
Subjectively:
- I prefer explicit declarations rather than implicit. This aids in maintenance and readability later. This means I want you to use
elasticsearch::instance
for every ES node you want to run. - If you move from 1 instance to 2 instances, it is less work and more obvious if we require
elasticsearch::instance
explicitly. "less work" because with this method, you can simply create a 2nd resource:
# Add a 'node2' !
elasticsearch::instance { ["node1", "node2"]: ... ; }
But with the multi_instance world, you have to do two things: First, set multi_instance => false
and then create two new elasticsearch::instance resources.
Maintenance feels easier to me if we require an instance for single node deployments.
from puppet-elasticsearch.
+1 @ #62 (comment)
at $work
, we try to construct our in-house modules such that you instantiate a "named instance" of a service. we've found this pattern to be extremely powerful for creating complex infrastructures and that it has long-term maintenance benefits.
from puppet-elasticsearch.
Been busy with the rewrite for the multi instance support.
Progress can be found in the multi_instance_rewrite branch.
The main implementation is working and currently running extensive system tests.
Still need to fix the rspec tests.
Hope to have it released in a few days.
from puppet-elasticsearch.
Forgot to update the thread.
The Multi instance support was released in version 0.4.0
Cheers.
from puppet-elasticsearch.
Related Issues (20)
- elasticsearch_keystore can't modify frozen String
- elasticsearch_keystore resource fails due to legacy instance code HOT 3
- Node facts not generated when Elasticsearch in cluster mode
- Could not evaluate: undefined method `join' for :name:Symbol HOT 2
- elasticsearch_plugin/ruby.rb wrong path to call plugin binary HOT 2
- ssl=true and version is not defined, gives an error HOT 1
- Setting an Elastic Version should be optional HOT 1
- New release, please HOT 1
- OpenSearch support ? HOT 1
- Elasticsearch API responded with: no handler found for uri [/_xpack/license?acknowledge=true] and method [PUT]
- File jvm.options in root is unmanaged/may be missing
- why was log4j2.properties.erb configuration commented ? HOT 1
- warnings when used with stlib 9. HOT 1
- fix lib directory permissions without changing the mode of files in the directory
- puppet-elastic_stack version HOT 1
- Use Elasticsearch 7+ as default for configuration HOT 1
- mediawiki needs 6.8.23 and I needed to change this
- Issues on the elasticsearch_component_template.rb and elasticsearch_index_template.rb HOT 2
- Problem with puppet CA and elasticsearch.keystore
- `bin/plugin` is used insead of `bin/elasticsearch-plugin` on the first run
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from puppet-elasticsearch.