Use a more elegant way to parse port ranges and reduce code complexity.

I prefer to work in Vagrant wherever possible so as to avoid polluting my laptop. I'm working on a centos7 box for myself right now anyway to solve this problem.

If there is demand for this, please let me know and also state your preferred distribution. I have no particular preference for centos.

The scheduler should use mesos/elasticsearch-cloud-mesos by default but you should be able to override this via a command line option.

Currently the framework launches elasticsearch on every node.

In large clusters this is not desired, therefore the framework should also support an auto-scaling strategy. With this strategy the framework monitors resource usage and for instance when the heap size becomes 70% of total memory it launches a new elasticsearch node to scale the load.

See https://github.com/soldair/docker-alpine-elasticsearch

Despite sending the commands to set the FrameworkID, mesos does not register the framework with the requested ID.

Required for DiscoveryInfo.

Please advise what is current project status, is it ready for use or still in development?
Is there any roadmap?

Currently the framework selects the first 2 ports from the offered port ranges. However it should select 2 random ports from the port ranges because in case of task failures there is the chance that the same port was used before by another task.

List of current issues

• Fix IndexCreationException #39 @frankscholten
• Only 2 out of 3 nodes is starting #37 @frankscholten
• Setup BigDesk @mrmrcoleman
• Add example data #38 @mwl

Currently we use the Master's state file to discover nodes in the cluster. Replace this with Mesos DNS node discovery mechanism.

• Expose services in mesos-dns
• Deploy mesosphere/mesos-dns
• Configure DNS on all nodes
• Look up ES instances through DNS in Mesos-cloud plugin
• Automate adding prepend domain-name-servers 10.11.142.163; to /etc/dhcp/dhclient.conf on all nodes

The scheduler should not use the master url parameter. Instead the master url should be configured through MasterInfo when the framework is registered.

See https://github.com/mesos/kafka/blob/master/src/scala/ly/stealth/mesos/kafka/HttpServer.scala in Kafka framework.

Cloud-mesos parses the state but when an exception is thrown it will be called again, creating an infinite loop.

One case where this happens is when the 'cluster' property is not found in the state.json file. This field is set by GCE but not in docker compose.

Consider changing the JSON API we use because it throws an exception if a property is not available.

Executor should send simple metrics from the Elasticsearch node such as number of searches back to the scheduler so the scheduler can decide whether to scale up or down.

Scheduler should launch Docker container with Elasticsearch + Elasticsearch Mesos cloud plugin

Let's setup a way to run automated system tests with docker-compose.

The steps are

• docker-compose up - either via Gradle or within JUnit
• Perform assertions such as calling the Elasticsearch endpoints to check if node discovery works
• docker-compose rm

When using discovery mode 'auto' or 'mesos-dns' the port numbers are not correctly parsed.

Add a unit test & docker-compose setup with the Mesos DNS container to test this scenario.

Cloud-mesos should use Mesos-DNS if possible or fall back to simple parsing of the state file if Mesos-DNS is not available.

Even if nothing has changed, it takes lots of time to determine that nothing has changed in :scheduler:jar and :executor:jar tasks

:elasticsearch-cloud-mesos:processResources UP-TO-DATE
:elasticsearch-cloud-mesos:classes UP-TO-DATE
:elasticsearch-cloud-mesos:jar UP-TO-DATE                   <--- several minutes
:elasticsearch-cloud-mesos:assemble UP-TO-DATE

it looks the problem is caused due to shadowing grade plugin

Currently the ES plugin communicates directly with mesos. It should communicate via the executor.

Java 7 is almost a dead animal. Should we upgrade to Java 8?

System test currently fails because they do not support 0.22.1's DiscoveryInfo.

Upgrade the master and slave images to Mesos 0.22.1

It should be possible to configure the client and transport port that is used for a framework. This way you can run multiple frameworks on a single slave, each using their own pairs of ports.

This is different than what is implemented in the https://github.com/mesos/elasticsearch/tree/docker-compose branch

Does this framework require the docker containers to run? If so will there be options to use and set local docker registries so we don't have to pre distribute the containers? If not will there be directions on how to get this setup as a native framework with no Docker containers? And also, if that's the case, will there be options to build without building the docker containers.

Thanks!

Waiting on Ferdi to contact the Apache Foundation.