Multi-host cluster, Apache Spark Cluster and Production Ready about hadoop-cluster-docker HOT 4 OPEN

1. Currently, this project does not support multiple physical nodes... I'm not sure how to implement it. The problem is Hadoop use SSH to commute, which will cause problem when containers run on different physical nodes.
2. Currently, this project runs HDFS in container, which means the data will be deleted when the container is deleted. So, to use it in production, you need to put HDFS on host node using "docker volume". In addition, you need to run multiple master to ensure availability. If you solve mentioned problems. I think it is OK to run Hadoop in container for production. Of course, you need to run more tests before using it.

from hadoop-cluster-docker.

Thanks @kiwenlau. For your first response, I think setting up docker swarm will work, however my team and I are yet to test this. Will update you when this is done.

We will also test docker volume and provide our feedback soon.

Thanks again

from hadoop-cluster-docker.

Any updated guys?. I would like to use HDFS on multi-host cluster i.e., docker running on each host machine and share data among themselves...

from hadoop-cluster-docker.

@kiwenlau I don't know why only with your docker code, the nodemanager in running in slaves. If i use your components of code in my docker file, it's not working(nodemanager in not running in slaves)...

from hadoop-cluster-docker.