multi-host hadoop cluster about hadoop-cluster-docker HOT 5 OPEN

My project is designed for single node, but it can be changed to multiple hosts:

1. Run Hadoop Container with "--net=host", so that containers will share the IP address with the host node
2. Use IP address for Hadoop communication.

from hadoop-cluster-docker.

Thanks Kai.
But the problem with this is, i cant ssh from hduser of a container in host1 to hduser of a container in host2, where hduser is a user created inside a container and the containers are set as --net=host.
As container and host are sharing the IP, it cannot differentiate whether the user we are trying to ssh belongs to host or a container.

from hadoop-cluster-docker.

Even we can't ssh from one container to another, as the host and the container are sharing the ip.

from hadoop-cluster-docker.

Sorry, I didn't expect this problem.

This problem from stackoverflow maybe helpful for you:

SSH into a docker container from another container on a different host

from hadoop-cluster-docker.

You could use the new swarm features build into docker 1.12-rc1
"Multi-host networking: You can specify an overlay network for your services. The swarm manager automatically assigns addresses to the containers on the overlay network when it initializes or updates the application."
https://docs.docker.com/engine/swarm/
Or you could use docker swarm which also allows you to create a software defined network.

from hadoop-cluster-docker.