I have a 3 node ensemble (zk1.dev, zk2.dev and zk3.dev) running on CentOS 6.2 using ZooKeeper 3.3.5+19.1 from Cloudera. Exhibitor is running on all three nodes and it successfully launched ZooKeeper on each node. Unfortunately, the Exhibitor Control Panel on each node shows the same thing - the local server is working, but it shows java.net.ConnectException: Connection timed out
for the remote nodes.
java -cp zookeeper-3.3.5-cdh3u4.jar:lib/log4j-1.2.15.jar:lib/jline-0.9.94.jar:conf org.apache.zookeeper.ZooKeeperMain -server zk3.dev:2181
Connecting to zk3.dev:2181
2012-07-27 10:03:49,404 [myid:] - INFO [main:Environment@97] - Client environment:zookeeper.version=3.3.5-cdh3u4--1, built on 05/07/2012 21:14 GMT
2012-07-27 10:03:49,413 [myid:] - INFO [main:Environment@97] - Client environment:host.name=zk1.dev
2012-07-27 10:03:49,414 [myid:] - INFO [main:Environment@97] - Client environment:java.version=1.6.0_24
2012-07-27 10:03:49,506 [myid:] - INFO [main:Environment@97] - Client environment:java.vendor=Sun Microsystems Inc.
2012-07-27 10:03:49,507 [myid:] - INFO [main:Environment@97] - Client environment:java.home=/usr/java/jdk1.6.0_24/jre
2012-07-27 10:03:49,507 [myid:] - INFO [main:Environment@97] - Client environment:java.class.path=zookeeper-3.3.5-cdh3u4.jar:lib/log4j-1.2.15.jar:lib/jline-0.9.94.jar:conf
2012-07-27 10:03:49,508 [myid:] - INFO [main:Environment@97] - Client environment:java.library.path=/usr/java/jdk1.6.0_24/jre/lib/amd64/server:/usr/java/jdk1.6.0_24/jre/lib/amd64:/usr/java/jdk1.6.0_24/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2012-07-27 10:03:49,508 [myid:] - INFO [main:Environment@97] - Client environment:java.io.tmpdir=/tmp
2012-07-27 10:03:49,509 [myid:] - INFO [main:Environment@97] - Client environment:java.compiler=<NA>
2012-07-27 10:03:49,509 [myid:] - INFO [main:Environment@97] - Client environment:os.name=Linux
2012-07-27 10:03:49,510 [myid:] - INFO [main:Environment@97] - Client environment:os.arch=amd64
2012-07-27 10:03:49,510 [myid:] - INFO [main:Environment@97] - Client environment:os.version=2.6.32-220.2.1.el6.x86_64
2012-07-27 10:03:49,511 [myid:] - INFO [main:Environment@97] - Client environment:user.name=root
2012-07-27 10:03:49,511 [myid:] - INFO [main:Environment@97] - Client environment:user.home=/root
2012-07-27 10:03:49,513 [myid:] - INFO [main:Environment@97] - Client environment:user.dir=/usr/lib/zookeeper
2012-07-27 10:03:49,515 [myid:] - INFO [main:ZooKeeper@379] - Initiating client connection, connectString=zk3.dev:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3343c8b3
2012-07-27 10:03:49,545 [myid:] - INFO [main-SendThread():ClientCnxn$SendThread@1058] - Opening socket connection to server zk3.dev/10.80.10.87:2181
Welcome to ZooKeeper!
2012-07-27 10:03:49,564 [myid:] - INFO [main-SendThread(zk3.dev:2181):ClientCnxn$SendThread@947] - Socket connection established to zk3.dev/10.80.10.87:2181, initiating session
JLine support is enabled
2012-07-27 10:03:49,642 [myid:] - INFO [main-SendThread(zk3.dev:2181):ClientCnxn$SendThread@736] - Session establishment complete on server zk3.dev/10.80.10.87:2181, sessionid = 0x238c8e3a08d0001, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: zk3.dev:2181(CONNECTED) 0]
#Auto-generated by Exhibitor zk1.dev
#Fri Jul 27 09:41:25 CDT 2012
com.netflix.exhibitor-rolling-hostnames=
com.netflix.exhibitor-rolling.zookeeper-data-directory=/var/lib/zookeeper
com.netflix.exhibitor-rolling.servers-spec=S\:0\:zk1.dev,S\:1\:zk2.dev,S\:2\:zk3.dev
com.netflix.exhibitor.java-environment=
com.netflix.exhibitor.zookeeper-data-directory=/var/lib/zookeeper
com.netflix.exhibitor-rolling-hostnames-index=0
com.netflix.exhibitor-rolling.java-environment=
com.netflix.exhibitor-rolling.observer-threshold=0
com.netflix.exhibitor.servers-spec=S\:0\:zk1.dev,S\:1\:zk2.dev,S\:2\:zk3.dev
com.netflix.exhibitor.cleanup-period-ms=60000
com.netflix.exhibitor.zookeeper-install-directory=/usr/lib/zookeeper
com.netflix.exhibitor.check-ms=5000
com.netflix.exhibitor-rolling.auto-manage-instances=1
com.netflix.exhibitor-rolling.cleanup-period-ms=60000
com.netflix.exhibitor-rolling.check-ms=5000
com.netflix.exhibitor.log-index-directory=/var/log/zookeeper
com.netflix.exhibitor.dead-instance-period-ms=30000
com.netflix.exhibitor-rolling.log-index-directory=/var/log/zookeeper
com.netflix.exhibitor.backup-period-ms=0
com.netflix.exhibitor-rolling.connect-port=2888
com.netflix.exhibitor-rolling.election-port=3888
com.netflix.exhibitor-rolling.backup-extra=
com.netflix.exhibitor.client-port=2181
com.netflix.exhibitor-rolling.zoo-cfg-extra=initLimit\=10&syncLimit\=5&tickTime\=2000
com.netflix.exhibitor-rolling.zookeeper-install-directory=/usr/lib/zookeeper
com.netflix.exhibitor.cleanup-max-files=5
com.netflix.exhibitor-rolling.backup-period-ms=0
com.netflix.exhibitor-rolling.client-port=2181
com.netflix.exhibitor.backup-max-store-ms=0
com.netflix.exhibitor-rolling.cleanup-max-files=5
com.netflix.exhibitor-rolling.backup-max-store-ms=0
com.netflix.exhibitor.connect-port=2888
com.netflix.exhibitor.log4j-properties=
com.netflix.exhibitor.observer-threshold=0
com.netflix.exhibitor.backup-extra=
com.netflix.exhibitor.election-port=3888
com.netflix.exhibitor.zoo-cfg-extra=initLimit\=10&syncLimit\=5&tickTime\=2000
com.netflix.exhibitor-rolling.log4j-properties=
com.netflix.exhibitor.auto-manage-instances=1
com.netflix.exhibitor-rolling.dead-instance-period-ms=30000
No errors in the logs.