Not recognizing containers about registrator HOT 34 CLOSED

Hmm, try -bootstrap-expect 1 ... pretty sure the docs were updated about
that?

On Sat, Sep 6, 2014 at 5:26 AM, jlovison [email protected] wrote:

Testing docker-consul and registrator out on a vagrant box (provisioned
with Ubuntu 14.04 and docker version 1.2.0), and it seems like registrator
simply keeps missing new containers.

Running the commands manually the exactly as in the README for both
projects (with the exception of the -d flag for docker-consul). And yet
firing up the redis container (or any other container for that matter),
totally bypasses registrator's notice (despite ports being opened).

docker-consul command:
docker run -d -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h node1
progrium/consul -server -bootstrap

registrator command:
docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h node1
progrium/registrator consul:

redis command:
docker run -d --name redis.0 -p 10000:6379 dockerfile/redis

The only logs I see from registrator no matter how often I try are it's
initialization log indicating the consul server, and the log saying it's
ignoring itself due to no ports.

lsof shows docker using the /var/run/docker.sock socket as well.

Using the exact same commands above, it is also failing to register
running locally on my laptop (with same OS version and docker version).

—
Reply to this email directly or view it on GitHub
https://github.com/progrium/registrator/issues/16.

Jeff Lindsay
http://progrium.com

from registrator.

Running the above commands with the following for the consul image makes no difference:

docker run -d -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h node1 progrium/consul -server -bootstrap-expect 1

In both cases the consul server starts fine and sets itself to leader.

The problem seems to sit within registrator somehow not recognizing the redis service as having been started or existing. Is it possible that the registrator detection through the go api is somehow not working with docker 1.2.0?

After a bit of research, I found an issue in jenkin's docker plugin (here) that referenced a possible api oddity over on the docker project post 1.1.1 (here).

from registrator.

Yes, this could be it. Have you tried with 1.1?

from registrator.

Hmmmm....unfortunately no dice. Downgraded to 1.1.1 as well as even farther to 1.0.1, and in both cases, registrator still fails to pick up on the added redis container.

from registrator.

So you're running registrator with consul: which is going to use localhost, but that's the container's localhost... you should try giving it consul://{machine ip}:8500 or attaching it to the consul container's network (-net container:<consul container id>)

from registrator.

It doesn't seem to be a networking issue. I tried both those suggestions, and still nothing. But in all cases, I'm never seeing any logs in registrator indicating failed attempts to register the service with consul. The only logs registrator ever generates (no matter how other services are brought up or down) is:

2014/09/06 21:27:23 registrator: Using consul registry backend at consul:192.168.1.101:8500
2014/09/06 21:27:23 registrator: ignored: 324aee1d0853 no published ports

Or in the case of the network attached to the consul container:

2014/09/06 21:29:26 registrator: Using consul registry backend at consul://
2014/09/06 21:29:26 registrator: ignored: cd077339e0cd no published ports

When I download the registrator binary in stage/ on this project, and run locally, I get the following logs immediately:

2014/09/06 17:34:25 registrator: Using consul registry backend at consul://
2014/09/06 17:34:25 registrator: added: ef68b4f209d8 localhost:redis.0:6379
2014/09/06 17:34:25 registrator: ignored: cd077339e0cd no published ports
2014/09/06 17:34:25 registrator: added: efd94552e3f3 localhost:furious_jones:8400
2014/09/06 17:34:25 registrator: added: efd94552e3f3 localhost:furious_jones:8500
2014/09/06 17:34:25 registrator: added: efd94552e3f3 localhost:furious_jones:53:udp
2014/09/06 17:34:25 registrator: Listening for Docker events...

So possibly there's an issue in the link between the volume mapping between the registrator container and the docker sock? I've checked and double-checked the command though, and it's definitely written exactly as above (-v /var/run/docker.sock:/tmp/docker.sock).

from registrator.

The fact it never gets to "Listening for Docker events" means it's getting blocked somewhere. My guess is it's getting blocked when trying to add the container. My guess as to why is again a connection to Consul ... but I'll go and see what else it could be...

In your first example can you show me the command? The logs imply you didn't include the // of the URL, but maybe that's an error in displaying it.

from registrator.

Ok, wow. No, you were totally correct. I'd not added the // in the case of the first command, and I tried it again adding those, and with a correct connection to the consul via the host interface, it worked correctly.

So using the command docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h node1 progrium/registrator consul://192.168.1.101:8500 worked splendidly.

My apologies. Since I didn't see any errors in the logs regarding the connection to the backend service, I'd thought there was something internal to the registrator service itself that was going haywire, when in fact it was just incorrectly set up.

from registrator.

Yeah, needs slightly better error handling for this case.

On Sat, Sep 6, 2014 at 5:07 PM, jlovison [email protected] wrote:

Ok, wow. No, you were totally correct. I'd not added the // in the case
of the first command, and I tried it again adding those, and with a correct
connection to the consul via the host interface, it worked correctly.

So using the command docker run -d -v
/var/run/docker.sock:/tmp/docker.sock -h node1 progrium/registrator
consul://192.168.1.101:8500 worked splendidly.

My apologies. Since I didn't see any errors in the logs regarding the
connection to the backend service, I'd thought there was something internal
to the registrator service itself that was going haywire, when in fact it
was just incorrectly set up.

—
Reply to this email directly or view it on GitHub
https://github.com/progrium/registrator/issues/16#issuecomment-54730065.

Jeff Lindsay
http://progrium.com

from registrator.

I have same issue where registrator not listening to docker events. i have used correct statement.

docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h node1 progrium/registrator consul://192.168.1.101:8500

from registrator.

Me too. I can see registrator is getting its own event and ignoring itself, but when I start and stop a redis container with -p 6379:6379 nothing happens.

core@core-02 ~ $ docker run -v /var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME progrium/registrator etcd://127.0.0.1:4001/services
2014/10/13 18:28:10 registrator: Using etcd registry backend at etcd://127.0.0.1:4001/services
2014/10/13 18:28:10 registrator: ignored: 47e4fcb197ec no published ports

<redis container started here but not 'seen'>

^Ccore@core-02 ~ $

from registrator.

Next release I'm going to be more verbose and make sure we're using latest client libraries, so hopefully we can get some insight into this soon.

from registrator.

Also seeing the same problem as @j0hnsmith is. I've made sure I've used explicit IP address, made sure I'm using double slashes in "consul://{ip address}:8500". Registrator sees only itself - it doesn't even find the running consul and link it's ports. When adding a redis container - registrator doesn't see any events.

This is on a CentOS 6.5 Minimal with docker 1.2, all running on VMware ESXi.

from registrator.

Same issue with centos7 and docker 1.2.0, don't get registrator: Listening for Docker events...

My advise stop iptables on centos6 or firewalld on centos7
If you can't stop add some firewall rules.

from registrator.

I had this issue with Ubuntu 14.04.1, and ufw enabled - it was interfering with docker container communication, between containers and the Docker bridge.

It's worth checking you can start a container and curl the Consul IP/port from it.

from registrator.

Hi,

same issue here with Ubuntu 12.04 and Docker 1.4.0 in VBox (Vagrant) and ufw disabled.

Update: have to correct me - there was an another network issue, runs now fine.

from registrator.

same issue - CentOS7 host, Docker 1.4.1
docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME progrium/registrator consul://10.142.205.231:8500 -ip
When I run another container with published port of 8080, it runs and I can see it start on the given port.
I can curl the consul remote uri from progrium container.
but when I tail the logs for registrator container I geet the "no published ports" message.

Would appreciate any thoughts.

Using remote consul cluster.

Docker Daemon has -H for fd:// and tcp://hostip:2375 could this be an issue?

from registrator.

Hello bryantidd,

maybe selinux is the issue or missing bind-utils package.

you can try this:

yum install bind-utils
systemctl stop firewalld
systemctl disable firewalld
systemctl restart docker
docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h registrator_node1 --name=registrator_node1 --privileged progrium/registrator consul://[ip-consul]:8500

from registrator.

Found that I was binding my subsequent docker containers to host that seemed to be causing the problem...by just publishing the docker internal port and not binding it to a specific host port, it seems to be working now.

from registrator.

I've got the same problem.

After starting the only log ist 2015/04/01 18:37:57 registrator: Using consul registry backend at consul://231.2.12.3:8500

But no containers are registered. How can I debug this?
Using Ubuntu 14.04.2 with docker 1.5.0 and using the latest gliderlabs/registrator image.

from registrator.

I am having a problem on Ubuntu 14.04. I am running Docker 1.5.0 and i'm using the gliderlabs/registrator:master image.

Here is my log:

2015/04/01 23:30:22 Starting registrator v5 ...
2015/04/01 23:30:22 consul: current leader  
2015/04/01 23:30:22 Using consul adapter: consul://IP:8500
2015/04/01 23:30:22 Listening for Docker events ...
2015/04/01 23:30:22 Syncing services on 3 containers
2015/04/01 23:30:22 ignored: bfe3e7f936e1 no published ports
2015/04/01 23:30:22 ignored: 0167c1121367 no published ports
2015/04/01 23:30:22 ignored: 5188ebfc6698 no published ports

from registrator.

I think you dont have public ports. Can you post your docker run command?

Regards
El abr 1, 2015 8:58 PM, "Alexandar Podobnik" [email protected]
escribió:

I am having a problem on Ubuntu 14.04. I am running Docker 1.5.0 and i'm
using the gliderlabs/registrator:master image.

Here is my log:

2015/04/01 23:30:22 Starting registrator v5 ...
2015/04/01 23:30:22 consul: current leader
2015/04/01 23:30:22 Using consul adapter: consul://IP:8500
2015/04/01 23:30:22 Listening for Docker events ...
2015/04/01 23:30:22 Syncing services on 3 containers
2015/04/01 23:30:22 ignored: bfe3e7f936e1 no published ports
2015/04/01 23:30:22 ignored: 0167c1121367 no published ports
2015/04/01 23:30:22 ignored: 5188ebfc6698 no published ports

—
Reply to this email directly or view it on GitHub
#16 (comment)
.

from registrator.

Here is my run command:

docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME gliderlabs/registrator:master consul://IP:8500

I appreciate the help.

from registrator.

And your service docker run?
El abr 1, 2015 9:08 PM, "Alexandar Podobnik" [email protected]
escribió:

Here is my run command:

docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME gliderlabs/registrator:master consul://IP:8500

I appreciate the help.

—
Reply to this email directly or view it on GitHub
#16 (comment)
.

from registrator.

@ch3lo what do you mean by service docker run?

from registrator.

I want yo view if the other services are running with -p or -P parameters
El abr 1, 2015 9:16 PM, "Alexandar Podobnik" [email protected]
escribió:

@ch3lo https://github.com/ch3lo what do you mean by service docker run?

—
Reply to this email directly or view it on GitHub
#16 (comment)
.

from registrator.

@ch3lo I ended up fixing it by assigning different ports. The containers now register, Thank you!

from registrator.

Anybody (maybe @ch3lo?) knows what could be wrong when I just see (my only log statement is)

2015/04/01 23:30:22 Using consul adapter: consul://IP:8500

and not the surrounding console logs

2015/04/01 23:30:22 Starting registrator v5 ...
2015/04/01 23:30:22 consul: current leader  
2015/04/01 23:30:22 Using consul adapter: consul://IP:8500
2015/04/01 23:30:22 Listening for Docker events ...

from registrator.

@lukaswelte I am running a newer image, try pulling and running gliderlabs/registrator:master.

from registrator.

@alexandarp I have also tried the master image.
No difference..

from registrator.

@lukaswelte it is hard to tell if your issue is the same as the original poster. Can you please open a new issue or chat with us in #gliderlabs to figure out the issue?

from registrator.

Figured it out.. had some problems with UFW rejecting the ip address coming from docker.
Would be quite awesome to see that connections fails and not just no output.

Thanks all for the help! 👍

from registrator.

Faced same issue, but overcame it. Maybe the readme needs to updated to ADD "//" explicitly. ;)

from registrator.

The master branch now has better logging and should cause the container to exit if there was an error connecting.

from registrator.