RabbitMQ not start about docker-unms HOT 13 CLOSED

Sorry for the late response, if this is still a problem, can you try to restart the container and post the contents of the compose log after the first couple of Waiting for RabbitMQ messages appear? The log probably contains RabbitMQs attempts to start (should be at the beginning of the log, before the Waiting for RabbitMQ messages appear).

You can also try to rename the /config/rabbitmq directory to something else like rabbitmq_old, restart the container and then see if RabbitMQ starts.

from docker-unms.

Hi @Nico640 ,

issue already solved, I had to wait a couple of hours and all was started.

Thanks for response.

from docker-unms.

Hi @Nico640 ,
I reopen the issue because the start of the container take too long (about 30-40 min) also if I start from scratch.

In the log I have this

unms | 2023/11/13 18:13:07 [error] 829#829: *1399 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.122, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms."
unms | 2023/11/13 18:13:08 [error] 829#829: *1401 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.121, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms."
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:09+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:12+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:15+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:18+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:21+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:25+01:00","severity":"INFO","level":200}
unms | 2023/11/13 18:13:26 [error] 829#829: *1403 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.122, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms."
unms | 2023/11/13 18:13:27 [error] 829#829: *1405 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.121, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms."
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:28+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:31+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:34+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:37+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:40+01:00","severity":"INFO","level":200}

from docker-unms.

Hmm, weird, RabbitMQ is definitely not supposed to take that long to start. Can you upload / post a full log after you start seeing the "Waiting for RabbitMQ" message looping in the log?

Directly after starting the container, RabbitMQ will try to start and log some messages that could tell us why it isn't starting, so it should be pretty early in the log.

from docker-unms.

Hi @Nico640 ,
here the full log

unms | s6-rc: info: service s6rc-oneshot-runner: starting
unms | s6-rc: info: service s6rc-oneshot-runner successfully started
unms | s6-rc: info: service fix-attrs: starting
unms | s6-rc: info: service fix-attrs successfully started
unms | s6-rc: info: service legacy-cont-init: starting
unms | cont-init: info: running /etc/cont-init.d/10-set-timezone
unms | cont-init: info: /etc/cont-init.d/10-set-timezone exited 0
unms | s6-rc: info: service legacy-cont-init successfully started
unms | s6-rc: info: service init-adduser: starting
unms |
unms | -------------------------------------
unms | GID/UID
unms | -------------------------------------
unms | User uid: 911
unms | User gid: 911
unms | -------------------------------------
unms |
unms | s6-rc: info: service init-adduser successfully started
unms | s6-rc: info: service init-prepare: starting
unms | s6-rc: info: service init-prepare successfully started
unms | s6-rc: info: service init-permissions: starting
unms | s6-rc: info: service init-permissions successfully started
unms | s6-rc: info: service init-postgres: starting
unms | s6-rc: info: service siridb: starting
unms | s6-rc: info: service rabbitmq: starting
unms | s6-rc: info: service nginx: starting
unms | s6-rc: info: service siridb successfully started
unms | s6-rc: info: service rabbitmq successfully started
unms | s6-rc: info: service nginx successfully started
unms | Starting siridb-server...
unms | Starting nginx...
unms | Starting rabbitmq-server...
unms | Database already configured
unms | s6-rc: info: service init-postgres successfully started
unms | Running entrypoint.sh
unms | s6-rc: info: service postgres: starting
unms | s6-rc: info: service postgres successfully started
unms | s6-rc: info: service unms: starting
unms | s6-rc: info: service ucrm: starting
unms | s6-rc: info: service netflow: starting
unms | Starting postgres...
unms | [W 2023-11-14 18:43:46] Update server 'c40d4f67c853:9010' to '72b03ab87af5:9010'
unms | Creating nginx configuration
unms | Waiting for rabbitmq to start...
unms | s6-rc: info: service unms successfully started
unms | s6-rc: info: service ucrm successfully started
unms | s6-rc: info: service netflow successfully started
unms | s6-rc: info: service legacy-services: starting
unms | /run/postgresql:5432 - no response
unms | Waiting for postgres to come up...
unms | /run/postgresql:5432 - no response
unms | Waiting for postgres to come up...
unms | s6-rc: info: service legacy-services successfully started
unms | Enabling UNMS https connections on port 443
unms | Enabling UNMS wss connections on port 8444
unms | 2023-11-14 19:43:46.774 CET [191] LOG: starting PostgreSQL 13.12 on x86_64-alpine-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
unms | 2023-11-14 19:43:46.774 CET [191] LOG: listening on IPv4 address "127.0.0.1", port 5432
unms | 2023-11-14 19:43:46.775 CET [191] LOG: could not bind IPv6 address "::1": Address not available
unms | 2023-11-14 19:43:46.775 CET [191] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
unms | 2023-11-14 19:43:46.777 CET [191] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
unms | 2023-11-14 19:43:46.780 CET [191] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
unms | Will use existing SSL certificate
unms | 2023-11-14 19:43:46.785 CET [349] LOG: database system was shut down at 2023-11-14 17:17:30 CET
unms | Entrypoint finished
unms | Calling exec
unms | 2023-11-14 19:43:46.799 CET [191] LOG: database system is ready to accept connections
unms | 2023-11-14 19:43:49.861 CET [408] FATAL: role "root" does not exist
unms | 2023-11-14 19:43:49.866 CET [409] FATAL: role "root" does not exist
unms | /run/postgresql:5432 - accepting connections
unms | /run/postgresql:5432 - accepting connections
unms | /usr/src/ucrm/scripts/init_log.sh
unms | /usr/src/ucrm/scripts/dirs.sh
unms | {"message":"Creating directories.","channel":"dirs.sh","datetime":"2023-11-14T19:43:49+01:00","severity":"INFO","level":200}
unms | {"message":"Creating /var/log/nginx.","channel":"dirs.sh","datetime":"2023-11-14T19:43:50+01:00","severity":"INFO","level":200}
unms | {"message":"Creating /run/nginx.","channel":"dirs.sh","datetime":"2023-11-14T19:43:50+01:00","severity":"INFO","level":200}
unms | {"message":"Creating /etc/nginx/enabled-servers.","channel":"dirs.sh","datetime":"2023-11-14T19:43:50+01:00","severity":"INFO","level":200}
unms | Running docker-entrypoint /home/app/unms/index.js
unms | Sentry release:
unms | Version: 2.3.57+a484f0ef73.2023-10-05T18:06:06+02:00
unms | Waiting for database containers
unms | 2023/11/14 19:43:51 [error] 398#398: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.180.238, server: , request: "GET /nms/ HTTP/1.1", upstream: "http://127.0.0.1:8081/nms/", host: "unms.xxx.xx"
unms | {"message":"Creating /tmp/plugins.","channel":"dirs.sh","datetime":"2023-11-14T19:43:51+01:00","severity":"INFO","level":200}
unms | {"message":"Done creating directories.","channel":"dirs.sh","datetime":"2023-11-14T19:44:00+01:00","severity":"INFO","level":200}
unms | {"message":"Creating symbolic links.","channel":"dirs.sh","datetime":"2023-11-14T19:44:01+01:00","severity":"INFO","level":200}
unms | {"message":"Creating symbolic link for /usr/src/ucrm/app/data.","channel":"dirs.sh","datetime":"2023-11-14T19:44:01+01:00","severity":"INFO","level":200}
unms | {"message":"Creating symbolic link for /usr/src/ucrm/app/EmailQueue.","channel":"dirs.sh","datetime":"2023-11-14T19:44:02+01:00","severity":"INFO","level":200}
unms | {"message":"Creating symbolic link for /usr/src/ucrm/app/logs.","channel":"dirs.sh","datetime":"2023-11-14T19:44:02+01:00","severity":"INFO","level":200}
unms | {"message":"Creating symbolic link for /usr/src/ucrm/web/uploads.","channel":"dirs.sh","datetime":"2023-11-14T19:44:02+01:00","severity":"INFO","level":200}
unms | {"message":"Creating symbolic links for /data/updates.","channel":"dirs.sh","datetime":"2023-11-14T19:44:03+01:00","severity":"INFO","level":200}
unms | {"message":"Publishing current /usr/src/ucrm/app/config/version.yml.","channel":"dirs.sh","datetime":"2023-11-14T19:44:04+01:00","severity":"INFO","level":200}
unms | {"message":"Done creating symbolic links.","channel":"dirs.sh","datetime":"2023-11-14T19:44:04+01:00","severity":"INFO","level":200}
unms | /usr/src/ucrm/scripts/parameters.sh
unms | {"message":"Replacing configuration parameters.","channel":"parameters.sh","datetime":"2023-11-14T19:44:04+01:00","severity":"INFO","level":200}
unms | {"message":"Done replacing configuration parameters.","channel":"parameters.sh","datetime":"2023-11-14T19:44:26+01:00","severity":"INFO","level":200}
unms | /usr/src/ucrm/scripts/cron_jobs_disable.sh
unms | crontab /tmp/crontabs/server
unms | su-exec unms /usr/src/ucrm/scripts/database_ready.sh
unms | {"message":"Waiting for database.","channel":"database_ready.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200}
unms | /usr/src/ucrm/scripts/database_ready.sh: line 7: /tmp/UCRM_init.log: Permission denied
unms | {"message":"Database ready.","channel":"database_ready.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200}
unms | su-exec unms /usr/src/ucrm/scripts/database_create_extensions.sh
unms | {"message":"Creating database extensions.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200}
unms | {"message":"Extension "citext" exists.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200}
unms | {"message":"Extension "unaccent" exists.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200}
unms | {"message":"Extension "uuid-ossp" exists.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200}
unms | {"message":"Finished creating database extensions.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200}
unms | /usr/src/ucrm/scripts/backup_control_do.sh
unms | su-exec unms /usr/src/ucrm/scripts/verify_database_migrations_integrity.sh
unms | {"message":"Verifying database migrations integrity.","channel":"verify_database_migrations_integrity.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200}
unms | {"message":"Database migrations integrity verified.","channel":"verify_database_migrations_integrity.sh","datetime":"2023-11-14T19:44:30+01:00","severity":"INFO","level":200}
unms | su-exec unms /usr/src/ucrm/scripts/migrate.sh
unms | {"message":"Start replace shared views.","channel":"migrate.sh","datetime":"2023-11-14T19:44:30+01:00","severity":"INFO","level":200}
unms | {"message":"Done replace shared views.","channel":"migrate.sh","datetime":"2023-11-14T19:44:31+01:00","severity":"INFO","level":200}
unms | {"message":"Start database migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:31+01:00","severity":"INFO","level":200}
unms | {"message":"Done database migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200}
unms | {"message":"Start shared views migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200}
unms | {"message":"Done shared views migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200}
unms | {"message":"Start notification templates migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200}
unms | {"message":"Done notification templates migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:35+01:00","severity":"INFO","level":200}
unms | {"message":"Bumping UNMS version from ENV variable to database.","channel":"migrate.sh","datetime":"2023-11-14T19:44:35+01:00","severity":"INFO","level":200}
unms | {"message":"Done bumping UNMS version from ENV variable to database.","channel":"migrate.sh","datetime":"2023-11-14T19:44:36+01:00","severity":"INFO","level":200}
unms | su-exec unms /usr/src/ucrm/scripts/database_migrations_ready.sh
unms | {"message":"Waiting for migrations.","channel":"database_migrations_ready.sh","datetime":"2023-11-14T19:44:36+01:00","severity":"INFO","level":200}
unms | /usr/src/ucrm/scripts/database_migrations_ready.sh: line 7: /tmp/UCRM_init.log: Permission denied
unms | {"message":"Migrations ready.","channel":"database_migrations_ready.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200}
unms | /usr/src/ucrm/scripts/web.sh
unms | {"message":"Preparing server configuration.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200}
unms | cp: cannot stat '/etc/nginx/available-servers/ucrm.conf': No such file or directory
unms | {"message":"Done preparing server configuration.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200}
unms | {"message":"Updating environment variables.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200}
unms | {"message":"Done updating environment variables.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200}
unms | {"message":"Updating UCRM version.","channel":"web.sh","datetime":"2023-11-14T19:44:38+01:00","severity":"INFO","level":200}
unms | {"message":"Done UCRM version.","channel":"web.sh","datetime":"2023-11-14T19:44:38+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:38+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:42+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:45+01:00","severity":"INFO","level":200}
unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:48+01:00","severity":"INFO","level":200}

from docker-unms.

RabbitMQs doesn't seem to start at all, can you try to start it manually by opening a shell in the container and running rabbitmq-server? Also, check if there are any RabbitMQ logs in /var/log/rabbitmq/

from docker-unms.

I running the command but logs are the same,
No log in /var/log/rabbitmq/

from docker-unms.

The command should output something in the console, not in the logs. Does anything happen after manually running rabbitmq-server? Does it hang or immediately return to the shell?

By the way, on what system are you running the container? Which OS / distro and which Docker version? Is there anything else noteworthy about your setup (e.g. is it running on a virtual machine (which virtualization platform?) or bare metal?). I haven't been able to reproduce this yet.

from docker-unms.

it doesn't appear nothing on shell, still blank. Not quit the process.

I am on RockyLinux 9 last updates with docker in VM

from docker-unms.

Subscribing to this topic.

I've had this same issue months ago and never managed to get it to work due to the endless loop of "Waiting for RabbitMQ" messages.

Today I figured I'd give it another attempt, but turns out it still doesn't work for me.

When I try to start rabbitmq-server from the shell it "hangs", there's no output or what so ever.
The var/log/rabbitmq directory is empty as well.

I'm also running this in a VM, different distro though. I'm running on Fedora 38.

from docker-unms.

I have the same issue. I tried to start from scratch.
I am running it in a Promox VM with NixOS as a guest system.

from docker-unms.

Should be fixed in the latest image.

from docker-unms.

I'm seeing this same issue, tested on M1 Mac and also Ubuntu 23 with AMD64. tried latest and 2.3.57.1

Just keeps looping on:
Setting up the Rabbit MQ fabric

and I see the following errors:
FATAL: role "root" does not exist

/usr/src/ucrm/scripts/database_migrations_ready.sh: line 7: /tmp/UCRM_init.log: Permission denied

cp: cannot stat '/etc/nginx/available-servers/ucrm.conf': No such file or directory

from docker-unms.