Just noticed that UNMS wasn't working and was throwing the following errors in the log:
[cont-init.d] 10-adduser: exited 0.,
[cont-init.d] 20-set-timezone: executing... ,
[cont-init.d] 20-set-timezone: exited 0.,
[cont-init.d] 40-prepare: executing... ,
ln: failed to create symbolic link '/www/firmwares/firmwares': File exists,
ln: failed to create symbolic link '/cert/cert': File exists,
ln: failed to create symbolic link '/usercert/usercert': File exists,
[cont-init.d] 40-prepare: exited 0.,
[cont-init.d] 50-postgres: executing... ,
Database already configured,
[cont-init.d] 50-postgres: exited 0.,
[cont-init.d] done.,
[services.d] starting services,
Starting postgres...,
Waiting for rabbitmq to start...,
Starting redis...,
Starting rabbitmq-server...,
[services.d] done.,
Starting siridb-server...,
. ,
.-__ ''-._ , _.-
.
. ''-._ Redis 3.2.6 (00000000/0) 64 bit,
.- .-```. ```\/ _.,_ ''-._ , ( ' , .-` | `, ) Running in standalone mode, |`-._`-...-` __...-.
-.|'_.-'| Port: 6379, |
-. ._ / _.-' | PID: 370,
-._ -._
-./ .-' .-' ,
|-._
-. -.__.-' _.-'_.-'| , |
-.-._ _.-'_.-' | http://redis.io ,
-._ -._
-..-'.-' .-' ,
|-._
-. -.__.-' _.-'_.-'| , |
-.-._ _.-'_.-' | ,
-._ -._
-..-'_.-' _.-' ,
-._
-..-' _.-' ,
-._ _.-' ,
-..-' ,
,
370:M 03 Dec 10:44:37.282 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.,
370:M 03 Dec 10:44:37.282 # Server started, Redis version 3.2.6,
370:M 03 Dec 10:44:37.282 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.,
Starting nginx...,
370:M 03 Dec 10:44:37.285 * DB loaded from append only file: 0.002 seconds,
370:M 03 Dec 10:44:37.285 * The server is now ready to accept connections on port 6379,
[W 2020-12-03 10:44:37] We want to set a max-open-files value which exceeds 50% of the current hard limit.,
,
We will use 32000 as max_open_files for now.,
Please increase the hard-limit using:,
ulimit -Hn 65535,
Running entrypoint.sh,
LOG: could not bind IPv6 socket: Cannot assign requested address,
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.,
Will use existing SSL certificate,
[E 2020-12-03 10:44:37] Unexpected Series ID 1606348805 is found in shard 1606348800 (/config/siridb/unms/shards/1606348800.sdb) at position 38. This indicates that this shard is probably corrupt. The next optimize cycle will most likely fix this shard but you might loose some data.,
[C 2020-12-03 10:44:37] Index and/or shard corrupt: '/config/siridb/unms/shards/1606348802.sdb',
[E 2020-12-03 10:44:37] Error while loading shard: '1606348802.sdb',
[E 2020-12-03 10:44:37] Could not read shards for database 'unms',
[E 2020-12-03 10:44:37] Could not load 'unms'.,
Entrypoint finished,
Calling exec ,
LOG: database system was shut down at 2020-12-03 10:44:32 EST,
LOG: MultiXact member wraparound protections are now enabled,
LOG: database system is ready to accept connections,
LOG: autovacuum launcher started,
FATAL: role "root" does not exist,
/var/run/postgresql:5432 - accepting connections,
FATAL: role "root" does not exist,
/var/run/postgresql:5432 - accepting connections,
1000,
2020/12/03 10:44:37 [alert] 419#419: detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html),
nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html),
/usr/src/ucrm/scripts/init_log.sh,
/usr/src/ucrm/scripts/dirs.sh,
{"message":"Creating directories.","channel":"dirs.sh","datetime":"2020-12-03T10:44:37-05:00","severity":"INFO","level":200},
Running docker-entrypoint npm start,
Sentry release: ,
Version: 1.2.7+258cbb8a49.2020-09-18T14:42:15+02:00,
Waiting for database containers,
LOG: incomplete startup packet,
NOTICE: extension "timescaledb" does not exist, skipping,
DROP EXTENSION,
{"message":"Done creating directories.","channel":"dirs.sh","datetime":"2020-12-03T10:44:38-05:00","severity":"INFO","level":200},
{"message":"Creating symbolic links.","channel":"dirs.sh","datetime":"2020-12-03T10:44:38-05:00","severity":"INFO","level":200},
2020/12/03 10:44:38 [error] 461#461: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "192.168.0.55",
{"message":"Creating symbolic link for /usr/src/ucrm/app/data.","channel":"dirs.sh","datetime":"2020-12-03T10:44:38-05:00","severity":"INFO","level":200},
{"message":"Creating symbolic link for /usr/src/ucrm/app/EmailQueue.","channel":"dirs.sh","datetime":"2020-12-03T10:44:39-05:00","severity":"INFO","level":200},
Waiting for rabbit@unms ...,
pid is 404 ...,
{"message":"Creating symbolic link for /usr/src/ucrm/app/logs.","channel":"dirs.sh","datetime":"2020-12-03T10:44:39-05:00","severity":"INFO","level":200},
{"message":"Creating symbolic link for /usr/src/ucrm/web/uploads.","channel":"dirs.sh","datetime":"2020-12-03T10:44:39-05:00","severity":"INFO","level":200},
{"message":"Creating symbolic links for /data/updates.","channel":"dirs.sh","datetime":"2020-12-03T10:44:39-05:00","severity":"INFO","level":200},
{"message":"Publishing current /usr/src/ucrm/app/config/version.yml.","channel":"dirs.sh","datetime":"2020-12-03T10:44:39-05:00","severity":"INFO","level":200},
{"message":"Done creating symbolic links.","channel":"dirs.sh","datetime":"2020-12-03T10:44:40-05:00","severity":"INFO","level":200},
/usr/src/ucrm/scripts/parameters.sh,
{"message":"Replacing configuration parameters.","channel":"parameters.sh","datetime":"2020-12-03T10:44:40-05:00","severity":"INFO","level":200},
,
RabbitMQ 3.6.6. Copyright (C) 2007-2016 Pivotal Software, Inc.,
##,
########## Logs: /var/log/rabbitmq/[email protected],
##########,
Starting broker...,
completed with 0 plugins.,
Starting unms-netflow...,
370:M 03 Dec 10:44:43.375 * Background append only file rewriting started by pid 1118,
Background append only file rewriting started,
Restoring backups and/or running migrations,
370:M 03 Dec 10:44:43.519 * AOF rewrite child asks to stop sending diffs.,
1118:C 03 Dec 10:44:43.519 * Parent agreed to stop sending diffs. Finalizing AOF...,
1118:C 03 Dec 10:44:43.519 * Concatenating 0.00 MB of AOF diff received from parent.,
1118:C 03 Dec 10:44:43.520 * SYNC append only file rewrite performed,
1118:C 03 Dec 10:44:43.520 * AOF rewrite: 0 MB of memory used by copy-on-write,
370:M 03 Dec 10:44:43.546 * Background AOF rewrite terminated with success,
370:M 03 Dec 10:44:43.546 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB),
370:M 03 Dec 10:44:43.546 * Background AOF rewrite finished successfully,
yarn run v1.21.1,
$ yarn backup:apply && yarn migrate && yarn check-crm-db,
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Listening for netflow packets on port 2055 (9 ms)","time":"2020-12-03T15:44:44.383Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Initializing database","time":"2020-12-03T15:44:44.391Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Database initialized (51 ms)","time":"2020-12-03T15:44:44.442Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Initializing RabbitMQ","time":"2020-12-03T15:44:44.442Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"RabbitMQ initialized (115 ms)","time":"2020-12-03T15:44:44.557Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for managed networks change notifications at queue 'settings.networks.changed'","time":"2020-12-03T15:44:44.578Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for managed networks change notifications at queue 'settings.blacklist.changed'","time":"2020-12-03T15:44:44.592Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for site IP addresses change notifications at queue 'site.ips.changed'","time":"2020-12-03T15:44:44.603Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for device authorizations at queue 'device..authorized'","time":"2020-12-03T15:44:44.614Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for device authorizations at queue 'device..settings.changed'","time":"2020-12-03T15:44:44.625Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for settings change notifications at queue 'settings.changed'","time":"2020-12-03T15:44:44.636Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Subscribed for log verbosity change notifications at queue 'server.log.verbosity'","time":"2020-12-03T15:44:44.647Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Loaded list of managed networks from database (10 ms) [ '192.168.0.0/24', '10.0.0.0/24', '11.0.0.0/24' ]","time":"2020-12-03T15:44:44.657Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Loaded blacklist from database (3 ms) []","time":"2020-12-03T15:44:44.662Z","v":0},
{"name":"netflow","hostname":"unms","pid":1106,"level":30,"msg":"Determined UNMS IP address '192.168.0.55' (5 ms)","time":"2020-12-03T15:44:44.679Z","v":0},
$ node ./cli/apply-backup.js,
$ node ./cli/migrate.js,
{"message":"Done replacing configuration parameters.","channel":"parameters.sh","datetime":"2020-12-03T10:44:47-05:00","severity":"INFO","level":200},
/usr/src/ucrm/scripts/cron_jobs_disable.sh,
crontab /tmp/crontabs/server,
/usr/src/ucrm/scripts/database_ready.sh,
{"message":"Waiting for database.","channel":"database_ready.sh","datetime":"2020-12-03T10:44:47-05:00","severity":"INFO","level":200},
{"message":"Database ready.","channel":"database_ready.sh","datetime":"2020-12-03T10:44:48-05:00","severity":"INFO","level":200},
/usr/src/ucrm/scripts/database_create_extensions.sh,
{"message":"Creating database extensions.","channel":"database_create_extensions.sh","datetime":"2020-12-03T10:44:48-05:00","severity":"INFO","level":200},
{"message":"Extension "citext" exists.","channel":"database_create_extensions.sh","datetime":"2020-12-03T10:44:48-05:00","severity":"INFO","level":200},
{"message":"Extension "unaccent" exists.","channel":"database_create_extensions.sh","datetime":"2020-12-03T10:44:49-05:00","severity":"INFO","level":200},
Extension 'uuid-ossp:1.1 installed.,
Extension 'pgcrypto:1.3 installed.,
Extension 'cube:1.2 installed.,
Extension 'earthdistance:1.1 installed.,
Extension 'unaccent:1.1 installed.,
Migrations finished in 0.096s,
Setting database schema version to 1.2.7,
Done,
{"message":"Extension "uuid-ossp" exists.","channel":"database_create_extensions.sh","datetime":"2020-12-03T10:44:49-05:00","severity":"INFO","level":200},
{"message":"Finished creating database extensions.","channel":"database_create_extensions.sh","datetime":"2020-12-03T10:44:49-05:00","severity":"INFO","level":200},
/usr/src/ucrm/scripts/migrate.sh,
{"message":"Start replace shared views.","channel":"migrate.sh","datetime":"2020-12-03T10:44:49-05:00","severity":"INFO","level":200},
$ node cli/check-crm-db.js,
{"message":"Done replace shared views.","channel":"migrate.sh","datetime":"2020-12-03T10:44:50-05:00","severity":"INFO","level":200},
{"message":"Start database migration.","channel":"migrate.sh","datetime":"2020-12-03T10:44:50-05:00","severity":"INFO","level":200},
Checking CRM database schema version.,
No data returned from the query.,
Retrying in 10s,
{"message":"Done database migration.","channel":"migrate.sh","datetime":"2020-12-03T10:44:51-05:00","severity":"INFO","level":200},
{"message":"Start shared views migration.","channel":"migrate.sh","datetime":"2020-12-03T10:44:51-05:00","severity":"INFO","level":200},
{"message":"Done shared views migration.","channel":"migrate.sh","datetime":"2020-12-03T10:44:52-05:00","severity":"INFO","level":200},
{"message":"Start notification templates migration.","channel":"migrate.sh","datetime":"2020-12-03T10:44:52-05:00","severity":"INFO","level":200},
{"message":"Done notification templates migration.","channel":"migrate.sh","datetime":"2020-12-03T10:44:53-05:00","severity":"INFO","level":200},
{"message":"Bumping UNMS version from ENV variable to database.","channel":"migrate.sh","datetime":"2020-12-03T10:44:53-05:00","severity":"INFO","level":200},
{"message":"Done bumping UNMS version from ENV variable to database.","channel":"migrate.sh","datetime":"2020-12-03T10:44:53-05:00","severity":"INFO","level":200},
/usr/src/ucrm/scripts/database_migrations_ready.sh,
{"message":"Waiting for migrations.","channel":"database_migrations_ready.sh","datetime":"2020-12-03T10:44:53-05:00","severity":"INFO","level":200},
{"message":"Migrations ready.","channel":"database_migrations_ready.sh","datetime":"2020-12-03T10:44:54-05:00","severity":"INFO","level":200},
/usr/src/ucrm/scripts/web.sh,
{"message":"Preparing server configuration.","channel":"web.sh","datetime":"2020-12-03T10:44:54-05:00","severity":"INFO","level":200},
cp: cannot stat '/etc/nginx/available-servers/ucrm.conf': No such file or directory,
{"message":"Done preparing server configuration.","channel":"web.sh","datetime":"2020-12-03T10:44:54-05:00","severity":"INFO","level":200},
{"message":"Updating environment variables.","channel":"web.sh","datetime":"2020-12-03T10:44:54-05:00","severity":"INFO","level":200},
{"message":"Done updating environment variables.","channel":"web.sh","datetime":"2020-12-03T10:44:55-05:00","severity":"INFO","level":200},
{"message":"Updating UCRM version.","channel":"web.sh","datetime":"2020-12-03T10:44:55-05:00","severity":"INFO","level":200},
{"message":"Done UCRM version.","channel":"web.sh","datetime":"2020-12-03T10:44:56-05:00","severity":"INFO","level":200},
Setting up the Rabbit MQ fabric,
{"message":"Building search index.","channel":"web.sh","datetime":"2020-12-03T10:44:57-05:00","severity":"INFO","level":200},
2020/12/03 10:44:58 [error] 461#461: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "192.168.0.55",
Checking CRM database schema version.,
CRM database schema version is '1.2.7'.,
Done in 17.49s.,
Exec npm start with memory limit 2048,
,
[email protected] prestart /home/app/unms,
yarn backup:apply && yarn migrate && yarn check-crm-db,
,
yarn run v1.21.1,
$ node ./cli/apply-backup.js,
{"message":"Generating static suspension page.","channel":"web.sh","datetime":"2020-12-03T10:45:05-05:00","severity":"INFO","level":200},
{"message":"Done generating static suspension page.","channel":"web.sh","datetime":"2020-12-03T10:45:06-05:00","severity":"INFO","level":200},
{"message":"Re/generating plugin symlinks and config files.","channel":"web.sh","datetime":"2020-12-03T10:45:06-05:00","severity":"INFO","level":200},
Done in 2.40s.,
yarn run v1.21.1,
$ node ./cli/migrate.js,
{"message":"Done re/generating plugin symlinks and config files.","channel":"web.sh","datetime":"2020-12-03T10:45:07-05:00","severity":"INFO","level":200},
{"message":"Resetting in-progress states.","channel":"web.sh","datetime":"2020-12-03T10:45:07-05:00","severity":"INFO","level":200},
Extension 'uuid-ossp:1.1 installed.,
Extension 'pgcrypto:1.3 installed.,
Extension 'cube:1.2 installed.,
Extension 'earthdistance:1.1 installed.,
Extension 'unaccent:1.1 installed.,
Migrations finished in 0.101s,
Setting database schema version to 1.2.7,
Done,
Done in 1.53s.,
{"message":"Done resetting in-progress states.","channel":"web.sh","datetime":"2020-12-03T10:45:09-05:00","severity":"INFO","level":200},
yarn run v1.21.1,
{"message":"Synchronizing with UNMS.","channel":"web.sh","datetime":"2020-12-03T10:45:09-05:00","severity":"INFO","level":200},
$ node cli/check-crm-db.js,
{"message":"Waiting for UNMS (127.0.0.1:8081).","channel":"unms_ready.sh","datetime":"2020-12-03T10:45:09-05:00","severity":"INFO","level":200},
Checking CRM database schema version.,
CRM database schema version is '1.2.7'.,
Done in 1.54s.,
,
[email protected] start /home/app/unms,
node --max_old_space_size=2048 index.js,
,
{"name":"UNMS","hostname":"unms","pid":2049,"level":30,"msg":"Master 2049 is running","time":"2020-12-03T15:45:12.013Z","v":0},
{"message":"Waiting for UNMS (127.0.0.1:8081).","channel":"unms_ready.sh","datetime":"2020-12-03T10:45:12-05:00","severity":"INFO","level":200},
{"name":"UNMS","hostname":"unms","pid":2049,"level":30,"msg":"Connected to SiriDB server version: 2.0.34","time":"2020-12-03T15:45:14.682Z","v":0},
{"name":"UNMS","hostname":"unms","pid":2049,"level":30,"msg":"SiriDB database 'unms' does not exists. Creating...","time":"2020-12-03T15:45:14.684Z","v":0},
[W 2020-12-03 10:45:14] Error handling manage request: database directory already exists: /config/siridb/unms/,
{ SiriDbError: database directory already exists: /config/siridb/unms/,
at SiriDbConnection.resolveRequest (/home/app/unms/lib/siridb/connection.js:155:17),
at SiriDbConnection.handleData (/home/app/unms/lib/siridb/connection.js:95:14),
at Socket.emit (events.js:198:13),
at Socket.EventEmitter.emit (domain.js:448:20),
at addChunk (_stream_readable.js:288:12),
at readableAddChunk (_stream_readable.js:269:11),
at Socket.Readable.push (_stream_readable.js:224:10),
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17) tp: 96, name: 'SiriDbError' } 'Unexpected error in master process',
npm ERR! code ELIFECYCLE,
npm ERR! errno 1,
npm ERR! [email protected] start: node --max_old_space_size=2048 index.js
,
npm ERR! Exit status 1,�,
,
npm ERR! Failed at the [email protected] start script.,
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.,
,
npm ERR! A complete log of this run can be found in:,
npm ERR! /home/app/.npm/_logs/2020-12-03T15_45_14_719Z-debug.log,
FATAL: role "root" does not exist,