For me, the container just stops doing anything randomly at some point. Does anybody else have this and/or knows a solution?
Right now, I've had it running for seven days and five days ago it just stopped logging anything stdout and service.log.0
, and it also stopped doing anything backup-wise. The main process is still alive, so Docker hasn't restarted the container. It's just not doing anything.
It had updated to 4.6.0 a couple days a while earlier (long enough before the freeze that I don't suspect a connection).
I don't find anything suspicious in the log files. There's a lot of stacktraces (IllegalArgumentException: Peer already existed, cannot add
), but they always occur every minute, even while it's running well. Here's the very last lines before it just went silent.
[04.03.16 10:24:43.127 WARN on-825222751 2.messaging.peer.NATContinuation] [NATContinuation[736404681217745414](0):733556053290072213<->733351749112369151][active/A_INV_SENT] INVITE timeout, STOP
[04.03.16 10:24:43.329 WARN on-825222751 de42.messaging.peer.NATConnector] [NATContinuation[736404681217745414](0):733556053290072213<->733351749112369151][active/STOP] RP:: TRAVERSAL FAILURE for com.code42.messaging.peer.NATConnector@7c3ce6ef
[04.03.16 10:24:54.915 INFO ystemWatcher ode42.messaging.nio.MessageQueue] MW:: STATS:UI : numWorkers=2, queue.size=0, MessageQueueStats[num=0, numSinceLast=0, handling=Counter[count = 0, rate = 0.00, rollingRate = 0.00], waiting=Counter[count = 0, rate = 0.00, rollingRate = 0.00], workload=1.000]
[04.03.16 10:25:36.945 INFO DefaultGroup .code42.messaging.peer.PeerGroup] PG::DefaultGroup DONE Managing connected remote peers. numConnected=7, numFailedConnectedCheck=0, duration(ms)=35
[04.03.16 10:25:39.300 INFO erTimeoutWrk e42.messaging.peer.PeerConnector] PC:: Cancelling connection attempt due to timeout - pending connection=PendingConnection[timeout(ms) = 10000, startTime = Sun Apr 03 10:25:25 GMT 2016, remotePeer = RemotePeer-[guid=733351749112369151, state=CONNECTING, mode=PRIVATE, location=192.168.178.56:4242, public=79.247.139.110:0, transportPbK=X509.checksum(dfb2987cb569a2e55152753fa25ab089), transportPbKRequestable=false, connecting=2016-04-03T10:25:25:139, connected=0, disconnected=2016-04-03T10:24:43:329, attempts=2, connectActivity=2016-04-03T10:25:25:139, keepAliveSent=0, minRetry=29648, retryDelay=0, reflector=na, #nat=1, session=null], sessionData = MessageConnection[ sessionInfo=Session[ null], readBuffer=com.code42.messaging.nio.MessageConnection$MessageBuffer@38c687d0, maxBufferSize = 65536, currentSize = 0, enqueued = false, #items = 0, Connection[481292855, state=0, context = Context@231989070[null->nullconnectionState=0], SendBufferQueue[closed=false, queueSize=0, maxSize=1310720, enqueueTimeout=60000, numItems=0, waitingCount=0, waitingPriorityCount=0, items.size()=0, priorityItems.size()=0, lastEnqueueTS=0]]]]
[04.03.16 10:25:49.442 INFO erTimeoutWrk e42.messaging.peer.PeerConnector] PC:: Cancelling connection attempt due to timeout - pending connection=PendingConnection[timeout(ms) = 5000, startTime = Sun Apr 03 10:25:39 GMT 2016, remotePeer = RemotePeer-[guid=733351749112369151, state=CONNECTING, mode=PRIVATE, location=192.168.178.56:4242, public=79.247.139.110:0, transportPbK=X509.checksum(dfb2987cb569a2e55152753fa25ab089), transportPbKRequestable=false, connecting=2016-04-03T10:25:39:430, connected=0, disconnected=2016-04-03T10:25:39:427, attempts=1, connectActivity=2016-04-03T10:25:39:430, keepAliveSent=0, minRetry=29648, retryDelay=0, reflector=na, #nat=1, session=null], sessionData = MessageConnection[ sessionInfo=Session[ null], readBuffer=com.code42.messaging.nio.MessageConnection$MessageBuffer@6137fca8, maxBufferSize = 65536, currentSize = 0, enqueued = false, #items = 0, Connection[1841504089, state=0, context = Context@245266076[null->nullconnectionState=0], SendBufferQueue[closed=false, queueSize=0, maxSize=1310720, enqueueTimeout=60000, numItems=0, waitingCount=0, waitingPriorityCount=0, items.size()=0, priorityItems.size()=0, lastEnqueueTS=0]]]]
[04.03.16 10:25:50.725 WARN MQ-Peer-1 2.messaging.MessageReceiverProxy] MessageReceiverProxy: InvocationTargetException receiving message=type=com.code42.peer.message.NATInviteMessage, contents=NATInviteMessage[736326220872622598] [[email protected]:53061], session=Session[id=736326220872622598, closed=false, isAcceptor=false, remoteIdentity=ENDPOINT, completedAuth=true, lat=2016-04-03T10:25:50:716, lrt=2016-04-03T10:25:50:716, lwt=2016-04-03T10:25:36:934, #pending=0, enqueued=true, local=172.17.0.7:50856, remote=216.17.8.3:443, usingProtoHeaders=true, usingEncryptedHeaders=true, WAN], e=java.lang.reflect.InvocationTargetException, cause=java.lang.IllegalArgumentException: Peer already existed, cannot add. PeerLocation=[[[email protected]:53061]], java.lang.IllegalArgumentException: Peer already existed, cannot add. PeerLocation=[[[email protected]:53061]]
STACKTRACE:: java.lang.IllegalArgumentException: Peer already existed, cannot add. PeerLocation=[[[email protected]:53061]]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at com.code42.messaging.peer.PeerGroup.addPeerFromLocation(PeerGroup.java:683)
at com.code42.messaging.peer.NATAgent.receiveMessage(NATAgent.java:94)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.code42.messaging.MessageReceiverProxy.receiveMessage(MessageReceiverProxy.java:149)
at com.code42.messaging.Session.receiveMessage(Session.java:379)
at com.code42.messaging.nio.MessageQueue$MessageWorker.handle(MessageQueue.java:161)
at com.code42.messaging.nio.MessageQueue$MessageWorker.handle(MessageQueue.java:80)
at com.code42.queue.QueueWorker.run(QueueWorker.java:47)
at com.code42.messaging.nio.MessageQueue$MessageWorker.run(MessageQueue.java:126)
[04.03.16 10:26:23.744 WARN n-1841228017 2.messaging.peer.NATContinuation] [NATContinuation[736404850046870022](0):733556053290072213<->734415399723664691][active/A_INV_SENT] INVITE timeout, STOP
[04.03.16 10:26:23.950 WARN n-1841228017 de42.messaging.peer.NATConnector] [NATContinuation[736404850046870022](0):733556053290072213<->734415399723664691][active/STOP] RP:: TRAVERSAL FAILURE for com.code42.messaging.peer.NATConnector@3010c4e2
[04.03.16 10:27:19.562 INFO erTimeoutWrk e42.messaging.peer.PeerConnector] PC:: Cancelling connection attempt due to timeout - pending connection=PendingConnection[timeout(ms) = 10000, startTime = Sun Apr 03 10:27:05 GMT 2016, remotePeer = RemotePeer-[guid=734415399723664691, state=CONNECTING, mode=PRIVATE, location=192.168.1.4:4242, public=217.51.136.219:0, transportPbK=X509.checksum(895e4250e002a5f9aedf1304f7226991), transportPbKRequestable=false, connecting=2016-04-03T10:27:05:083, connected=0, disconnected=2016-04-03T10:26:23:951, attempts=2, connectActivity=2016-04-03T10:27:05:083, keepAliveSent=0, minRetry=29930, retryDelay=0, reflector=na, #nat=1, session=null], sessionData = MessageConnection[ sessionInfo=Session[ null], readBuffer=com.code42.messaging.nio.MessageConnection$MessageBuffer@1da4e1ae, maxBufferSize = 65536, currentSize = 0, enqueued = false, #items = 0, Connection[1126734862, state=0, context = Context@920939237[null->nullconnectionState=0], SendBufferQueue[closed=false, queueSize=0, maxSize=1310720, enqueueTimeout=60000, numItems=0, waitingCount=0, waitingPriorityCount=0, items.size()=0, priorityItems.size()=0, lastEnqueueTS=0]]]]
[04.03.16 10:27:29.667 INFO erTimeoutWrk e42.messaging.peer.PeerConnector] PC:: Cancelling connection attempt due to timeout - pending connection=PendingConnection[timeout(ms) = 5000, startTime = Sun Apr 03 10:27:19 GMT 2016, remotePeer = RemotePeer-[guid=734415399723664691, state=CONNECTING, mode=PRIVATE, location=192.168.1.4:4242, public=217.51.136.219:0, transportPbK=X509.checksum(895e4250e002a5f9aedf1304f7226991), transportPbKRequestable=false, connecting=2016-04-03T10:27:19:667, connected=0, disconnected=2016-04-03T10:27:19:667, attempts=1, connectActivity=2016-04-03T10:27:19:667, keepAliveSent=0, minRetry=29930, retryDelay=0, reflector=na, #nat=1, session=null], sessionData = MessageConnection[ sessionInfo=Session[ null], readBuffer=com.code42.messaging.nio.MessageConnection$MessageBuffer@4bb3d033, maxBufferSize = 65536, currentSize = 0, enqueued = false, #items = 0, Connection[1906368890, state=0, context = Context@1995430828[null->nullconnectionState=0], SendBufferQueue[closed=false, queueSize=0, maxSize=1310720, enqueueTimeout=60000, numItems=0, waitingCount=0, waitingPriorityCount=0, items.size()=0, priorityItems.size()=0, lastEnqueueTS=0]]]]
[04.03.16 10:27:39.340 WARN MQ-Peer-1 2.messaging.MessageReceiverProxy] MessageReceiverProxy: InvocationTargetException receiving message=type=com.code42.peer.message.NATInviteMessage, contents=NATInviteMessage[736326220872622598] [[email protected]:52973], session=Session[id=736326220872622598, closed=false, isAcceptor=false, remoteIdentity=ENDPOINT, completedAuth=true, lat=2016-04-03T10:27:39:336, lrt=2016-04-03T10:27:39:336, lwt=2016-04-03T10:26:36:711, #pending=0, enqueued=true, local=172.17.0.7:50856, remote=216.17.8.3:443, usingProtoHeaders=true, usingEncryptedHeaders=true, WAN], e=java.lang.reflect.InvocationTargetException, cause=java.lang.IllegalArgumentException: Peer already existed, cannot add. PeerLocation=[[[email protected]:52973]], java.lang.IllegalArgumentException: Peer already existed, cannot add. PeerLocation=[[[email protected]:52973]]
STACKTRACE:: java.lang.IllegalArgumentException: Peer already existed, cannot add. PeerLocation=[[[email protected]:52973]]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at com.code42.messaging.peer.PeerGroup.addPeerFromLocation(PeerGroup.java:683)
at com.code42.messaging.peer.NATAgent.receiveMessage(NATAgent.java:94)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.code42.messaging.MessageReceiverProxy.receiveMessage(MessageReceiverProxy.java:149)
at com.code42.messaging.Session.receiveMessage(Session.java:379)
at com.code42.messaging.nio.MessageQueue$MessageWorker.handle(MessageQueue.java:161)
at com.code42.messaging.nio.MessageQueue$MessageWorker.handle(MessageQueue.java:80)
at com.code42.queue.QueueWorker.run(QueueWorker.java:47)
at com.code42.messaging.nio.MessageQueue$MessageWorker.run(MessageQueue.java:126)
[04.03.16 10:28:03.712 WARN ion-46752562 2.messaging.peer.NATContinuation] [NATContinuation[736405017768698374](0):733556053290072213<->733351749112369151][active/A_INV_SENT] INVITE timeout, STOP
[04.03.16 10:28:03.916 WARN ion-46752562 de42.messaging.peer.NATConnector] [NATContinuation[736405017768698374](0):733556053290072213<->733351749112369151][active/STOP] RP:: TRAVERSAL FAILURE for com.code42.messaging.peer.NATConnector@7c3ce6ef
[04.03.16 10:28:55.125 INFO ystemWatcher ode42.messaging.nio.MessageQueue] MW:: STATS:Peer : numWorkers=4, queue.size=0, MessageQueueStats[num=1199, numSinceLast=2, handling=Counter[count = 4301, rate = 0.10, rollingRate = 0.09], waiting=Counter[count = 188176583, rate = 1810.08, rollingRate = 4211.56], workload=0.000]
[04.03.16 10:28:59.777 INFO erTimeoutWrk e42.messaging.peer.PeerConnector] PC:: Cancelling connection attempt due to timeout - pending connection=PendingConnection[timeout(ms) = 10000, startTime = Sun Apr 03 10:28:45 GMT 2016, remotePeer = RemotePeer-[guid=733351749112369151, state=CONNECTING, mode=PRIVATE, location=192.168.178.56:4242, public=79.247.139.110:0, transportPbK=X509.checksum(dfb2987cb569a2e55152753fa25ab089), transportPbKRequestable=false, connecting=2016-04-03T10:28:45:001, connected=0, disconnected=2016-04-03T10:28:03:916, attempts=2, connectActivity=2016-04-03T10:28:45:001, keepAliveSent=0, minRetry=29648, retryDelay=0, reflector=na, #nat=1, session=null], sessionData = MessageConnection[ sessionInfo=Session[ null], readBuffer=com.code42.messaging.nio.MessageConnection$MessageBuffer@634924d6, maxBufferSize = 65536, currentSize = 0, enqueued = false, #items = 0, Connection[917394116, state=0, context = Context@1089723994[null->nullconnectionState=0], SendBufferQueue[closed=false, queueSize=0, maxSize=1310720, enqueueTimeout=60000, numItems=0, waitingCount=0, waitingPriorityCount=0, items.size()=0, priorityItems.size()=0, lastEnqueueTS=0]]]]
[04.03.16 10:29:09.885 INFO erTimeoutWrk e42.messaging.peer.PeerConnector] PC:: Cancelling connection attempt due to timeout - pending connection=PendingConnection[timeout(ms) = 5000, startTime = Sun Apr 03 10:28:59 GMT 2016, remotePeer = RemotePeer-[guid=733351749112369151, state=CONNECTING, mode=PRIVATE, location=192.168.178.56:4242, public=79.247.139.110:0, transportPbK=X509.checksum(dfb2987cb569a2e55152753fa25ab089), transportPbKRequestable=false, connecting=2016-04-03T10:28:59:884, connected=0, disconnected=2016-04-03T10:28:59:884, attempts=1, connectActivity=2016-04-03T10:28:59:884, keepAliveSent=0, minRetry=29648, retryDelay=0, reflector=na, #nat=1, session=null], sessionData = MessageConnection[ sessionInfo=Session[ null], readBuffer=com.code42.messaging.nio.MessageConnection$MessageBuffer@48c95f53, maxBufferSize = 65536, currentSize = 0, enqueued = false, #items = 0, Connection[1653386504, state=0, context = Context@644934639[null->nullconnectionState=0], SendBufferQueue[closed=false, queueSize=0, maxSize=1310720, enqueueTimeout=60000, numItems=0, waitingCount=0, waitingPriorityCount=0, items.size()=0, priorityItems.size()=0, lastEnqueueTS=0]]]]
[04.03.16 10:29:43.662 WARN on-558619013 2.messaging.peer.NATContinuation] [NATContinuation[736405185406640646](0):733556053290072213<->734415399723664691][active/A_INV_SENT] INVITE timeout, STOP
[04.03.16 10:29:43.870 WARN on-558619013 de42.messaging.peer.NATConnector] [NATContinuation[736405185406640646](0):733556053290072213<->734415399723664691][active/STOP] RP:: TRAVERSAL FAILURE for com.code42.messaging.peer.NATConnector@3010c4e2
[04.03.16 10:29:55.130 INFO ystemWatcher ode42.messaging.nio.MessageQueue] MW:: STATS:UI : numWorkers=2, queue.size=0, MessageQueueStats[num=0, numSinceLast=0, handling=Counter[count = 0, rate = 0.00, rollingRate = 0.00], waiting=Counter[count = 0, rate = 0.00, rollingRate = 0.00], workload=1.000]