<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="72

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

All right guys, <a class="user-mention notranslate" data-hovercard-type="user" data-ho

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

we'll take a look at this - wonder if we made any changes to <code class="notranslate"

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

We appreicate your help <a class="user-mention notranslate" data-hovercard-type="user"

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Petabridge.Cmd `actor hierarchy` prints different results each time.,about petabridge/petabridge.cmd-issues

Comments (24)

IgorFedchenko commented on July 21, 2024 5

@seungyongshim

You said, akka.tcp://Sys@localhost:90/remote/akka/tcp/localhost:89/$a is temporary "service" actor, isn't it?

No, my question is about c1 actor. There is no doubt actors with names starting with $a , $b are some internal Akka.NET actors, and there is no problem with them in this issue. The problem is, how I see it right now, that sometimes c1 actors are printed out in the hierarchy, and sometimes they are not.

My current theory is that this c1 actors might not be live all the time - they may be activated on demand to send a remote message for example, or due to some other stuff required for Akka.Remote. Living for a short period of time. And that's why they may be included / excluded from the cmd logs. We do not want petabridge.cmd to show actors that are not working at the moment, right?

And my question is that, I could not find piece of code in Akka that is responsible for such remote reference representation of remote actor link. If I would found it, I could see if this is normal that this reference is not displayed. Or, as @Aaronontheweb said, they might not be discovered due to the timeout of some kind.

Actually, let me try to increase command timeout and see if we will get c1 actors of all three nodes displayed stable.

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024 3

All right guys, @Aaronontheweb was absolutely right. Current timeout per layer is 40ms, and when changed to 400ms utility gives all c1 actors on all three nodes, all the time. Not sure why this is happening with this c1 actors only, but anyway, increasing the timeout does the trick.

Now the bad news is that this timeout is hardcoded as a readonly static field of internal class, so there is no way to configure this on your side. I am going to update our tool to support -t parameter, like actor hierarchy -d 10 -t 400 for example, to make this timeout configurable when someone is getting issues like that.

And we will push this update with 0.8.2 version of the Petabridge.Cmd.Host package. Does it sound good @kimbyungeun ?

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024 3

@kimbyungeun We have just released v0.8.2 of Petabridge.Cmd package pack.

Try to update and use actor hierarchy -d 10 -t 400 and let us know if the problem is resolved this way.

We will also update tool documentation soon, but the -t parameter stands for timeout-per-layer and allows to increase default 40ms timeout to 400ms for example.

from petabridge.cmd-issues.

seungyongshim commented on July 21, 2024 2

@IgorFedchenko Excuse, me.

I don't understand temporary "service" actor.
You said, akka.tcp://Sys@localhost:90/remote/akka/tcp/localhost:89/$a is temporary "service" actor, isn't it?
- https://getakka.net/articles/clustering/cluster-routing.html#clustered-pool-routers

/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1 is akka.tcp://[email protected]:8728/remote/akka.tcp/[email protected]:8722/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1#939527172
TestClusterApi's actor hierarchy

...
[akka://test-cluster-system/user]
[akka://test-cluster-system/user/ApiMasterActor#226114292]
[akka://test-cluster-system/user/ApiMasterActor/singleton#1061564729]
[akka://test-cluster-system/user/ApiMasterActor/singleton/TestClusterRole1HelloActor#487008015]
[akka.tcp://[email protected]:8728/remote/akka.tcp/[email protected]:8722/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1#939527172]
[akka://test-cluster-system/user/ApiMasterActor/singleton/TestClusterRole2HelloActor#25528566]
[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:8722/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1#737878211]
[akka://test-cluster-system/user/ApiMasterActor/singleton/TestClusterRole3HelloActor#35980901]
[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:8722/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1#544185678]
[akka://test-cluster-system/user/ApiMasterActor/$a#1800049128]
[akka://test-cluster-system/user/petabridge.cmd#1953829908]
[akka://test-cluster-system/user/petabridge.cmd-log-memorizer#302235048]
[akka://test-cluster-system/user/pbm-uptime#1968095170]

@kimbyungeun

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024 2

@IgorFedchenko @Aaronontheweb

It will be very helpful. Thank you very much.

actor hierarchy -d 10 -t 90 All actors are printed.

PS C:\Users\be.kim> pbm 127.0.0.1:9111
 ____      _        _          _     _
|  _ \ ___| |_ __ _| |__  _ __(_) __| | __ _  ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
|  __|  __| || (_| | |_) | |  | | (_| | (_| |  __/
|_|   \___|\__\__,_|_.__/|_|  |_|\__,_|\__, |\___|
                                       |___/

petabridge.cmd (0.8.2.0)
Copyright 2017 - 2020, Petabridge®.

successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$a/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$b/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$c/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>

@seungyongshim

from petabridge.cmd-issues.

Aaronontheweb commented on July 21, 2024 2

Glad to hear it - please let us know if there's anything else we can do to help

from petabridge.cmd-issues.

Aaronontheweb commented on July 21, 2024 1

we'll take a look at this - wonder if we made any changes to ActorSelection in the meantime.

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024 1

@IgorFedchenko

Today I will separate the source code and upload it.
Please wait a little.

from petabridge.cmd-issues.

Aaronontheweb commented on July 21, 2024 1

We appreicate your help @kimbyungeun !

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024 1

@IgorFedchenko Upload the sample program. I hope it helps.
ClusterSample.zip

Hello. (안녕하세요) @Aaronontheweb

I have 2 questions about the hocon configuration of our sample(ClusterSample.zip).

Q1

Upload sample has the same structure as our application.
The hocon settings are almost the same.
Please review the sample's hocon settings for any problems.
I doubt that our hocon settings are correct.

Q2 : `acceptable-heartbeat-pause = 240s`

I don't think the 240s setting is correct.
However, when 15s is set in the customer environment, it is disassociated from an unspecified service and Marking node(s) as UNREACHABLE occurs.
In our environment, disassociated does not occur when setting 15s.
There is not enough information, but please advise.

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024 1

@kimbyungeun Thank you very much! I or another member of the team will check this out

from petabridge.cmd-issues.

Aaronontheweb commented on July 21, 2024 1

Resolved via Petabridge.Cmd v0.8.2

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024

The reasons why the results are in different orders each time is because the actor hierarchy scan is done recursively in parallel down in the actor hierarchy using wildcard ActorSelections and the completion order of those parallel tasks is arbitrary. We're able to guarantee that a parent doesn't complete its scan until its children have completed theirs and so on, all the way until we reach the specified level of depth. But we don't guarantee the completion order of siblings - i.e. the order in which two children of the /user actor complete their scans.

Using akka 1.3.16 will print all actors.

pbm api log

[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c3
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c2
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3064
/user/petabridge.cmd/127.0.0.1%3A3064/actor
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler/$h
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler/$h/$a
/user/petabridge.cmd-log-memorizer

Using akka 1.4.10 will print some actors.

pbm lighthouse log

[127.0.0.1:9115] pbm> cluster show
akka.tcp://[email protected]:3507 | [api] | up |
akka.tcp://[email protected]:3511 | [role1] | up |
akka.tcp://[email protected]:4062 | [lighthouse] | up |
akka.tcp://[email protected]:8112 | [role2] | up |
akka.tcp://[email protected]:8113 | [role3] | up |
Count: 5 nodes
[127.0.0.1:9115] pbm>

pbm api log

print some actors.

 ____      _        _          _     _
|  _ \ ___| |_ __ _| |__  _ __(_) __| | __ _  ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
|  __|  __| || (_| | |_) | |  | | (_| | (_| |  __/
|_|   \___|\__\__,_|_.__/|_|  |_|\__,_|\__, |\___|
                                       |___/

petabridge.cmd (0.8.0.0)
Copyright 2017 - 2020, Petabridge®.

successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>

The following Received unexpected ActorIdentity warning log is displayed in the [api] role.

[2020/10/22 19:50:48.525][   27][ INFO]akka.tcp://[email protected]:3925/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] Received petabridge.cmd connection from client [127.0.0.1:3943]
[2020/10/22 19:50:48.525][   27][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending palettes to client...
[2020/10/22 19:50:57.714][    7][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]]
[2020/10/22 19:50:57.776][   32][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://[email protected]:3927/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 : <ActorIdentity>: [akka.tcp://[email protected]:3927/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#1878102528] - MessageId=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
[2020/10/22 19:50:57.805][   10][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1 : <ActorIdentity>: [akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792] - MessageId=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a to client [64 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b to client [67 bytes]
[2020/10/22 19:50:58.302][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes]
[2020/10/22 19:50:58.309][   30][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes]
[2020/10/22 19:51:01.446][   10][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]]
[2020/10/22 19:51:01.508][    9][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792]]
[2020/10/22 19:51:02.100][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes]
[2020/10/22 19:51:02.100][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes]
[2020/10/22 19:51:02.100][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes]
[2020/10/22 19:51:02.100][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes]
[2020/10/22 19:51:02.100][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to client [77 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes]
[2020/10/22 19:51:02.109][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes]
[2020/10/22 19:51:02.126][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b to client [64 bytes]
[2020/10/22 19:51:02.126][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b to client [67 bytes]
[2020/10/22 19:51:02.126][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes]
[2020/10/22 19:51:02.126][    5][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes]
[2020/10/22 19:51:04.114][   26][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]]
[2020/10/22 19:51:04.131][   29][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792]]
[2020/10/22 19:51:04.721][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes]
[2020/10/22 19:51:04.721][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes]
[2020/10/22 19:51:04.721][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes]
[2020/10/22 19:51:04.721][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to client [77 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes]
[2020/10/22 19:51:04.731][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes]
[2020/10/22 19:51:04.748][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes]
[2020/10/22 19:51:04.748][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c to client [64 bytes]
[2020/10/22 19:51:04.774][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b to client [67 bytes]
[2020/10/22 19:51:04.774][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes]
[2020/10/22 19:51:04.774][   26][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes]
[2020/10/22 19:51:06.455][    4][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]]
[2020/10/22 19:51:06.517][   22][ WARN]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792]]
[2020/10/22 19:51:06.581][    6][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://[email protected]:3927/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 : <ActorIdentity>: [akka.tcp://[email protected]:3927/remote/akka.tcp/[email protected]:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#1878102528] - MessageId=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
[2020/10/22 19:51:07.066][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes]
[2020/10/22 19:51:07.066][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes]
[2020/10/22 19:51:07.066][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes]
[2020/10/22 19:51:07.066][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes]
[2020/10/22 19:51:07.066][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes]
[2020/10/22 19:51:07.066][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d to client [64 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b to client [67 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes]
[2020/10/22 19:51:07.076][   10][DEBUG]akka.tcp://[email protected]:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes]

[api] role hocon

petabridge.cmd 
{
    host = "0.0.0.0"
    port = 9111
    log-palettes-on-startup = on
}

akka 
{
	service.name = "TestClusterApi"
	service.description = "TestClusterApi"
	system.name = "mls-cluster-system" 	

	stdout-loglevel = DEBUG
	loglevel = DEBUG
	log-config-on-start = on
	loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

	actor
	{
		provider = cluster
		guardian-supervisor-strategy = "TestClusterApi.ActorService.UserSupervisorStrategy, TestClusterApi"

		debug
		{
			receive = off         
			autoreceive = off     
			lifecycle = off       
			event-stream = off    
			unhandled = on        
		}

		deployment 
		{
			/ApiMasterActor 
			{
			
			}

			"/ApiMasterActor/*/TestClusterRole1HelloActor"
			{
				router = round-robin-pool
				nr-of-instances = 100
				
				cluster 
				{
					enabled = on
					max-nr-of-instances-per-node = 1
					allow-local-routees = off
					use-role = role1
				}
			}

			"/ApiMasterActor/*/TestClusterRole2HelloActor"
			{
				router = round-robin-pool
				nr-of-instances = 100
				
				cluster 
				{
					enabled = on
					max-nr-of-instances-per-node = 1
					allow-local-routees = off
					use-role = role2
				}
			}

			"/ApiMasterActor/*/TestClusterRole3HelloActor"
			{
				router = round-robin-pool
				nr-of-instances = 100

				cluster 
				{
					enabled = on
					max-nr-of-instances-per-node = 1
					allow-local-routees = off
					use-role = role3
				}
			}
		}
	}

	remote
	{
		dot-netty.tcp 
		{
			public-hostname = 127.0.0.1
			port = 0

			send-buffer-size    = 2097152b  # 2MB
			receive-buffer-size = 2097152b  # 2MB
			maximum-frame-size  = 1048576b  # 1MB
		}
		
		transport-failure-detector
		{
			heartbeat-interval = 5s                
			acceptable-heartbeat-pause = 50s       
		}
	}

	extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]

	cluster 
	{
		seed-nodes = ["akka.tcp://[email protected]:4062"]
		roles = [api]

		auto-down-unreachable-after = 10s 

		gossip-interval = 5s 
		gossip-time-to-live = 30s 

		failure-detector 
		{
			heartbeat-interval = 5s
			acceptable-heartbeat-pause = 50s
		}

		singleton
		{
			singleton-name = "ClusterSingleton"
			role = "role1"
			hand-over-retry-interval = 1s
			min-number-of-hand-over-retries = 10
		}
	}
}

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024

@kimbyungeun I am going to look into this. From the logs in your last message, it seems like under Akka 1.4+ not all actors are displayed, right? But all c1/c2/c3 child actors under TestClusterRole1HelloActor are running all the time, right (I mean, it is not that they are shutting down and starting from time to time)?

Do you have a sample project you used to generate this logs? If you do, could you share in as a zip file or Github repo to help me with reproduction? That would be extremely useful.

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024

@IgorFedchenko

Q1

I am going to look into this. From the logs in your last message, it seems like under Akka 1.4+ not all actors are displayed, right?

Q2

But all c1/c2/c3 child actors under TestClusterRole1HelloActor are running all the time, right (I mean, it is not that they are shutting down and starting from time to time)?

Lighthouse.log

[2020-10-23 09:18:22.1925][ INFO][P:13308][T:  1]  Configuration Result:
[Success] Name Mirero.MLS.TestLightHouse4061
[Success] Description Mirero.MLS.TestLightHouse.4061
[Success] ServiceName Mirero.MLS.TestLightHouse4061 
[2020-10-23 09:18:22.2125][ INFO][P:13308][T:  1]  Topshelf v4.2.1.215, .NET Framework v4.0.30319.42000 
[2020-10-23 09:18:22.2355][DEBUG][P:13308][T:  1]  Running as a console application, creating the console host. 
[2020-10-23 09:18:22.2355][DEBUG][P:13308][T:  1]  Starting up as a console application 
[2020-10-23 09:18:22.5005][DEBUG][P:13308][T:  5] EventStream(mls-cluster-system) Logger log1-NLogLogger [NLogLogger] started 
[2020-10-23 09:18:22.5005][DEBUG][P:13308][T:  5] EventStream(mls-cluster-system) StandardOutLogger being removed 
[2020-10-23 09:18:22.5005][DEBUG][P:13308][T:  5] EventStream(mls-cluster-system) Default Loggers started 
[2020-10-23 09:18:22.5135][ INFO][P:13308][T:  8] remoting (akka://mls-cluster-system) Starting remoting 
[2020-10-23 09:18:22.5285][DEBUG][P:13308][T:  8] remoting (akka://mls-cluster-system) Starting prune timer for endpoint manager... 
[2020-10-23 09:18:22.7185][ INFO][P:13308][T:  5] remoting (akka://mls-cluster-system) Remoting started; listening on addresses : [akka.tcp://[email protected]:4062] 
[2020-10-23 09:18:22.7185][ INFO][P:13308][T:  5] remoting (akka://mls-cluster-system) Remoting now listens on addresses: [akka.tcp://[email protected]:4062] 
[2020-10-23 09:18:22.7515][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Starting up... 
[2020-10-23 09:18:22.7755][ INFO][P:13308][T:  6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Started up successfully 
[2020-10-23 09:18:22.8015][ INFO][P:13308][T: 19] ActorSystem(mls-cluster-system)   akka : {
    version : "0.0.1 Akka"
    home : 
    loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
    loggers-dispatcher : akka.actor.default-dispatcher
    logger-startup-timeout : 5s
    logger-async-start : false
    loglevel : DEBUG
    suppress-json-serializer-warning : on
    stdout-loglevel : DEBUG
    log-config-on-start : on
    log-dead-letters : 10
    log-dead-letters-during-shutdown : off
    log-dead-letters-suspend-duration : "5 minutes"
    extensions : ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]
    daemonic : off
    actor : {
      provider : cluster
      guardian-supervisor-strategy : Akka.Actor.DefaultSupervisorStrategy
      creation-timeout : 20s
      reaper-interval : 5
      serialize-messages : off
      serialize-creators : off
      unstarted-push-timeout : 10s
      ask-timeout : infinite
      typed : {
        timeout : 5
      }
      inbox : {
        inbox-size : 1000
        default-timeout : 5s
      }
      router : {
        type-mapping : {
          from-code : Akka.Routing.NoRouter
          round-robin-pool : Akka.Routing.RoundRobinPool
          round-robin-group : Akka.Routing.RoundRobinGroup
          random-pool : Akka.Routing.RandomPool
          random-group : Akka.Routing.RandomGroup
          smallest-mailbox-pool : Akka.Routing.SmallestMailboxPool
          broadcast-pool : Akka.Routing.BroadcastPool
          broadcast-group : Akka.Routing.BroadcastGroup
          scatter-gather-pool : Akka.Routing.ScatterGatherFirstCompletedPool
          scatter-gather-group : Akka.Routing.ScatterGatherFirstCompletedGroup
          consistent-hashing-pool : Akka.Routing.ConsistentHashingPool
          consistent-hashing-group : Akka.Routing.ConsistentHashingGroup
          tail-chopping-pool : Akka.Routing.TailChoppingPool
          tail-chopping-group : Akka.Routing.TailChoppingGroup
        }
      }
      deployment : {
        default : {
          dispatcher : 
          mailbox : 
          router : from-code
          nr-of-instances : 1
          within : "5 s"
          virtual-nodes-factor : 10
          routees : {
            paths : []
          }
          resizer : {
            enabled : off
            lower-bound : 1
            upper-bound : 10
            pressure-threshold : 1
            rampup-rate : 0.2
            backoff-threshold : 0.3
            backoff-rate : 0.1
            messages-per-resize : 10
          }
          remote : 
          target : {
            nodes : []
          }
          metrics-selector : mix
          cluster : {
            enabled : off
            max-nr-of-instances-per-node : 1
            max-total-nr-of-instances : 10000
            allow-local-routees : on
            use-role : 
          }
        }
      }
      synchronized-dispatcher : {
        type : SynchronizedDispatcher
        executor : current-context-executor
        throughput : 10
      }
      task-dispatcher : {
        type : TaskDispatcher
        executor : task-executor
        throughput : 30
      }
      default-fork-join-dispatcher : {
        type : ForkJoinDispatcher
        executor : fork-join-executor
        throughput : 30
        dedicated-thread-pool : {
          thread-count : 3
          threadtype : background
        }
      }
      default-dispatcher : {
        type : Dispatcher
        executor : default-executor
        default-executor : {
        }
        thread-pool-executor : {
        }
        fork-join-executor : {
          dedicated-thread-pool : {
            thread-count : 3
            threadtype : background
          }
        }
        current-context-executor : {
        }
        shutdown-timeout : 1s
        throughput : 30
        throughput-deadline-time : 0ms
        attempt-teamwork : on
        mailbox-requirement : 
      }
      default-mailbox : {
        mailbox-type : Akka.Dispatch.UnboundedMailbox
        mailbox-capacity : 1000
        mailbox-push-timeout-time : 10s
        stash-capacity : -1
      }
      mailbox : {
        requirements : {
          Akka.Dispatch.IUnboundedMessageQueueSemantics : akka.actor.mailbox.unbounded-queue-based
          Akka.Dispatch.IBoundedMessageQueueSemantics : akka.actor.mailbox.bounded-queue-based
          Akka.Dispatch.IDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based
          Akka.Dispatch.IUnboundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based
          Akka.Dispatch.IBoundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.bounded-deque-based
          Akka.Dispatch.IMultipleConsumerSemantics : akka.actor.mailbox.unbounded-queue-based
          Akka.Event.ILoggerMessageQueueSemantics : akka.actor.mailbox.logger-queue
        }
        unbounded-queue-based : {
          mailbox-type : Akka.Dispatch.UnboundedMailbox
        }
        bounded-queue-based : {
          mailbox-type : Akka.Dispatch.BoundedMailbox
        }
        unbounded-deque-based : {
          mailbox-type : Akka.Dispatch.UnboundedDequeBasedMailbox
        }
        bounded-deque-based : {
          mailbox-type : Akka.Dispatch.BoundedDequeBasedMailbox
        }
        logger-queue : {
          mailbox-type : Akka.Event.LoggerMailboxType
        }
      }
      debug : {
        receive : off
        autoreceive : off
        lifecycle : off
        fsm : off
        event-stream : off
        unhandled : on
        router-misconfiguration : off
      }
      serializers : {
        json : "Akka.Serialization.NewtonSoftJsonSerializer, Akka"
        bytes : "Akka.Serialization.ByteArraySerializer, Akka"
        akka-containers : "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote"
        akka-misc : "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote"
        primitive : "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote"
        proto : "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote"
        daemon-create : "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote"
        akka-system-msg : "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote"
        akka-cluster : "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster"
        akka-cluster-client : "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools"
        akka-pubsub : "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools"
      }
      serialization-bindings : {
        System.Byte[] : bytes
        System.Object : json
        "Akka.Actor.ActorSelectionMessage, Akka" : akka-containers
        "Akka.Remote.DaemonMsgCreate, Akka.Remote" : daemon-create
        "Google.Protobuf.IMessage, Google.Protobuf" : proto
        "Akka.Actor.Identify, Akka" : akka-misc
        "Akka.Actor.ActorIdentity, Akka" : akka-misc
        "Akka.Actor.IActorRef, Akka" : akka-misc
        "Akka.Actor.PoisonPill, Akka" : akka-misc
        "Akka.Actor.Kill, Akka" : akka-misc
        "Akka.Actor.RemoteScope, Akka" : akka-misc
        "Akka.Routing.FromConfig, Akka" : akka-misc
        "Akka.Routing.DefaultResizer, Akka" : akka-misc
        "Akka.Routing.RoundRobinPool, Akka" : akka-misc
        "Akka.Routing.BroadcastPool, Akka" : akka-misc
        "Akka.Routing.RandomPool, Akka" : akka-misc
        "Akka.Routing.ScatterGatherFirstCompletedPool, Akka" : akka-misc
        "Akka.Routing.TailChoppingPool, Akka" : akka-misc
        "Akka.Routing.ConsistentHashingPool, Akka" : akka-misc
        "Akka.Configuration.Config, Akka" : akka-misc
        "Akka.Remote.RemoteWatcher+Heartbeat, Akka.Remote" : akka-misc
        "Akka.Remote.RemoteWatcher+HeartbeatRsp, Akka.Remote" : akka-misc
        "Akka.Remote.Routing.RemoteRouterConfig, Akka.Remote" : akka-misc
        "Akka.Dispatch.SysMsg.SystemMessage, Akka" : akka-system-msg
        System.String : primitive
        System.Int32 : primitive
        System.Int64 : primitive
        "Akka.Cluster.IClusterMessage, Akka.Cluster" : akka-cluster
        "Akka.Cluster.Routing.ClusterRouterPool, Akka.Cluster" : akka-cluster
        "Akka.Cluster.Tools.Client.IClusterClientMessage, Akka.Cluster.Tools" : akka-cluster-client
        "Akka.Cluster.Tools.PublishSubscribe.IDistributedPubSubMessage, Akka.Cluster.Tools" : akka-pubsub
        "Akka.Cluster.Tools.PublishSubscribe.Internal.SendToOneSubscriber, Akka.Cluster.Tools" : akka-pubsub
      }
      serialization-identifiers : {
        "Akka.Serialization.ByteArraySerializer, Akka" : 4
        "Akka.Serialization.NewtonSoftJsonSerializer, Akka" : 1
        "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" : 2
        "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" : 3
        "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" : 6
        "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" : 16
        "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" : 17
        "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" : 22
        "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" : 5
        "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" : 15
        "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" : 9
      }
      serialization-settings : {
      }
    }
    scheduler : {
      tick-duration : 10ms
      ticks-per-wheel : 512
      implementation : Akka.Actor.HashedWheelTimerScheduler
      shutdown-timeout : 5s
    }
    io : {
      pinned-dispatcher : {
        type : PinnedDispatcher
        executor : fork-join-executor
      }
      tcp : {
        direct-buffer-pool : {
          class : "Akka.IO.Buffers.DirectBufferPool, Akka"
          buffer-size : 512
          buffers-per-segment : 500
          initial-segments : 1
          buffer-pool-limit : 1024
        }
        disabled-buffer-pool : {
          class : "Akka.IO.Buffers.DisabledBufferPool, Akka"
          buffer-size : 512
        }
        buffer-pool : akka.io.tcp.disabled-buffer-pool
        max-channels : 256000
        selector-association-retries : 10
        batch-accept-limit : 10
        register-timeout : 5s
        max-received-message-size : unlimited
        trace-logging : off
        selector-dispatcher : akka.io.pinned-dispatcher
        worker-dispatcher : akka.actor.default-dispatcher
        management-dispatcher : akka.actor.default-dispatcher
        file-io-dispatcher : akka.actor.default-dispatcher
        file-io-transferTo-limit : 524288
        finish-connect-retries : 5
        windows-connection-abort-workaround-enabled : off
        outgoing-socket-force-ipv4 : false
      }
      udp : {
        direct-buffer-pool : {
          class : "Akka.IO.Buffers.DirectBufferPool, Akka"
          buffer-size : 512
          buffers-per-segment : 500
          initial-segments : 1
          buffer-pool-limit : 1024
        }
        buffer-pool : akka.io.udp.direct-buffer-pool
        nr-of-socket-async-event-args : 32
        max-channels : 4096
        select-timeout : infinite
        selector-association-retries : 10
        receive-throughput : 3
        direct-buffer-size : 18432
        direct-buffer-pool-limit : 1000
        received-message-size-limit : unlimited
        trace-logging : off
        selector-dispatcher : akka.io.pinned-dispatcher
        worker-dispatcher : akka.actor.default-dispatcher
        management-dispatcher : akka.actor.default-dispatcher
      }
      udp-connected : {
        direct-buffer-pool : {
          class : "Akka.IO.Buffers.DirectBufferPool, Akka"
          buffer-size : 512
          buffers-per-segment : 500
          initial-segments : 1
          buffer-pool-limit : 1024
        }
        buffer-pool : akka.io.udp-connected.direct-buffer-pool
        nr-of-socket-async-event-args : 32
        max-channels : 4096
        select-timeout : infinite
        selector-association-retries : 10
        receive-throughput : 3
        direct-buffer-size : 18432
        direct-buffer-pool-limit : 1000
        received-message-size-limit : unlimited
        trace-logging : off
        selector-dispatcher : akka.io.pinned-dispatcher
        worker-dispatcher : akka.actor.default-dispatcher
        management-dispatcher : akka.actor.default-dispatcher
      }
      dns : {
        dispatcher : akka.actor.default-dispatcher
        resolver : inet-address
        inet-address : {
          provider-object : Akka.IO.InetAddressDnsProvider
          positive-ttl : 30s
          negative-ttl : 10s
          cache-cleanup-interval : 120s
          use-ipv6 : true
        }
      }
    }
    coordinated-shutdown : {
      default-phase-timeout : "5 s"
      terminate-actor-system : on
      exit-clr : off
      run-by-clr-shutdown-hook : on
      run-by-actor-system-terminate : on
      phases : {
        before-service-unbind : {
        }
        service-unbind : {
          depends-on : [before-service-unbind]
        }
        service-requests-done : {
          depends-on : [service-unbind]
        }
        service-stop : {
          depends-on : [service-requests-done]
        }
        before-cluster-shutdown : {
          depends-on : [service-stop]
        }
        cluster-sharding-shutdown-region : {
          timeout : "10 s"
          depends-on : [before-cluster-shutdown]
        }
        cluster-leave : {
          depends-on : [cluster-sharding-shutdown-region]
        }
        cluster-exiting : {
          timeout : "10 s"
          depends-on : [cluster-leave]
        }
        cluster-exiting-done : {
          depends-on : [cluster-exiting]
        }
        cluster-shutdown : {
          depends-on : [cluster-exiting-done]
        }
        before-actor-system-terminate : {
          depends-on : [cluster-shutdown]
        }
        actor-system-terminate : {
          timeout : "10 s"
          depends-on : [before-actor-system-terminate]
        }
      }
    }
    remote : {
      startup-timeout : "10 s"
      shutdown-timeout : "10 s"
      flush-wait-on-shutdown : "2 s"
      use-passive-connections : on
      backoff-interval : "0.05 s"
      command-ack-timeout : "30 s"
      handshake-timeout : "15 s"
      use-dispatcher : akka.remote.default-remote-dispatcher
      untrusted-mode : off
      trusted-selection-paths : []
      require-cookie : off
      secure-cookie : 
      log-received-messages : off
      log-sent-messages : off
      log-remote-lifecycle-events : DEBUG
      log-frame-size-exceeding : off
      log-buffer-size-exceeding : 50000
      transport-failure-detector : {
        implementation-class : Akka.Remote.DeadlineFailureDetector,Akka.Remote
        heartbeat-interval : 5s
        acceptable-heartbeat-pause : 50s
      }
      watch-failure-detector : {
        implementation-class : Akka.Remote.PhiAccrualFailureDetector,Akka.Remote
        heartbeat-interval : "1 s"
        threshold : 10.0
        max-sample-size : 200
        min-std-deviation : "100 ms"
        acceptable-heartbeat-pause : "10 s"
        unreachable-nodes-reaper-interval : 1s
        expected-response-after : "1 s"
      }
      retry-gate-closed-for : "5 s"
      prune-quarantine-marker-after : "5 d"
      quarantine-after-silence : "2 d"
      system-message-buffer-size : 20000
      system-message-ack-piggyback-timeout : "0.3 s"
      resend-interval : "2 s"
      resend-limit : 200
      initial-system-message-delivery-timeout : "3 m"
      enabled-transports : [akka.remote.dot-netty.tcp]
      adapters : {
        gremlin : Akka.Remote.Transport.FailureInjectorProvider,Akka.Remote
        trttl : Akka.Remote.Transport.ThrottlerProvider,Akka.Remote
      }
      helios : {
        tcp : {
          transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote.Transport.Helios"
        }
      }
      dot-netty : {
        tcp : {
          transport-class : Akka.Remote.Transport.DotNetty.TcpTransport,Akka.Remote
          applied-adapters : []
          transport-protocol : tcp
          byte-order : little-endian
          port : 4062
          public-port : 0
          hostname : 
          public-hostname : 127.0.0.1
          dns-use-ipv6 : false
          enforce-ip-family : false
          enable-ssl : false
          enable-backwards-compatibility : false
          connection-timeout : "15 s"
          batching : {
            enabled : true
            max-pending-writes : 30
            max-pending-bytes : 16k
            flush-interval : 40ms
          }
          use-dispatcher-for-io : 
          write-buffer-high-water-mark : 0b
          write-buffer-low-water-mark : 0b
          send-buffer-size : 256000b
          receive-buffer-size : 256000b
          maximum-frame-size : 128000b
          backlog : 4096
          tcp-nodelay : on
          tcp-keepalive : on
          tcp-reuse-addr : off-for-windows
          server-socket-worker-pool : {
            pool-size-min : 2
            pool-size-factor : 1.0
            pool-size-max : 2
          }
          client-socket-worker-pool : {
            pool-size-min : 2
            pool-size-factor : 1.0
            pool-size-max : 2
          }
        }
        udp : {
          transport-protocol : udp
        }
        ssl : {
          certificate : {
            path : 
            password : 
            use-thumprint-over-file : false
            thumpbrint : 
            store-name : 
            store-location : current-user
          }
          suppress-validation : false
        }
      }
      gremlin : {
        debug : off
      }
      default-remote-dispatcher : {
        type : ForkJoinDispatcher
        executor : fork-join-executor
        dedicated-thread-pool : {
          thread-count : 4
        }
      }
      backoff-remote-dispatcher : {
        type : ForkJoinDispatcher
        executor : fork-join-executor
        dedicated-thread-pool : {
          thread-count : 4
        }
      }
    }
    cluster : {
      seed-nodes : ["akka.tcp://[email protected]:4062"]
      seed-node-timeout : 5s
      retry-unsuccessful-join-after : 10s
      shutdown-after-unsuccessful-join-seed-nodes : off
      auto-down-unreachable-after : 10s
      down-removal-margin : off
      downing-provider-class : 
      allow-weakly-up-members : on
      roles : [lighthouse]
      run-coordinated-shutdown-when-down : on
      role : {
      }
      min-nr-of-members : 1
      log-info : on
      log-info-verbose : off
      periodic-tasks-initial-delay : 1s
      gossip-interval : 5s
      gossip-time-to-live : 30s
      leader-actions-interval : 1s
      unreachable-nodes-reaper-interval : 1s
      publish-stats-interval : off
      use-dispatcher : 
      gossip-different-view-probability : 0.8
      reduce-gossip-different-view-probability : 400
      failure-detector : {
        implementation-class : "Akka.Remote.PhiAccrualFailureDetector, Akka.Remote"
        heartbeat-interval : 5s
        threshold : 8.0
        max-sample-size : 1000
        min-std-deviation : "100 ms"
        acceptable-heartbeat-pause : 50s
        monitored-by-nr-of-members : 9
        expected-response-after : "1 s"
      }
      split-brain-resolver : {
        active-strategy : off
        stable-after : 20s
        static-quorum : {
          quorum-size : undefined
          role : 
        }
        keep-majority : {
          role : 
        }
        keep-oldest : {
          down-if-alone : on
          role : 
        }
        keep-refree : {
          address : 
          down-all-if-less-than-nodes : 1
        }
      }
      scheduler : {
        tick-duration : 33ms
        ticks-per-wheel : 512
      }
      debug : {
        verbose-heartbeat-logging : off
        verbose-receive-gossip-logging : off
      }
      client : {
        receptionist : {
          name : receptionist
          role : lighthouse
          number-of-contacts : 3
          response-tunnel-receive-timeout : 1d
          use-dispatcher : 
          heartbeat-interval : 2s
          acceptable-heartbeat-pause : 13s
          failure-detection-interval : 2s
        }
        initial-contacts : []
        establishing-get-contacts-interval : 3s
        refresh-contacts-interval : 60s
        heartbeat-interval : 2s
        acceptable-heartbeat-pause : 13s
        buffer-size : 1000
        reconnect-timeout : off
      }
      pub-sub : {
        name : distributedPubSubMediator
        role : 
        routing-logic : random
        gossip-interval : 1s
        removed-time-to-live : 120s
        max-delta-elements : 3000
        use-dispatcher : 
      }
    }
    service : {
      name : Mirero.MLS.TestLightHouse4061
      description : Mirero.MLS.TestLightHouse.4061
    }
    system : {
      name : mls-cluster-testlighthouse-4061
    }
  }
  lighthouse : {
    actorsystem : mls-cluster-system
  }
  petabridge : {
    cmd : {
      host : 0.0.0.0
      port : 9115
    }
  }
 
[2020-10-23 09:18:22.8135][ INFO][P:13308][T:  1]  The Mirero.MLS.TestLightHouse4061 service is now running, press Control+C to exit. 
[2020-10-23 09:18:22.8305][ INFO][P:13308][T:  6] akka.tcp://[email protected]:4062/user/petabridge.cmd petabridge.cmd host bound to [0.0.0.0:9115] 
[2020-10-23 09:18:22.8305][ INFO][P:13308][T: 22] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Node [akka.tcp://[email protected]:4062] is JOINING itself (with roles [lighthouse]) and forming a new cluster 
[2020-10-23 09:18:22.8435][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - is the new leader among reachable nodes (more leaders may exist) 
[2020-10-23 09:18:22.8655][ INFO][P:13308][T:  5] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is moving node [akka.tcp://[email protected]:4062] to [Up] 
[2020-10-23 09:18:30.8114][DEBUG][P:13308][T: 19] remoting (akka://mls-cluster-system) Associated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:3893 
[2020-10-23 09:18:30.9214][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Received InitJoin message from [[akka.tcp://[email protected]:3893/system/cluster/core/daemon/joinSeedNodeProcess-1#928698614]] to [akka.tcp://[email protected]:4062] 
[2020-10-23 09:18:30.9214][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Sending InitJoinNack message from node [akka.tcp://[email protected]:4062] to [[akka.tcp://[email protected]:3893/system/cluster/core/daemon/joinSeedNodeProcess-1#928698614]] 
[2020-10-23 09:18:31.0354][ INFO][P:13308][T: 19] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Node [akka.tcp://[email protected]:3893] is JOINING, roles [api] 
[2020-10-23 09:18:31.2664][DEBUG][P:13308][T: 22] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:18:31.9224][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is moving node [akka.tcp://[email protected]:3893] to [Up] 
[2020-10-23 09:18:32.8939][DEBUG][P:13308][T:  9] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:18:32.9409][ INFO][P:13308][T:  9] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - is no longer leader 
[2020-10-23 09:18:37.7321][DEBUG][P:13308][T:  4] remoting (akka://mls-cluster-system) Associated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:3895 
[2020-10-23 09:18:37.8771][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Received InitJoin message from [[akka.tcp://[email protected]:3895/system/cluster/core/daemon/joinSeedNodeProcess-1#370319934]] to [akka.tcp://[email protected]:4062] 
[2020-10-23 09:18:37.8771][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Sending InitJoinNack message from node [akka.tcp://[email protected]:4062] to [[akka.tcp://[email protected]:3895/system/cluster/core/daemon/joinSeedNodeProcess-1#370319934]] 
[2020-10-23 09:18:37.9731][ INFO][P:13308][T:  9] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Node [akka.tcp://[email protected]:3895] is JOINING, roles [role1] 
[2020-10-23 09:18:38.1161][DEBUG][P:13308][T: 21] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:18:40.7272][DEBUG][P:13308][T: 26] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:18:42.9415][DEBUG][P:13308][T:  6] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:18:45.4732][DEBUG][P:13308][T:  5] remoting (akka://mls-cluster-system) Associated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:8112 
[2020-10-23 09:18:45.6072][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Received InitJoin message from [[akka.tcp://[email protected]:8112/system/cluster/core/daemon/joinSeedNodeProcess-1#713857872]] to [akka.tcp://[email protected]:4062] 
[2020-10-23 09:18:45.6072][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Sending InitJoinNack message from node [akka.tcp://[email protected]:4062] to [[akka.tcp://[email protected]:8112/system/cluster/core/daemon/joinSeedNodeProcess-1#713857872]] 
[2020-10-23 09:18:45.6392][DEBUG][P:13308][T: 30] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:18:45.7042][ INFO][P:13308][T: 24] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Node [akka.tcp://[email protected]:8112] is JOINING, roles [role2] 
[2020-10-23 09:18:45.8442][DEBUG][P:13308][T: 24] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8112, 1552571697)] 
[2020-10-23 09:18:47.9410][DEBUG][P:13308][T: 21] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:18:50.7473][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:18:51.9473][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Associated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:8113 
[2020-10-23 09:18:52.0973][ INFO][P:13308][T:  8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Received InitJoin message from [[akka.tcp://[email protected]:8113/system/cluster/core/daemon/joinSeedNodeProcess-1#203439572]] to [akka.tcp://[email protected]:4062] 
[2020-10-23 09:18:52.0973][ INFO][P:13308][T:  8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Sending InitJoinNack message from node [akka.tcp://[email protected]:4062] to [[akka.tcp://[email protected]:8113/system/cluster/core/daemon/joinSeedNodeProcess-1#203439572]] 
[2020-10-23 09:18:52.1923][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Node [akka.tcp://[email protected]:8113] is JOINING, roles [role3] 
[2020-10-23 09:18:52.3333][DEBUG][P:13308][T: 23] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8113, 2078514987)] 
[2020-10-23 09:18:52.9553][DEBUG][P:13308][T: 22] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8112, 1552571697)] 
[2020-10-23 09:18:55.6648][DEBUG][P:13308][T:  6] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:18:57.7278][DEBUG][P:13308][T: 22] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:18:57.8228][DEBUG][P:13308][T: 26] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:18:57.9648][DEBUG][P:13308][T: 21] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8113, 2078514987)] 
[2020-10-23 09:19:01.8677][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8113, 2078514987)] 
[2020-10-23 09:19:02.9681][DEBUG][P:13308][T: 30] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8112, 1552571697)] 
[2020-10-23 09:19:05.6717][DEBUG][P:13308][T: 24] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:19:12.9930][DEBUG][P:13308][T:  9] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:19:18.0291][DEBUG][P:13308][T: 21] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:19:20.4801][DEBUG][P:13308][T: 26] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8112, 1552571697)] 
[2020-10-23 09:19:22.7684][DEBUG][P:13308][T: 24] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:35:03.0348][DEBUG][P:13308][T:  9] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8112, 1552571697)] 
[2020-10-23 09:35:04.8020][DEBUG][P:13308][T:  6] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:35:05.6941][DEBUG][P:13308][T:  8] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:35:05.9421][ INFO][P:13308][T: 22]  Control+C detected, attempting to stop service. 
[2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-service-unbind] with [0] tasks. 
[2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-unbind] with [0] tasks. 
[2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-requests-done] with [0] tasks. 
[2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-stop] with [0] tasks. 
[2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-cluster-shutdown] with [0] tasks. 
[2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-sharding-shutdown-region] with [0] tasks. 
[2020-10-23 09:35:05.9531][DEBUG][P:13308][T:  4] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-leave] with [1] tasks: [leave] 
[2020-10-23 09:35:05.9531][ INFO][P:13308][T: 20] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Marked address [akka.tcp://[email protected]:4062] as [Leaving] 
[2020-10-23 09:35:05.9531][DEBUG][P:13308][T:  5] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-exiting] with [1] tasks: [wait-exiting] 
[2020-10-23 09:35:06.0446][DEBUG][P:13308][T:  5] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:35:08.5166][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3894-1 Association between local [tcp://[email protected]:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3894] was disassociated because the ProtocolStateActor failed: Shutdown 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/cluster/core/daemon "Couldn't establish a causal relationship between "remote" gossip and "local" gossip - Remote[Gossip(members = [Member(address = akka.tcp://[email protected]:3893, Uid=1373298527 status = Exiting, role=[api], upNumber=2), Member(address = akka.tcp://[email protected]:3895, Uid=2141762004 status = Exiting, role=[role1], upNumber=3), Member(address = akka.tcp://[email protected]:4062, Uid=1685981556 status = Up, role=[lighthouse], upNumber=1), Member(address = akka.tcp://[email protected]:8112, Uid=1552571697 status = Exiting, role=[role2], upNumber=4), Member(address = akka.tcp://[email protected]:8113, Uid=2078514987 status = Exiting, role=[role3], upNumber=5)], overview = GossipOverview(seen=[], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->7, B0FD6E6E85BC21C543C59A240889BC7C->6, BFDECCB8DF57F49EDBD5FF0F943FF688->1)] - Local[Gossip(members = [Member(address = akka.tcp://[email protected]:3893, Uid=1373298527 status = Leaving, role=[api], upNumber=2), Member(address = akka.tcp://[email protected]:3895, Uid=2141762004 status = Leaving, role=[role1], upNumber=3), Member(address = akka.tcp://[email protected]:4062, Uid=1685981556 status = Leaving, role=[lighthouse], upNumber=1), Member(address = akka.tcp://[email protected]:8112, Uid=1552571697 status = Leaving, role=[role2], upNumber=4), Member(address = akka.tcp://[email protected]:8113, Uid=2078514987 status = Leaving, role=[role3], upNumber=5)], overview = GossipOverview(seen=[UniqueAddress: (akka.tcp://[email protected]:4062, 1685981556), UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->8, B0FD6E6E85BC21C543C59A240889BC7C->5, BFDECCB8DF57F49EDBD5FF0F943FF688->1)] - merged them into [Gossip(members = [Member(address = akka.tcp://[email protected]:3893, Uid=1373298527 status = Exiting, role=[api], upNumber=2), Member(address = akka.tcp://[email protected]:3895, Uid=2141762004 status = Exiting, role=[role1], upNumber=3), Member(address = akka.tcp://[email protected]:4062, Uid=1685981556 status = Leaving, role=[lighthouse], upNumber=1), Member(address = akka.tcp://[email protected]:8112, Uid=1552571697 status = Exiting, role=[role2], upNumber=4), Member(address = akka.tcp://[email protected]:8113, Uid=2078514987 status = Exiting, role=[role3], upNumber=5)], overview = GossipOverview(seen=[], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->8, B0FD6E6E85BC21C543C59A240889BC7C->6, BFDECCB8DF57F49EDBD5FF0F943FF688->1)]" 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3893, 1373298527)] 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://[email protected]:3893] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. 
[2020-10-23 09:35:08.5316][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Exiting confirmed [akka.tcp://[email protected]:3893] 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:3895, 2141762004)] 
[2020-10-23 09:35:08.5316][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Exiting confirmed [akka.tcp://[email protected]:3895] 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://[email protected]:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3896-2 Association between local [tcp://[email protected]:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3896] was disassociated because the ProtocolStateActor failed: Shutdown 
[2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://[email protected]:3895] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [1] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [2] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  5] akka.tcp://[email protected]:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3898-3 Association between local [tcp://[email protected]:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3898] was disassociated because the ProtocolStateActor failed: Shutdown 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 27] akka.tcp://[email protected]:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3901-4 Association between local [tcp://[email protected]:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3901] was disassociated because the ProtocolStateActor failed: Shutdown 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  4] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8112, 1552571697)] 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Exiting confirmed [akka.tcp://[email protected]:8112] 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 27] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [akka.tcp://[email protected]:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://[email protected]:8113, 2078514987)] 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Exiting confirmed [akka.tcp://[email protected]:8113] 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  9] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://[email protected]:8112] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  9] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://[email protected]:8113] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  8] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3893-1/endpointWriter Disassociated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:3893 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  9] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8113-4/endpointWriter Disassociated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:8113 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  9] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8112-3/endpointWriter Disassociated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:8112 
[2020-10-23 09:35:08.5446][DEBUG][P:13308][T:  9] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3895-2/endpointWriter Disassociated [akka.tcp://[email protected]:4062] <- akka.tcp://[email protected]:3895 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3895-2 Removing receive buffers for [akka.tcp://[email protected]:4062]->[akka.tcp://[email protected]:3895] 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3893-1 Removing receive buffers for [akka.tcp://[email protected]:4062]->[akka.tcp://[email protected]:3893] 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  9] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8112-3 Removing receive buffers for [akka.tcp://[email protected]:4062]->[akka.tcp://[email protected]:8112] 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  9] akka.tcp://[email protected]:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8113-4 Removing receive buffers for [akka.tcp://[email protected]:4062]->[akka.tcp://[email protected]:8113] 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  9] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [3] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 
[2020-10-23 09:35:08.5446][ INFO][P:13308][T:  8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [4] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - is the new leader among reachable nodes (more leaders may exist) 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 20] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-exiting-done] with [1] tasks: [exiting-completed] 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is moving node [akka.tcp://[email protected]:4062] to [Exiting] 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is removing confirmed Exiting node [akka.tcp://[email protected]:3893] 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is removing confirmed Exiting node [akka.tcp://[email protected]:8112] 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is removing confirmed Exiting node [akka.tcp://[email protected]:8113] 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Leader is removing confirmed Exiting node [akka.tcp://[email protected]:3895] 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 26] akka.tcp://[email protected]:4062/system/cluster/core/daemon Cluster Node [UniqueAddress: (akka.tcp://[email protected]:4062, 1685981556)] - Gossip exiting members [Member(address = akka.tcp://[email protected]:4062, Uid=1685981556 status = Exiting, role=[lighthouse], upNumber=1)] to the two oldest (per role) [Member(address = akka.tcp://[email protected]:4062, Uid=1685981556 status = Exiting, role=[lighthouse], upNumber=1)] (singleton optimization). 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Exiting completed. 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Shutting down... 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T:  6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://[email protected]:4062] - Successfully shut down 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 30] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-shutdown] with [1] tasks: [wait-shutdown] 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-actor-system-terminate] with [0] tasks. 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [actor-system-terminate] with [1] tasks: [terminate-system] 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 23] ActorSystem(mls-cluster-system) System shutdown initiated 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T: 24] akka.tcp://[email protected]:4062/user/petabridge.cmd Shutting down petabridge.cmd host... 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T:  6] akka.tcp://[email protected]:4062/system/IO-TCP/$a Unbinding endpoint 0.0.0.0:9115 
[2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 20] akka.tcp://[email protected]:4062/system/IO-TCP/$a Unbound endpoint 0.0.0.0:9115, stopping listener 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T: 24] remoting-terminator Shutting down remote daemon. 
[2020-10-23 09:35:08.6996][ INFO][P:13308][T: 38] remoting-terminator Remote daemon shut down; proceeding with flushing remote transports. 
[2020-10-23 09:35:08.7356][ INFO][P:13308][T:  5] remoting (akka://mls-cluster-system) Remoting shut down 
[2020-10-23 09:35:08.7356][ INFO][P:13308][T: 37] remoting-terminator Remoting shut down. 
[2020-10-23 09:35:08.7356][DEBUG][P:13308][T: 39] EventStream Shutting down: StandardOutLogger started 
[2020-10-23 09:35:08.7646][ INFO][P:13308][T:  1]  The Mirero.MLS.TestLightHouse4061 service has stopped.

TestClusterApi.log

[2020/10/23 09:18:30.051][    1][ INFO][Topshelf.HostFactory] Configuration Result:
[Success] Name Mirero.MLS.TestClusterApi
[Success] ServiceName Mirero.MLS.TestClusterApi 
[2020/10/23 09:18:30.106][    1][ INFO][Topshelf.HostConfigurators.HostConfiguratorImpl] Topshelf v4.2.1.215, .NET Framework v4.0.30319.42000 
[2020/10/23 09:18:30.340][    4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Starting remoting 
[2020/10/23 09:18:30.518][    4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Remoting started; listening on addresses : [akka.tcp://[email protected]:3893] 
[2020/10/23 09:18:30.518][    4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Remoting now listens on addresses: [akka.tcp://[email protected]:3893] 
[2020/10/23 09:18:30.544][    5][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Starting up... 
[2020/10/23 09:18:30.563][    9][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Started up successfully 
[2020/10/23 09:18:30.580][   10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl]   akka : {
    version : "0.0.1 Akka"
    home : 
    loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
    loggers-dispatcher : akka.actor.default-dispatcher
    logger-startup-timeout : 5s
    logger-async-start : false
    loglevel : DEBUG
    suppress-json-serializer-warning : on
    stdout-loglevel : DEBUG
    log-config-on-start : on
    log-dead-letters : 10
    log-dead-letters-during-shutdown : off
    log-dead-letters-suspend-duration : "5 minutes"
    extensions : ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]
    daemonic : off
    actor : {
      provider : cluster
      guardian-supervisor-strategy : "Mirero.MLS.TestClusterApi.ActorService.UserSupervisorStrategy, Mirero.MLS.TestClusterApi"
      creation-timeout : 20s
      reaper-interval : 5
      serialize-messages : off
      serialize-creators : off
      unstarted-push-timeout : 10s
      ask-timeout : infinite
      typed : {
        timeout : 5
      }
      inbox : {
        inbox-size : 1000
        default-timeout : 5s
      }
      router : {
        type-mapping : {
          from-code : Akka.Routing.NoRouter
          round-robin-pool : Akka.Routing.RoundRobinPool
          round-robin-group : Akka.Routing.RoundRobinGroup
          random-pool : Akka.Routing.RandomPool
          random-group : Akka.Routing.RandomGroup
          smallest-mailbox-pool : Akka.Routing.SmallestMailboxPool
          broadcast-pool : Akka.Routing.BroadcastPool
          broadcast-group : Akka.Routing.BroadcastGroup
          scatter-gather-pool : Akka.Routing.ScatterGatherFirstCompletedPool
          scatter-gather-group : Akka.Routing.ScatterGatherFirstCompletedGroup
          consistent-hashing-pool : Akka.Routing.ConsistentHashingPool
          consistent-hashing-group : Akka.Routing.ConsistentHashingGroup
          tail-chopping-pool : Akka.Routing.TailChoppingPool
          tail-chopping-group : Akka.Routing.TailChoppingGroup
        }
      }
      deployment : {
        default : {
          dispatcher : 
          mailbox : 
          router : from-code
          nr-of-instances : 1
          within : "5 s"
          virtual-nodes-factor : 10
          routees : {
            paths : []
          }
          resizer : {
            enabled : off
            lower-bound : 1
            upper-bound : 10
            pressure-threshold : 1
            rampup-rate : 0.2
            backoff-threshold : 0.3
            backoff-rate : 0.1
            messages-per-resize : 10
          }
          remote : 
          target : {
            nodes : []
          }
          metrics-selector : mix
          cluster : {
            enabled : off
            max-nr-of-instances-per-node : 1
            max-total-nr-of-instances : 10000
            allow-local-routees : on
            use-role : 
          }
        }
        /ApiMasterActor : {
        }
        /ApiMasterActor/*/TestClusterRole1HelloActor : {
          router : round-robin-pool
          nr-of-instances : 100
          cluster : {
            enabled : on
            max-nr-of-instances-per-node : 1
            allow-local-routees : off
            use-role : role1
          }
        }
        /ApiMasterActor/*/TestClusterRole2HelloActor : {
          router : round-robin-pool
          nr-of-instances : 100
          cluster : {
            enabled : on
            max-nr-of-instances-per-node : 1
            allow-local-routees : off
            use-role : role2
          }
        }
        /ApiMasterActor/*/TestClusterRole3HelloActor : {
          router : round-robin-pool
          nr-of-instances : 100
          cluster : {
            enabled : on
            max-nr-of-instances-per-node : 1
            allow-local-routees : off
            use-role : role3
          }
        }
      }
      synchronized-dispatcher : {
        type : SynchronizedDispatcher
        executor : current-context-executor
        throughput : 10
      }
      task-dispatcher : {
        type : TaskDispatcher
        executor : task-executor
        throughput : 30
      }
      default-fork-join-dispatcher : {
        type : ForkJoinDispatcher
        executor : fork-join-executor
        throughput : 30
        dedicated-thread-pool : {
          thread-count : 3
          threadtype : background
        }
      }
      default-dispatcher : {
        type : Dispatcher
        executor : default-executor
        default-executor : {
        }
        thread-pool-executor : {
        }
        fork-join-executor : {
          dedicated-thread-pool : {
            thread-count : 3
            threadtype : background
          }
        }
        current-context-executor : {
        }
        shutdown-timeout : 1s
        throughput : 30
        throughput-deadline-time : 0ms
        attempt-teamwork : on
        mailbox-requirement : 
      }
      default-mailbox : {
        mailbox-type : Akka.Dispatch.UnboundedMailbox
        mailbox-capacity : 1000
        mailbox-push-timeout-time : 10s
        stash-capacity : -1
      }
      mailbox : {
        requirements : {
          Akka.Dispatch.IUnboundedMessageQueueSemantics : akka.actor.mailbox.unbounded-queue-based
          Akka.Dispatch.IBoundedMessageQueueSemantics : akka.actor.mailbox.bounded-queue-based
          Akka.Dispatch.IDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based
          Akka.Dispatch.IUnboundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based
          Akka.Dispatch.IBoundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.bounded-deque-based
          Akka.Dispatch.IMultipleConsumerSemantics : akka.actor.mailbox.unbounded-queue-based
          Akka.Event.ILoggerMessageQueueSemantics : akka.actor.mailbox.logger-queue
        }
        unbounded-queue-based : {
          mailbox-type : Akka.Dispatch.UnboundedMailbox
        }
        bounded-queue-based : {
          mailbox-type : Akka.Dispatch.BoundedMailbox
        }
        unbounded-deque-based : {
          mailbox-type : Akka.Dispatch.UnboundedDequeBasedMailbox
        }
        bounded-deque-based : {
          mailbox-type : Akka.Dispatch.BoundedDequeBasedMailbox
        }
        logger-queue : {
          mailbox-type : Akka.Event.LoggerMailboxType
        }
      }
      debug : {
        receive : off
        autoreceive : off
        lifecycle : off
        fsm : off
        event-stream : off
        unhandled : on
        router-misconfiguration : off
      }
      serializers : {
        json : "Akka.Serialization.NewtonSoftJsonSerializer, Akka"
        bytes : "Akka.Serialization.ByteArraySerializer, Akka"
        akka-containers : "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote"
        akka-misc : "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote"
        primitive : "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote"
        proto : "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote"
        daemon-create : "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote"
        akka-system-msg : "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote"
        akka-cluster : "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster"
        akka-cluster-client : "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools"
        akka-pubsub : "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools"
      }
      serialization-bindings : {
        System.Byte[] : bytes
        System.Object : json
        "Akka.Actor.ActorSelectionMessage, Akka" : akka-containers
        "Akka.Remote.DaemonMsgCreate, Akka.Remote" : daemon-create
        "Google.Protobuf.IMessage, Google.Protobuf" : proto
        "Akka.Actor.Identify, Akka" : akka-misc
        "Akka.Actor.ActorIdentity, Akka" : akka-misc
        "Akka.Actor.IActorRef, Akka" : akka-misc
        "Akka.Actor.PoisonPill, Akka" : akka-misc
        "Akka.Actor.Kill, Akka" : akka-misc
        "Akka.Actor.RemoteScope, Akka" : akka-misc
        "Akka.Routing.FromConfig, Akka" : akka-misc
        "Akka.Routing.DefaultResizer, Akka" : akka-misc
        "Akka.Routing.RoundRobinPool, Akka" : akka-misc
        "Akka.Routing.BroadcastPool, Akka" : akka-misc
        "Akka.Routing.RandomPool, Akka" : akka-misc
        "Akka.Routing.ScatterGatherFirstCompletedPool, Akka" : akka-misc
        "Akka.Routing.TailChoppingPool, Akka" : akka-misc
        "Akka.Routing.ConsistentHashingPool, Akka" : akka-misc
        "Akka.Configuration.Config, Akka" : akka-misc
        "Akka.Remote.RemoteWatcher+Heartbeat, Akka.Remote" : akka-misc
        "Akka.Remote.RemoteWatcher+HeartbeatRsp, Akka.Remote" : akka-misc
        "Akka.Remote.Routing.RemoteRouterConfig, Akka.Remote" : akka-misc
        "Akka.Dispatch.SysMsg.SystemMessage, Akka" : akka-system-msg
        System.String : primitive
        System.Int32 : primitive
        System.Int64 : primitive
        "Akka.Cluster.IClusterMessage, Akka.Cluster" : akka-cluster
        "Akka.Cluster.Routing.ClusterRouterPool, Akka.Cluster" : akka-cluster
        "Akka.Cluster.Tools.Client.IClusterClientMessage, Akka.Cluster.Tools" : akka-cluster-client
        "Akka.Cluster.Tools.PublishSubscribe.IDistributedPubSubMessage, Akka.Cluster.Tools" : akka-pubsub
        "Akka.Cluster.Tools.PublishSubscribe.Internal.SendToOneSubscriber, Akka.Cluster.Tools" : akka-pubsub
      }
      serialization-identifiers : {
        "Akka.Serialization.ByteArraySerializer, Akka" : 4
        "Akka.Serialization.NewtonSoftJsonSerializer, Akka" : 1
        "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" : 2
        "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" : 3
        "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" : 6
        "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" : 16
        "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" : 17
        "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" : 22
        "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" : 5
        "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" : 15
        "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" : 9
      }
      serialization-settings : {
      }
    }
    scheduler : {
      tick-duration : 10ms
      ticks-per-wheel : 512
      implementation : Akka.Actor.HashedWheelTimerScheduler
      shutdown-timeout : 5s
    }
    io : {
      pinned-dispatcher : {
        type : PinnedDispatcher
        executor : fork-join-executor
      }
      tcp : {
        direct-buffer-pool : {
          class : "Akka.IO.Buffers.DirectBufferPool, Akka"
          buffer-size : 512
          buffers-per-segment : 500
          initial-segments : 1
          buffer-pool-limit : 1024
        }
        disabled-buffer-pool : {
          class : "Akka.IO.Buffers.DisabledBufferPool, Akka"
          buffer-size : 512
        }
        buffer-pool : akka.io.tcp.disabled-buffer-pool
        max-channels : 256000
        selector-association-retries : 10
        batch-accept-limit : 10
        register-timeout : 5s
        max-received-message-size : unlimited
        trace-logging : off
        selector-dispatcher : akka.io.pinned-dispatcher
        worker-dispatcher : akka.actor.default-dispatcher
        management-dispatcher : akka.actor.default-dispatcher
        file-io-dispatcher : akka.actor.default-dispatcher
        file-io-transferTo-limit : 524288
        finish-connect-retries : 5
        windows-connection-abort-workaround-enabled : off
        outgoing-socket-force-ipv4 : false
      }
      udp : {
        direct-buffer-pool : {
          class : "Akka.IO.Buffers.DirectBufferPool, Akka"
          buffer-size : 512
          buffers-per-segment : 500
          initial-segments : 1
          buffer-pool-limit : 1024
        }
        buffer-pool : akka.io.udp.direct-buffer-pool
        nr-of-socket-async-event-args : 32
        max-channels : 4096
        select-timeout : infinite
        selector-association-retries : 10
        receive-throughput : 3
        direct-buffer-size : 18432
        direct-buffer-pool-limit : 1000
        received-message-size-limit : unlimited
        trace-logging : off
        selector-dispatcher : akka.io.pinned-dispatcher
        worker-dispatcher : akka.actor.default-dispatcher
        management-dispatcher : akka.actor.default-dispatcher
      }
      udp-connected : {
        direct-buffer-pool : {
          class : "Akka.IO.Buffers.DirectBufferPool, Akka"
          buffer-size : 512
          buffers-per-segment : 500
          initial-segments : 1
          buffer-pool-limit : 1024
        }
        buffer-pool : akka.io.udp-connected.direct-buffer-pool
        nr-of-socket-async-event-args : 32
        max-channels : 4096
        select-timeout : infinite
        selector-association-retries : 10
        receive-throughput : 3
        direct-buffer-size : 18432
        direct-buffer-pool-limit : 1000
        received-message-size-limit : unlimited
        trace-logging : off
        selector-dispatcher : akka.io.pinned-dispatcher
        worker-dispatcher : akka.actor.default-dispatcher
        management-dispatcher : akka.actor.default-dispatcher
      }
      dns : {
        dispatcher : akka.actor.default-dispatcher
        resolver : inet-address
        inet-address : {
          provider-object : Akka.IO.InetAddressDnsProvider
          positive-ttl : 30s
          negative-ttl : 10s
          cache-cleanup-interval : 120s
          use-ipv6 : true
        }
      }
    }
    coordinated-shutdown : {
      default-phase-timeout : "5 s"
      terminate-actor-system : on
      exit-clr : off
      run-by-clr-shutdown-hook : on
      run-by-actor-system-terminate : on
      phases : {
        before-service-unbind : {
        }
        service-unbind : {
          depends-on : [before-service-unbind]
        }
        service-requests-done : {
          depends-on : [service-unbind]
        }
        service-stop : {
          depends-on : [service-requests-done]
        }
        before-cluster-shutdown : {
          depends-on : [service-stop]
        }
        cluster-sharding-shutdown-region : {
          timeout : "10 s"
          depends-on : [before-cluster-shutdown]
        }
        cluster-leave : {
          depends-on : [cluster-sharding-shutdown-region]
        }
        cluster-exiting : {
          timeout : "10 s"
          depends-on : [cluster-leave]
        }
        cluster-exiting-done : {
          depends-on : [cluster-exiting]
        }
        cluster-shutdown : {
          depends-on : [cluster-exiting-done]
        }
        before-actor-system-terminate : {
          depends-on : [cluster-shutdown]
        }
        actor-system-terminate : {
          timeout : "10 s"
          depends-on : [before-actor-system-terminate]
        }
      }
    }
    remote : {
      startup-timeout : "10 s"
      shutdown-timeout : "10 s"
      flush-wait-on-shutdown : "2 s"
      use-passive-connections : on
      backoff-interval : "0.05 s"
      command-ack-timeout : "30 s"
      handshake-timeout : "15 s"
      use-dispatcher : akka.remote.default-remote-dispatcher
      untrusted-mode : off
      trusted-selection-paths : []
      require-cookie : off
      secure-cookie : 
      log-received-messages : off
      log-sent-messages : off
      log-remote-lifecycle-events : on
      log-frame-size-exceeding : off
      log-buffer-size-exceeding : 50000
      transport-failure-detector : {
        implementation-class : Akka.Remote.DeadlineFailureDetector,Akka.Remote
        heartbeat-interval : 5s
        acceptable-heartbeat-pause : 50s
      }
      watch-failure-detector : {
        implementation-class : Akka.Remote.PhiAccrualFailureDetector,Akka.Remote
        heartbeat-interval : "1 s"
        threshold : 10.0
        max-sample-size : 200
        min-std-deviation : "100 ms"
        acceptable-heartbeat-pause : "10 s"
        unreachable-nodes-reaper-interval : 1s
        expected-response-after : "1 s"
      }
      retry-gate-closed-for : "5 s"
      prune-quarantine-marker-after : "5 d"
      quarantine-after-silence : "2 d"
      system-message-buffer-size : 20000
      system-message-ack-piggyback-timeout : "0.3 s"
      resend-interval : "2 s"
      resend-limit : 200
      initial-system-message-delivery-timeout : "3 m"
      enabled-transports : [akka.remote.dot-netty.tcp]
      adapters : {
        gremlin : Akka.Remote.Transport.FailureInjectorProvider,Akka.Remote
        trttl : Akka.Remote.Transport.ThrottlerProvider,Akka.Remote
      }
      helios : {
        tcp : {
          transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote.Transport.Helios"
        }
      }
      dot-netty : {
        tcp : {
          transport-class : Akka.Remote.Transport.DotNetty.TcpTransport,Akka.Remote
          applied-adapters : []
          transport-protocol : tcp
          byte-order : little-endian
          port : 0
          public-port : 0
          hostname : 
          public-hostname : 127.0.0.1
          dns-use-ipv6 : false
          enforce-ip-family : false
          enable-ssl : false
          enable-backwards-compatibility : false
          connection-timeout : "15 s"
          batching : {
            enabled : true
            max-pending-writes : 30
            max-pending-bytes : 16k
            flush-interval : 40ms
          }
          use-dispatcher-for-io : 
          write-buffer-high-water-mark : 0b
          write-buffer-low-water-mark : 0b
          send-buffer-size : 2097152b
          receive-buffer-size : 2097152b
          maximum-frame-size : 1048576b
          backlog : 4096
          tcp-nodelay : on
          tcp-keepalive : on
          tcp-reuse-addr : off-for-windows
          server-socket-worker-pool : {
            pool-size-min : 2
            pool-size-factor : 1.0
            pool-size-max : 2
          }
          client-socket-worker-pool : {
            pool-size-min : 2
            pool-size-factor : 1.0
            pool-size-max : 2
          }
        }
        udp : {
          transport-protocol : udp
        }
        ssl : {
          certificate : {
            path : 
            password : 
            use-thumprint-over-file : false
            thumpbrint : 
            store-name : 
            store-location : current-user
          }
          suppress-validation : false
        }
      }
      gremlin : {
        debug : off
      }
      default-remote-dispatcher : {
        type : ForkJoinDispatcher
        executor : fork-join-executor
        dedicated-thread-pool : {
          thread-count : 4
        }
      }
      backoff-remote-dispatcher : {
        type : ForkJoinDispatcher
        executor : fork-join-executor
        dedicated-thread-pool : {
          thread-count : 4
        }
      }
    }
    cluster : {
      seed-nodes : ["akka.tcp://[email protected]:4062"]
      seed-node-timeout : 5s
      retry-unsuccessful-join-after : 10s
      shutdown-after-unsuccessful-join-seed-nodes : off
      auto-down-unreachable-after : 10s
      down-removal-margin : off
      downing-provider-class : 
      allow-weakly-up-members : on
      roles : [api]
      run-coordinated-shutdown-when-down : on
      role : {
      }
      min-nr-of-members : 1
      log-info : on
      log-info-verbose : off
      periodic-tasks-initial-delay : 1s
      gossip-interval : 5s
      gossip-time-to-live : 30s
      leader-actions-interval : 1s
      unreachable-nodes-reaper-interval : 1s
      publish-stats-interval : off
      use-dispatcher : 
      gossip-different-view-probability : 0.8
      reduce-gossip-different-view-probability : 400
      failure-detector : {
        implementation-class : "Akka.Remote.PhiAccrualFailureDetector, Akka.Remote"
        heartbeat-interval : 5s
        threshold : 8.0
        max-sample-size : 1000
        min-std-deviation : "100 ms"
        acceptable-heartbeat-pause : 50s
        monitored-by-nr-of-members : 9
        expected-response-after : "1 s"
      }
      split-brain-resolver : {
        active-strategy : off
        stable-after : 20s
        static-quorum : {
          quorum-size : undefined
          role : 
        }
        keep-majority : {
          role : 
        }
        keep-oldest : {
          down-if-alone : on
          role : 
        }
        keep-refree : {
          address : 
          down-all-if-less-than-nodes : 1
        }
      }
      scheduler : {
        tick-duration : 33ms
        ticks-per-wheel : 512
      }
      debug : {
        verbose-heartbeat-logging : off
        verbose-receive-gossip-logging : off
      }
      client : {
        receptionist : {
          name : receptionist
          role : 
          number-of-contacts : 3
          response-tunnel-receive-timeout : 30s
          use-dispatcher : 
          heartbeat-interval : 2s
          acceptable-heartbeat-pause : 13s
          failure-detection-interval : 2s
        }
        initial-contacts : []
        establishing-get-contacts-interval : 3s
        refresh-contacts-interval : 60s
        heartbeat-interval : 2s
        acceptable-heartbeat-pause : 13s
        buffer-size : 1000
        reconnect-timeout : off
      }
      pub-sub : {
        name : distributedPubSubMediator
        role : 
        routing-logic : random
        gossip-interval : 1s
        removed-time-to-live : 120s
        max-delta-elements : 3000
        use-dispatcher : 
      }
      singleton : {
        singleton-name : ClusterSingleton
        role : role1
        hand-over-retry-interval : 1s
        min-number-of-hand-over-retries : 10
      }
    }
    service : {
      name : Mirero.MLS.TestClusterApi
      description : Mirero.MLS.TestClusterApi
    }
    system : {
      name : mls-cluster-system
    }
  }
  petabridge : {
    cmd : {
      host : 0.0.0.0
      port : 9111
      log-palettes-on-startup : on
    }
  }
  mls : {
    service : {
      dbConnect : {
        ip : 192.168.60.193
        port : 1521
        service : MLSDB
        user : mlsuser
        password : mlsuser
      }
    }
  }
 
[2020/10/23 09:18:30.606][    1][ INFO][Topshelf.Hosts.ConsoleRunHost] The Mirero.MLS.TestClusterApi service is now running, press Control+C to exit. 
[2020/10/23 09:18:31.395][   10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - path:  
[2020/10/23 09:18:31.395][   10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - frequency: 0 
[2020/10/23 09:18:31.395][   10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - active: False 
[2020/10/23 09:18:31.395][   10][ INFO]akka.tcp://[email protected]:3893/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] petabridge.cmd host bound to [0.0.0.0:9111] 
[2020/10/23 09:18:31.403][   10][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Welcome from [akka.tcp://[email protected]:4062] 
[2020/10/23 09:18:32.867][   27][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton manager started singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton]  
[2020/10/23 09:18:32.867][   27][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [Start -> Oldest] Akka.Cluster.Tools.Singleton.Uninitialized 
[2020/10/23 09:18:33.626][   24][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - is the new leader among reachable nodes (more leaders may exist) 
[2020/10/23 09:18:40.751][   10][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:3895] to [Up] 
[2020/10/23 09:18:50.907][   25][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:8112] to [Up] 
[2020/10/23 09:18:55.983][   21][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:8113] to [WeaklyUp] 
[2020/10/23 09:18:58.454][   25][ INFO]akka.tcp://[email protected]:3893/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] Received petabridge.cmd connection from client [127.0.0.1:3908] 
[2020/10/23 09:19:09.655][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$a/$c[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:09.675][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$a/$c[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:11.206][    7][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:8113] to [Up] 
[2020/10/23 09:19:15.035][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$b/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:15.194][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$b/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:19:17.039][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:17.118][   21][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:17.294][    5][ INFO]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Akka.Actor.EmptyLocalActorRef] Message [ActorIdentity] from akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a was not delivered. [1] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 
[2020/10/23 09:19:18.779][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:19:18.938][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:20.511][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$e/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:20.638][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$e/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:19:22.575][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$f/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:22.735][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$f/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:19:26.900][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$g/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:27.028][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$g/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:19:28.735][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$h/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:28.894][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$h/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:19:30.301][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$i/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:30.365][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$i/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:31.844][    7][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$j/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:31.924][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$j/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:19:33.475][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$k/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:19:33.635][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$k/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:20:09.618][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$l/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:09.674][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$l/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:11.763][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$m/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:11.842][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$m/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:20:13.840][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$n/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:13.918][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$n/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:16.127][    7][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$o/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:16.205][   20][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$o/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:17.968][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$p/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:20:18.126][    7][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$p/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:19.697][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:20:19.851][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:21.315][    7][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$r/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:21.376][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$r/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:20:24.216][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$s/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:24.278][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$s/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:20:26.782][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$t/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:26.859][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$t/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:28.825][   29][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$u/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:28.979][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$u/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1, Subject=] 
[2020/10/23 09:20:30.991][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$v/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:31.070][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$v/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:20:34.214][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$w/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:20:34.294][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$w/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:22:05.036][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$x/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:05.193][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$x/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:22:08.407][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:08.486][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:22:11.401][   21][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$z/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:11.480][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$z/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:22:14.348][   20][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$A/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:14.427][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$A/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:22:17.155][   21][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$B/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:17.234][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$B/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:22:32.395][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$C/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:32.471][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$C/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:22:34.026][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$D/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:22:34.104][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$D/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:23:53.942][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$E/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:23:54.005][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$E/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:23:56.216][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$F/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:23:58.383][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$G/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:23:58.446][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$G/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:23:59.900][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$H/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:23:59.963][   29][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$H/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:01.583][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$I/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:01.661][   29][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$I/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:03.174][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$J/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:03.251][   20][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$J/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:04.764][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$K/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:04.922][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$K/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:06.583][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$L/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:06.658][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$L/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:08.312][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$M/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:08.390][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$M/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:09.809][   29][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$N/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:09.887][    7][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$N/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:11.612][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$O/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:11.690][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$O/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:13.079][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$P/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:13.157][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$P/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:14.673][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:14.734][   29][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:16.172][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$R/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:16.251][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$R/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:24.520][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$S/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:24.598][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$S/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:25.824][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$T/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:25.902][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$T/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:27.040][   21][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$U/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:27.118][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$U/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:28.425][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$V/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:28.504][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$V/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:29.567][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$W/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:29.627][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$W/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:30.735][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$X/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:30.799][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$X/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:31.905][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:31.968][   29][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:33.195][    6][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Z/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:33.352][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Z/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1, Subject=] 
[2020/10/23 09:24:43.135][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$0/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:24:43.292][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$0/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:45.645][   20][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$1/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:24:45.723][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$1/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:47.876][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$2/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:24:48.049][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$2/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:25:57.482][   32][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$3/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:25:57.560][   23][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$3/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:25:59.163][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$4/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:25:59.242][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$4/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:00.792][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$5/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:00.872][   21][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$5/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:02.413][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$6/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:02.473][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$6/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:03.995][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$7/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:04.166][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$7/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:26:05.810][    5][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$8/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:26:05.871][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$8/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:13.033][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$9/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:13.096][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$9/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:26:14.829][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$+/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:26:14.889][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$+/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:16.573][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$~/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:16.634][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$~/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:18.166][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$ab/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:18.246][    4][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$ab/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:26:19.947][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$bb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:20.008][   31][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$bb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:22.014][   26][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$cb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:22.092][   24][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$cb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:23.744][   21][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$db/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:23.808][   22][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$db/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://[email protected]:3895/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] 
[2020/10/23 09:26:25.759][   20][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$eb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:25.835][   25][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$eb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:27.673][    7][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$fb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:26:27.750][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$fb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:29.202][   10][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$gb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://[email protected]:8113/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] 
[2020/10/23 09:26:29.357][    9][ WARN]akka.tcp://[email protected]:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$gb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://[email protected]:8112/remote/akka.tcp/[email protected]:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] 
[2020/10/23 09:35:05.318][   22][ INFO][Topshelf.Hosts.ConsoleRunHost] Control+C detected, attempting to stop service. 
[2020/10/23 09:35:05.324][   20][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Marked address [akka.tcp://[email protected]:3893] as [Leaving] 
[2020/10/23 09:35:05.324][   20][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Exited [akka.tcp://[email protected]:3893] 
[2020/10/23 09:35:05.324][   20][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Oldest observed OldestChanged: [akka.tcp://[email protected]:3893 -> ] 
[2020/10/23 09:35:05.324][   20][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [Oldest -> WasOldest] Akka.Cluster.Tools.Singleton.OldestData 
[2020/10/23 09:35:06.355][    5][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton manager stopping singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton] 
[2020/10/23 09:35:06.355][    5][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [WasOldest -> Stopping] Akka.Cluster.Tools.Singleton.WasOldestData 
[2020/10/23 09:35:06.355][    5][ INFO]akka.tcp://[email protected]:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton] was terminated 
[2020/10/23 09:35:08.477][   31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:3893] to [Exiting] 
[2020/10/23 09:35:08.481][   31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:3895] to [Exiting] 
[2020/10/23 09:35:08.481][   31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:8112] to [Exiting] 
[2020/10/23 09:35:08.481][   31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Leader is moving node [akka.tcp://[email protected]:8113] to [Exiting] 
[2020/10/23 09:35:08.520][   31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Exiting completed. 
[2020/10/23 09:35:08.540][   31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://[email protected]:3893] - Shutting down... 
[2020/10/23 09:35:08.682][    1][ INFO][Topshelf.Hosts.ConsoleRunHost] The Mirero.MLS.TestClusterApi service has stopped.

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024

@IgorFedchenko

hocon

TestClusterApi.hocon

petabridge.cmd 
{
    host = "0.0.0.0"
    port = 9111
    log-palettes-on-startup = on
}

akka 
{
    service.name = "TestClusterApi"
	service.description = "TestClusterApi"
    system.name = "mls-cluster-system" 	

    stdout-loglevel = DEBUG
    loglevel = DEBUG
    log-config-on-start = on
    loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

    actor
    {
        provider = cluster
		guardian-supervisor-strategy = "TestClusterApi.ActorService.UserSupervisorStrategy, TestClusterApi"

        debug
        {
            receive = off         
            autoreceive = off  
            lifecycle = off       
            event-stream = off
            unhandled = on    
        }

        deployment 
        {
			/ApiMasterActor 
			{
			
			}

            "/ApiMasterActor/*/TestClusterRole1HelloActor"
            {
				router = round-robin-pool
				nr-of-instances = 100
				
			    cluster
                {
					enabled = on
                    max-nr-of-instances-per-node = 1
                    allow-local-routees = off
                    use-role = role1
                }
            }

			"/ApiMasterActor/*/TestClusterRole2HelloActor"
            {
				router = round-robin-pool
				nr-of-instances = 100
				
			    cluster
                {
					enabled = on
                    max-nr-of-instances-per-node = 1
                    allow-local-routees = off
                    use-role = role2
                }
            }

			"/ApiMasterActor/*/TestClusterRole3HelloActor"
			{
				router = round-robin-pool
				nr-of-instances = 100

			    cluster
                {
					enabled = on
                    max-nr-of-instances-per-node = 1
                    allow-local-routees = off
                    use-role = role3
                }
            }
        }
    }

    remote
    {
        dot-netty.tcp 
        {
            public-hostname = 127.0.0.1
			port = 0

            send-buffer-size    = 2097152b  # 2MB
            receive-buffer-size = 2097152b  # 2MB
            maximum-frame-size  = 1048576b  # 1MB
        }
        
        transport-failure-detector
        {
            heartbeat-interval = 5s                
            acceptable-heartbeat-pause = 50s   
        }
    }
    
    extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]

    cluster 
    {
        seed-nodes = ["akka.tcp://[email protected]:4062"]
        roles = [api]

        auto-down-unreachable-after = 10s 

		gossip-interval = 5s 
        gossip-time-to-live = 30s 

        failure-detector 
        {
            heartbeat-interval = 5s
            acceptable-heartbeat-pause = 50s
        }

		singleton
		{
			singleton-name = "ClusterSingleton"
			role = "role1"
			hand-over-retry-interval = 1s
			min-number-of-hand-over-retries = 10
		}
		
		#singleton-proxy 
		#{
		#	singleton-name = ${akka.cluster.singleton.singleton-name}
		#	role = "role1"
		#	singleton-identification-interval = 1s
		#	buffer-size = 1000
		#}
    }
}

Lighthouse.hocon

lighthouse
{
    actorsystem: "mls-cluster-system" #POPULATE NAME OF YOUR ACTOR SYSTEM HERE
}

petabridge.cmd
{
    host = "0.0.0.0"
    port = 9115
}

akka 
{
    service.name = "TestLightHouse4061"
    service.description = "TestLightHouse.4061"
    system.name = "mls-cluster-testlighthouse-4061"

    stdout-loglevel = DEBUG
    loglevel = DEBUG
    log-config-on-start = on
    loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

    actor 
    {
        provider = cluster

		debug
        {  
            receive = off      
            autoreceive = off  
            lifecycle = off    
            event-stream = off
            unhandled = on
        }
    }

    remote 
    {
        log-remote-lifecycle-events = DEBUG

        dot-netty.tcp
		{
			public-hostname = 127.0.0.1
			port = 4062
		}
        
        transport-failure-detector
		{
			heartbeat-interval = 5s                
			acceptable-heartbeat-pause = 50s       
		}
    }


    extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]

    cluster 
    {
        seed-nodes = ["akka.tcp://[email protected]:4062"] 

        roles = [lighthouse]
        auto-down-unreachable-after = 10s

        gossip-interval = 5s # how often should the node send out gossip information?
        gossip-time-to-live = 30s # discard incoming gossip messages if not handled within this duration

        client.receptionist
		{
			role = lighthouse

			response-tunnel-receive-timeout = 1d    # 1day
		}

        failure-detector 
        {
            heartbeat-interval = 5s
            acceptable-heartbeat-pause = 50s
        }
    }
}

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024

@kimbyungeun Ok, thanks! I was just hoping that you have a .net project/solution source you can share here to make reproduction simpler.

If you do not have any, we will try do make our own, this will just take more time.

from petabridge.cmd-issues.

kimbyungeun commented on July 21, 2024

@IgorFedchenko

I tested it with Akka.NET v1.4.12 version but it is the same.

Akka.Remote v1.4.12
Akka.Cluster v1.4.12
Petabridge.Cmd.Cluster v0.8.0.1
Petabridge.Cmd.Common v0.8.0.1
Petabridge.Cmd.Host v0.8.0.1

PS C:\Users\be.kim> pbm 127.0.0.1:9111
 ____      _        _          _     _
|  _ \ ___| |_ __ _| |__  _ __(_) __| | __ _  ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
|  __|  __| || (_| | |_) | |  | | (_| | (_| |  __/
|_|   \___|\__\__,_|_.__/|_|  |_|\__,_|\__, |\___|
                                       |___/

petabridge.cmd (0.8.0.0)
Copyright 2017 - 2020, Petabridge®.

successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$a/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$b/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$c/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$d
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$d/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$e
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$e/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$f
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$f/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$g
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$g/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024

@kimbyungeun Thanks, was thinking of checking that!

from petabridge.cmd-issues.

Aaronontheweb commented on July 21, 2024

@IgorFedchenko I think the issue here is we print out the results in any order when they arrive, so the output isn't sorted on a per-level basis in the hierarchy. i.e. all children at the same position in the hierarchy might print out differently each time it's run.

Answer would be to sort the actor paths I guess

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024

@Aaronontheweb If I understood the issue right, the problem is that child actors sometimes are not displayed at all. From the latest logs above:

@kimbyungeun said that actors are running all the time, so they should be displayed all the time.

Right?

from petabridge.cmd-issues.

Aaronontheweb commented on July 21, 2024

Ah that makes sense @IgorFedchenko - we do time out the search after a while so that may be what is happening here.

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024

Hmm... I will check that - thanks for pointing the direction!

from petabridge.cmd-issues.

IgorFedchenko commented on July 21, 2024

@Aaronontheweb I was able to reproduce this issue. But I have a question to you.

Long story short: we have Akka.net cluster, 3 nodes (there are more, but this is not important). Cluster singleton named ApiMasterActor is created, and this singleton on start creates 3 child actors. Then first actor on scheduled time is sending message via IActorRef to second actor, second is sending message to third, third just logging a message. And so on each 5 seconds, timer -> 1 -> 2 -> 3.

Sometimes, Petabridge.Cmd does not give any c1 actor:

[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor

but - sometimes, there is some c1 actor popping up, here and there, as a child of this "hello" actors. Sometimes, as a child of one of this actors, sometimes as a child of two, or even all three of them, for example:

[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1

So, the question is: does Akka.NET create some child actor automatically somehow, when actors are sending or maybe receiving messages, or they are children of a singleton, or under some other conditions?.. Because, there is nothing that creates this child c1 actor on the user side. And I failed to find anything like such child actors creation in the akka.net source in reasonable time.

@kimbyungeun So far, it seems to me like Petabridge.cmd is working fine, and it's just some temporary "service" actor created by Akka.NET under the hood, that is displayed in the hierarchy from time to time.

from petabridge.cmd-issues.

Petabridge.Cmd `actor hierarchy` prints different results each time. about petabridge.cmd-issues HOT 24 CLOSED

Comments (24)

Q1

Q2 : `acceptable-heartbeat-pause = 240s`

Using akka 1.3.16 will print all actors.

pbm api log

Using akka 1.4.10 will print some actors.

pbm lighthouse log

pbm api log

[api] role hocon

Q1

Q2

Lighthouse.log

TestClusterApi.log

hocon

TestClusterApi.hocon

Lighthouse.hocon

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Comments (24)

Q1

Q2 : acceptable-heartbeat-pause = 240s

Using akka 1.3.16 will print all actors.

pbm api log

Using akka 1.4.10 will print some actors.

pbm lighthouse log

pbm api log

[api] role hocon

Q1

Q2

Lighthouse.log

TestClusterApi.log

hocon

TestClusterApi.hocon

Lighthouse.hocon

Related Issues (20)

Recommend Projects

Recommend Topics

Recommend Org

Q2 : `acceptable-heartbeat-pause = 240s`