Review error scenario of full network outage (no replica can be reached) about voyage HOT 2 OPEN

Another issue is that some of the pooled Mongo instances are not closed. Leading to populate raising an exception about the external semaphores not being able to be registered.

from voyage.

With a session pool of max 10 entries I end up with 101 open Mongo connections during a stepdown from primary to the new primary.

This seems to be due lack of concurrency control in the VOMongoRepositoryResolver. In case a query fails >>#resetPool or just >>#reset will be invoked. In the latter the existing pool will be flushed and a new one is created. There are three race conditions.

1. The >>#flush and >>#populate are ran concurrently. New connections are added to a discarded pool after it has been flushed. This can be mitigated by setting the desired pool size to 0 before calling flush. It does seem to fix the issue I see.

2. With multiple calls to >>#reset at the same time we can populate/discard pools that we have not seen yet. There should be some locking around the pool...

from voyage.