"no healthy upstream" on NorthFlank about cgm-remote-monitor HOT 4 CLOSED

Same issue again just now. Only now also Safari does not work anymore. Anyone else seeing this, on NorthFlank?

from cgm-remote-monitor.

Looks like an issue with Mongo, just now got "Unable to connect to Mongo". Looking into, but I wonder why I did not get this message earlier.

from cgm-remote-monitor.

For some reason, after doing a restart of the rollout, the connection suddenly started working again:

2024-03-10T13:20:06.222187406Z stdout F Successfully established connection to MongoDB
2024-03-10T13:19:05.938507986Z stdout F Error connecting to MongoDB: {"reason":{"type":"ReplicaSetNoPrimary","servers":{},"stale":false,"compatible":true,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"setName":"rs0","maxElectionId":null,"maxSetVersion":null,"commonWireVersion":0,"logicalSessionTimeoutMinutes":null}} - retrying in 60 sec
2024-03-10T13:17:35.91513115Z stdout F Error connecting to MongoDB: {"reason":{"type":"ReplicaSetNoPrimary","servers":{},"stale":false,"compatible":true,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"setName":"rs0","maxElectionId":null,"maxSetVersion":null,"commonWireVersion":0,"logicalSessionTimeoutMinutes":null}} - retrying in 60 sec
...

No idea yet as to what the cause might be. Also see #6775.

from cgm-remote-monitor.

Discussing with NorthFlank's (very kind and helpful) support, this seems to have been caused by an unknown (race) condition which caused two instances to be up at the same time, where one of the two was receiving traffic while in a broken state. Theory for now that this was due to a (temporarily) overloaded server in the free tier - although support confirmed that this "should" not have happened. Closing for now.

from cgm-remote-monitor.