Investigate and fix source of errors in WMArchive about wmarchive HOT 6 OPEN

I updated WMArchive configuration to use 1min threshold for recv/send timeouts on production and testbed clusters (FYI: @arooshap , @muhammadimranfarooqi) . Apart from that as I explained earlier is no longer allocated to development on services outside of WM area and further development efforts should be addressed via @klannon

from wmarchive.

@vkuznet
What is the average duration of WMarchive connection to AMQ brokers? Do you have monitoring to share?

from wmarchive.

Yuyi, you can find relevant information over here: https://monit-grafana.cern.ch/d/u_qOeVqZk/wmarchive-monit?orgId=11 and https://monit-grafana.cern.ch/d/wma-service/wmarchive-service?orgId=11 The first one contains the latency plot.

from wmarchive.

Valentin, Which plots show the WMArchive to AMQ connection duration or disconnection rate?

from wmarchive.

Yuyi, I pointed out to existing dashboard, but it does not have duration of AMQ connection, someone should add this to the code. Said that, it is trivial to see from wmarhive logs (vocms750:/cephfs/product/wma-logs/):

...
2023/05/14 00:05:32 stomp.go:168: send data to 188.185.13.100:61313 endpoint /topic/cms.jobmon.wmarchive
2023/05/14 00:05:32 stomp.go:168: send data to 188.185.11.68:61313 endpoint /topic/cms.jobmon.wmarchive
2023/05/14 00:06:28 stomp.go:168: send data to 188.185.35.176:61313 endpoint /topic/cms.jobmon.wmarchive
2023/05/14 00:06:28 wmarchive.go:298: POST /wmarchive/data/ 10.100.36.192:60508 [WMCore.Services.Requests/v002] [/DC=ch/DC=cern/OU=computers/CN=wmagent/vocms0255.cern.ch] [188.185.89.194] {"result":[{"ids":["c5b8aed966fc4585865d2da2ebfd1b0d"],"status":"ok"}]}
2023/05/14 00:06:31 stomp.go:168: send data to 188.184.92.147:61313 endpoint /topic/cms.jobmon.wmarchive
2023/05/14 00:06:31 stomp.go:168: send data to 188.184.92.147:61313 endpoint /topic/cms.jobmon.wmarchive

So, connection did not last more than a minute since logs shows every time WMArchive sends the data and timestamp in logs shows that usually we have few log entries within a minute.

from wmarchive.

FWIW, I can confirm that the problem is still present. I still see an abnormal number of warnings coming from the cmsweb machines and linked to the small (1.5s) heart-beat threshold.

from wmarchive.