Cloudserver memory leak about zenko HOT 8 CLOSED

Hi @descrepes,

Thank you for your bug report!

If at least one of the Zenko locations is an AWS-S3-compatible location, it might be due to a socket leak issue that we are currently fixing for the next patch release of Zenko, that can happen when connection errors/timeouts occur on the connections to the AWS-compatible backend. The leaked sockets usually retain some data in their TCP buffers, causing a memory leak as well. Not guaranteed it is the same issue that you are witnessing, but once we have the fix ready you may give it a try and see if the memory leak is resolved for you.

Another idea could be to instrument the running cloudserver process with node --inspect, which can give an idea of where the memory is being consumed.

from zenko.

@descrepes in the meantime, if you would like to try out a provisional fix before we release a patch, you can apply the following patch to cloudserver 8.2 and re-build the image (the actual fix is in a branch in Arsenal repository, hence this is just a dependency update, and it actually also contains another fix for a cloudserver worker crash):

diff --git a/package.json b/package.json
index a0417b82..a3527108 100644
--- a/package.json
+++ b/package.json
@@ -20,7 +20,7 @@
   "homepage": "https://github.com/scality/S3#readme",
   "dependencies": {
     "@hapi/joi": "^17.1.0",
-    "arsenal": "github:scality/Arsenal#2461b5c",
+    "arsenal": "github:scality/Arsenal#1106b7f",
     "async": "~2.5.0",
     "aws-sdk": "2.363.0",
     "azure-storage": "^2.1.0",

from zenko.

Hi,
Thx for the information 😄

I rebuild the image with the patch applied to 8.2 but the memory leak is still here:

from zenko.

Hi,

We upgraded to 8.2.7 and we still have the memory leak.
This leak is not present in the 8.0.22.

Regards.

from zenko.

@jonathan-gramain one important thing to note is that we are mostly using Azure Blob as backend.

Regards.

from zenko.

I can send you some nodejs profiling memories if it helps :)

from zenko.

It is possible that there is also a memory leak in the Azure Blob backend. Please send the nodejs profiling !
Thanks a lot
Vianney

from zenko.

Closing this as it was confirmed offline that this issue has been fixed in 1.2.2

from zenko.