Memory leak on constant /predict requests about deepdetect HOT 8 CLOSED

should be better with #1262

from deepdetect.

Hi @rguilmont , thanks for the report. We started to investigate, but it seems that the leak does not happen when using valgrind (!), which makes it quite more difficult to investigate than expected. We will keep you informed as soon as we find somthing.

from deepdetect.

Oh, thanks for the update !

FYI i tried to reproduce locally with CPU, and i was not able to. I do see an increase in ram usage, but then it get stable. So i was able to replicate only with models loaded with tensorrt. If i can provide you more information, don't hesitate :)

from deepdetect.

yes, same behavior using tensorrt/GPU within valgrind 😢

from deepdetect.

I'll try to get more information about it, and maybe try to reproduce too with Valgrind if get the time to :)

I just hope this is not hardware related... That would make things even more complicated 😂

FYI we've spotted the leak on g4dn.xlarge from Amazon with Tesla T4 gpu

from deepdetect.

Hi @rguilmont , we found the issue. Indeed @beniz has run in the same problem some time ago with tensorflow backend. We are currently trying to create a clean fix

from deepdetect.

Awesome news !! Out of curiosity, did you find it with Valgrind in the end ? :)

from deepdetect.

It's not a leak, you could actually run as is. It's the malloc library that fragments the memory due to underlying micro allocations, probably by trt.
Using tcmalloc instead of the standard malloc may do the trick.

from deepdetect.