Memory leak on compressed predict requests with oatpp about deepdetect HOT 7 CLOSED

@rguilmont @YaYaB gzip/deflate encryption is handled by https://github.com/oatpp/oatpp-zlib from within https://github.com/oatpp/oatpp. The components are simply added here: https://github.com/jolibrain/deepdetect/blob/master/src/http/app_component.hpp#L114

Running valgrind on dede with gzip queries only shows the possible leak below. This looks like an init from libz directly, from the oatpp send function.

@lganzzzo Hi, the ::send function seems to leak from deflateInit, have you seen this before, or are we doing something wrong ? Thanks.

Libz init memory reported by valgrind:

==3020638== 536,192 (11,904 direct, 524,288 indirect) bytes in 2 blocks are definitely lost in loss record 4,799 of 4,801
==3020638==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3020638==    by 0x5AA3418: deflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.11)
==3020638==    by 0x5AA3651: deflateInit_ (in /lib/x86_64-linux-gnu/libz.so.1.2.11)
==3020638==    by 0x71C063: oatpp::zlib::DeflateEncoder::DeflateEncoder(long, bool, int) (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638==    by 0x71B2E8: oatpp::zlib::DeflateEncoderProvider::getProcessor() (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638==    by 0x6EA02D: oatpp::web::protocol::http::outgoing::Response::send(oatpp::data::stream::OutputStream*, oatpp::data::stream::BufferOutputStream*, oatpp::web::protocol::http::encoding::EncoderProvider*) (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638==    by 0x6F6DB6: oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&) (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638==    by 0x6FB28F: oatpp::web::server::HttpProcessor::Task::run() (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638==    by 0x945BDE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==3020638==    by 0x936B608: start_thread (pthread_create.c:477)
==3020638==    by 0x9837292: clone (clone.S:95)

from deepdetect.

More info on that : we managed to reproduce this issues with the above Python scripts, but not with curl.

So after testing more with @YaYaB, we had a strong intuition that it has something to do with the HTTP serving.
After analysing HTTP headers, we found that Python requests by default asks for GZIP encoded answer ( Accept-Encoding: gzip, deflate ) while curl doesn't.
So we manually set this header in curl, and finally reproduced the issue with curl too.

We also tested to send gzip-compressed queries, asking for uncompressed responses, and no memory leak was noticed. So really looks like it's something related to GZIP compression.

from deepdetect.

Actualy it is even not related to tensorrt but even with classical caffe predictions with or without gpu

from deepdetect.

Hey @beniz ,

Your code looks good. Most probably it's on oatpp side.
I'll take a closer look.

from deepdetect.

Hi @lganzzzo how are things ? Do you have any fresh lead on this by any chance ? I've seen issues with libz a long time ago, this could still be outside oatpp.

from deepdetect.

Hey @beniz ,

Yes, at this point it looks like a libz issue.
I'm filing an issue in oatpp to investigate possible fixes.

It might take a while

from deepdetect.

Thanks a lot guys.

FYI we've mitigated this gzip issue by setting an Envoy proxy in front of deepdetect, taking care of compression and decompression of requests.

from deepdetect.