export_to_file running out of memory about pyexasol HOT 8 CLOSED

Hi @Nicoretti ,

trying to nail down the exact spot where this happens...
On my local machine the export to file doesn't seem to cause a memory spike.

I'll post again once I know more.

best regards

from pyexasol.

Hi Nico,

thanks for your answer!
In that case we'll probably contact the support, though I need to discuss this internally beforehand.

Best regards

from pyexasol.

It should not be the case. export_to_file() is a basic wrapper for shutil.copyfileobj: https://github.com/exasol/pyexasol/blob/master/pyexasol/callback.py#L45-L52

Could you post full error message and a bit more context?

Thank you.

from pyexasol.

Currently I don't have any further data on the error, will try to get some more information when the job runs again tomorrow.

from pyexasol.

Hi @xyxz-web,

are there any news/updates from your side?
We are happy to help, but without more details we can't really do anything.

Also feel free to close the issue and open/reopen an issue, once you have more details.
I will add issue templates, to give issues creators more guidance in the future.

best
Nico

from pyexasol.

I've run the exact same code on Windows and a Kubernetes pod based on a Debian Bullseye image with the following results:

• On Windows the Job needs 63Mib RAM according to memory_profiler.
• On Kubernetes I currently don't have memory_profiler, but the Pod is killed with OOM at 8Gi RAM halfway through.

Do you have an idea what could cause this?

from pyexasol.

Hi @xyxz-web

Thanks for the update, still this information is too vague to provide any sophisticated help.

In order to provide sophisticated help, we generally require the following things:

1. Detailed Description: Please give us a clear and detailed explanation of the issue, including any error messages and steps to reproduce it.

2. Version Info: Let us know which version of the library or software you're using.

3. Expected vs. Actual Behavior: Tell us what you expected to happen and what actually happened.

4. Reproducibility: If possible, provide steps to consistently reproduce the issue.

5. Logs/Screenshots: Include relevant logs or screenshots if available.

That said, as the library generally works well (also in your windows deployment), the issue also could be related to the Kubernetes deployment itself.

As alternative to this "public" issue tracking board, you also can reach out to the official Exasol-Support which isn't publicly disclosed (e.g. if the problem contains sensitive information).

best
Nico

from pyexasol.

@xyxz-web I'll, close this issue then. If you later want to come back to it with more details, feel free to reopen it.

best
Nico

from pyexasol.