The pin_memory option in the MultiThreadedAugmenter might have a memory leak about batchgenerators HOT 4 CLOSED

Hi,
I agree that there can be a memory leak if you repeatedly delete and re instantiate the multithreaded augmenter. The pin_memory feature is experimental and I added it only last week. I will try to get this fixed as soon as possible. Until then, don't reinstantiate it or use pin_memory=False.

Following the way pytorch does this is very complicated (and a lot of work to integrate into our code) and I think we can try to get it running easier somehow. If you have ideas, please let me know =)

Thanks for pointing this out!
Best,
Fabian

from batchgenerators.

All workers and the pin memory thread now use an abort_event (multiprocessing.Event) and will exit if it was set. This event can be set from various places, such as crashing workers (which would previously have resulted in the main process waiting for new data indefinitely), crashing other things or calling the destructor of MultiThreadedAugmenter. Please test and see if you still experience memory leaks. If you do, please provide a minimalistic example and I will figure out a way to solve this.
Best,
Fabian

from batchgenerators.

Works perfectly now! Thanks for the quick fix!

from batchgenerators.

No worries, it's always good to get user feedback =) Thanks a lot again for pointing out the issue

from batchgenerators.